"3.3.1 … Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited)."

This is the very bad (sad) addition (restriction) to the iPhone developers program everyone is talking about right now. And I feel very concerned about this... and now everyone is wondering if software like the Flash-to-iPhone packager, Mono or Unity are definitely banned from the iPhone programming tool-chain.

I never talked about it before, but Aerys is working on a project targeting the iPhone and the iPad (among a lot of other platforms). And this very project is leveraged by some home-made code generation tool. Therefore, we are very scared of what this kind of restrictions might imply for our business.

But as scary as this statement might look at first, it is actually even funnier when you understand it makes any iPhone/iPad application illegal. And it gets even better because even Apple's standard tool-chain finds itself... prohibited!

The reason is very simple: Objective-C, C or C++ never "link" to any library providing the Documented iPhone APIs. Objective-C, C or C++ actually never link to anything. When compiling your code, the compiler creates an intermediary representation of your program in order to make it easier to (eventually cross-)compile it. In the case of GCC, the compiler used to build such applications, those intermediary languages are:

• the historical Register Transfer Language (RTL)
• the tree language GENERIC
• the SSA-based GIMPLE

(Source: Wikipedia - Intermediate language)

This very intermediate representation is built before the symbols resolution and before any linking is done. Thus, linking against the "Documented APIs" has always been done "through an intermediary translation or compatibility layer". And it won't change anytime soon! But, still, I wish good luck to Apple's engineers in their future work to patch GCC and make it "compliant" (sic!).

To me, it is the most obvious sign that Apple doesn't care about its developers (but I was still pretty thrilled to eventually become one of them about a few days ago...). Not because it bannishes any interesting cross-compiling software venture. And not even because any iPhone/iPad developper who knows about other platforms openness and ease of production will feel sick just by reading this kind of restrictions.

No. To me, Apple is losing it all simply because this kind of idiocy can only be written by someone who doesn't have a clue about how a computer/compiler works! Apple started building casual devices and ends up targeting casual developers. Could this be the first sign of how (so called) intuitiveness - when enforced by completely hermeticly closed devices - finally makes people... dumber? So now the real question is "What happens when your developers program EULA is written by a marketing control-freak moron?".

PS : The title of this article was desgined to attract the eye. It is my interpretation of the latest Apple's developers program EULA additions and I might be completely off or wrong. Anyway, the doubt and worry caused by this very fuzzy and unclear addition stands as a major issue for all the companies and developers targeting the iPhone and iPad platforms.

It was inspired by Mr. Doob's Stats class. But my Monitor class is entirely dynamic and you can watch any property of any object.

The following simple (and useless?) code sample show how to track the (x, y) position of the mouse cursor on the stage:

package
{
  import aerys.monitor.Monitor;
 
  import flash.display.Sprite;
 
  public class MonitorDemo extends Sprite
  {
    public function MonitorDemo()
    {
      var monitor : Monitor	= new Monitor(30);
 
      monitor.watch(stage, "mouseX", 0xff0000, 1. / stage.stageWidth);
      monitor.watch(stage, "mouseY", 0x008800, 1. / stage.stageHeight);
      addChild(monitor);
    }
  }
}

And here is the result:

Here is another simple use case: tracking the (x, y, z) rotation of a 3D camera (the actual application is available on the official Aerys website):

The code is open source and available on Google Code. Comments and suggestions would be greatly appreciated!

I would be more than happy to talk about the software, the way I built it and the technical choices that drove its development. I will  also try to emphasize what makes this library different through a few demonstrations.

Depending on the agenda of one of my co-worker, we might also present a very cool piece of software I never spoke about!

You can read more about the event here (in french).

Rich of this new and incredible ability, Christophe Coenraets posted a small but yet very powerful code snippet to build an HTTP web server using AIR 2.0!

Optimization is always important. But when it comes to 3D for the Flash Platform, it's an everyday battle. The first ideas that come to mind are to avoid:

1. redrawing the same regions : each pixel value must be set once and only once
2. rendering invisible objects : objects that are out of sight still take a lot of CPU horsepower

While Flash takes care of the first one in its very renderer, the second one is not handled. But that is something we can easily address!

The Technic

The method is called "frustum culling". The big picture is that every mesh is approximated using a bounding volume (typically a sphere or a box). If the bounding volume is visible, the corresponding mesh is rendered. The two following pictures show the frustum culling caught in action:

The mesh is visible: TPS counter indicates 18900 triangles per seconds

The mesh is out of sight: frustum culling is acting and TPS counter indicates 0!

1. The frustum

First things first: before performing frustum culling, you need to build the frustum! The frustum is the 3D shape that contains everything the camera can see. It's made of 6 planes and looks like a truncated pyramid:

The view frustum

Each plane is defined by a plane equation: ax + by + cz + d, where (a, b, c) is the normal of the plane and d the distance on that normal. The 6 planes must be extracted from one of the Matrix3D object used to define the transform from one space to another (world to camera for example). Here are the 6 plane equations:

• left: (a = m14 + m11, b = m24 + m21, c = m34 + m31, d = m44 + m41)
• right: (a = m14 − m11, b = m24 − m21, c = m34 − m31, d = m44 − m41)
• bottom: (a = m14 + m12, b = m24 + m22, c = m34 + m32, d = m44 + m42)
• top: (a = m14 − m12, b = m24 − m22, c = m34 − m32, d = m44 − m42)
• near: (a = m13, b = m23, c = m33, d = m43)
• far: (a = m13, b = m23, c = m33, d = m43)

The source matrix you chose defines the coordinates systems of the extracted planes:

• the projection matrix provides planes in view (or camera) space
• the view * projection matrix provides planes in world space
• the world * view * projection matrix provides planes in object (or local) space

2. Bounding volumes

Testing every polygon against the frustum is a very expensive task. In order to speed things up, we will rather test a primitive shape that bounds the mesh: a "bounding volume". Common bounding volumes are spheres or boxes. I'll focus on the first one in the rest of the article.

A bounding sphere is pretty simple to implement: you just need to store the center of the sphere and it's radius. That being said, a BoundingSphere class would just contain the center as a Vector3D and the radius as a Number. When you have all the vertices of a mesh, building the bounding sphere is easy :

public function computeBoundingSphere(myVertices : Vector.<Number>) : BoundingSphere
{
	var min : Vector3D = new Vector3D(Number.MAX_VALUE, Number.MAX_VALUE, Number.MAX_VALUE);
	var max : Vector3D = new Vector3D(Number.MIN_VALUE, Number.MIN_VALUE, Number.MIN_VALUE);
	var length : int = myVertices.length;
	var center : Vector3D = null;
	var radius : Number = null;
 
	for (var i : uint = 0; i < length; i += 3)
	{
		min.x = myVertices [i] < min.x ? myVertices[i] : min.x;
		min.y = myVertices [uint(i + 1)] < min.y ? myVertices[int(i + 1)] : min.y;
		min.z = myVertices [uint(i + 2)] < min.z ? myVertices[int(i + 2)] : min.z;
 
 		max.x = myVertices [i] > max.x ? myVertices [i] : max.x;
		max.y = myVertices [uint(i + 1)] > max.y ? myVertices[int(i + 1)] : max.y;
		max.z = myVertices [uint(i + 2)] > max.z ? myVertices[int(i + 2)] : max.z;
	}
 
	center = new Vector3D((myMax.x + myMin.x) / 2.0, (myMax.y + myMin.y) / 2.0, (myMax.z + myMin.z) / 2.0);
        radius = Math.max(myMax.x - center.x, myMax.y - center.y, myMax.z - center.z,
		          center.x - myMin.x, center.y - myMin.y, center.z - myMin.z);	
 
	return new BoundingSphere(center, radius);
}

3. Point to plane distance

The plane equation (ax + by + cz + d) can be stored as a Vector3D object. Indeed, the Vector3D implements x, y and z but also a w Number attribute. This last one is supposed to store the homogenous coordinate. Nonetheless, we can use those 4 Number attributes to store the (a, b, c, d) values that defines our plane equation.

In order to test whether a bounding volume is inside or outside the frustum, we must be able to compute the distance of this object to the plane. To do so, we use the dot product of the point with the plane normal (which must be normalized first) :

public function planeToPointDistance(myPlane : Vector3D, myPoint : Vector3D) : Number
{
	return myPlane.x * myPoint.x + myPlane.y * myPoint.y + myPlane.z * myPoint.z + myPlane.w;
}

4. The test function

Now that our frustum and our bounding volumes are ready, we can add this little test function that checks whether an object is visible or not:

public function testBoundingSphere(myPlanes      : Vector.,
                                   mySphere      : BoundingSphere) : Boolean
{
	var length : int = myPlanes.length;
 
	for (var i : uint = 0; i < length; ++i)
	{
		var d 	: Number = planeToPointDistance(myPlanes[i]);
 
		// bouding sphere is out of the frustum
		if (d + mySphere.radius < 0.0)
			return false;
		// bounding sphere is spanning
		else if (d - mySphere.radius <= 0.0)
			return true;
	}
 
	// bounding sphere is inside the frustum
	return true;
}

The Experiment

The following experiment demonstrates frustum culling in action (click on the picture to open the application) :

Frustum culling experiment (use the arrow keys to move)

Use the arrow keys to move the robot across the scene. You can notice the TPS counter drops down to zero whenever the character is out of sight (on the near left or near right side of the screen). When the TPS indicates 0, no triangles are rendered and CPU usage drops down to 0%. It actually means that the rest of the rendering process has a minimal overhead!

And it's even more awesome...

This experiment also introduce my brand new mini-debugger inspired of Mr. Doob's work. I replaced the maximum memory footprint counter with a per-second rendered triangles counter (or TPS counter). CPU usage or framerate alone are not absolute measurements but the TPS counter reflects the true performance of the application the old fashioned way: the more polygons per second the better. I also updated the timer to display the processing time and the overall rendering time.

You can compare this experiment to the "old" one I made up a few weeks ago: the displayed 3D model features 30% more polygons but the CPU usage is the same.

The rendering process is done in such a way that absolutely no computation occurs on invisible objects, not even the slightest animation or z-sorting operation.

I enabled dynamic bounding volumes for animated meshes: if you place the mesh on the edges of the viewing space, you might notice that the TPS counter sometimes goes down to zero to raise up again to a few hundreds/thousands of triangles per second. The explanation is simple: the visibility test is done dynamically and follows the mesh animations. If the animation drives the mesh out of the viewing frustum, culling operates and it is not renderered.

Augmented reality (AR) is a term for a live direct or indirect view of a physical real-world environment whose elements are merged with (or augmented by) virtual computer-generated imagery - creating a mixed reality.

Sounds a bit blury? Well... I'll try to make it clearer with a demo...

Demo

First, you will have to print a little black and white "marker". The AR software scans the webcam picture and look for this very marker. When it is found, its 3D position, rotation and scale are computed and used to embed a 3D object. You can found the marker here and it looks like this:

FLARToolkit Marker

Then, launch the following application and place the printed marker in front of your camera:

FLARToolkit demo application (webcam required)

What it does...

• Load a MD2 file and a JPEG texture
• Loop through the animation of the model
• Define multiple materials (press "w" to switch)
• Use the FLARToolkit library for the AR
• Translate FLARToolkit matrices to fit the 3D API

How it works

• The viewport and a dummy camera are initialized using the camera parameters and the size of the capture
• The webcam image is binarized using a threshold filter: aeras that are darker than a threshold value end up completely white on a black background. Areas that are too small or too big are rejected.
• Each remaining area is detected and its bounding box and position are computed
• For each area that "looks like" a square, the data of what should be the pattern is extracted
• This data is then compared to the actual pattern
• If the match confidence is above a certain value (80% is this case), a matrix that contains the position/rotation/scale of the center of the marker is computed
• This matrix is converted into a native Matrix3D object that can be used as the world transform for the 3D objects we want to display

What it does not do...

• Use FLARManager to handle the AR. Lee "The Flash Blog" Brimelow juste made this excellent tutorial about FLARManager: thank you Lee!

Known Issues

The FLARToolkit library is very CPU intensive! I looked into it and beside the fact that it is poorly engineered, it looks like the blob extraction algorithm is the bottleneck. Sadly, there is not much to do about it...

It was ported from a Java port of a C++ library called ARToolkit. I never read the original C++ code or its Java port, but the AS3 version seems to be just a straight forward rewriting without much thinking. AS3 native types are totally ignored, if not re-implemented, and the design clearly perspires C++ (and not only because the original code is embeded in comments all over the place...). FLARManager apparently leverages the lack of a proper API.

To put it in a nutshell, FLARToolkit is an impressive tool but it needs a big cleanup!

Credits

• Framerate counter by Mr. Doob
• Augmented reality using FLARToolkit by Saqoosha

Details, pictures and a demo application right after the jump...

MD2 file format

Quake 2 uses the MD2 file format  to store animated 3D models. It is a quite simple file format: each MD2 file stores a single mesh and its animations. Each animation is just copy of the original mesh with different positions for each vertex. The number of vertices and the indices are the same for each frame. Therefore, the MD2 file format is easy to adapt to the Flash Platform because no complicated shaders/bones are required to animate the model: you just have to display the right vertices!

But it is not THAT simple. Each animation is actually subdivided into frames (vertices really). Because storing a fluid animation would take a lot of frames (and a lot of memory), only a few keyframes are stored. When rendering, one must compute the interpolation coefficient between the current and the next keyframe and use it to get the interpolated vertices. The following code does it all:

var frame1	: Vector.<Number>	= _currentFrame.vertices;
var frame2	: Vector.<Number>	= _nextFrame.vertices;
var c2		: Number		= _frame - int(_frame);
var c1		: Number		= 1 - c2;
var length	: Number		= frame1.length;
 
for (var i : uint = 0; i < length; ++i)
	_vertices[i] = frame1[i] * c1 + frame2[i] * c2;

... and here is the demo application :

Flash 10 MD2 loader

What it does...

• load Quake 2's MD2 files
• basic z-sorting using faces barycenter
• add a "frame" property to the mesh to handle animations
• interpolate and update vertices only when necessary
• wireframe material (press "w")
• "free chase" camera (mouse click + drag'n'drop to turn around the model)

What it doesn't do...

• use BSP trees to handle z-sorting (1 BSP tree / frame takes a lot of time and memory...)
• loop through a specific animation
• read Quake 2's animation files

Known issue(s)

None. If you find one please let me know! Performance seems to be very good actually...

There are so many reasons why loading Quake 2 files inside the Flash player would be considered as just "impossible". Performance would be the first and main one. One would just consider that Flash is not fast enough to display complex 3D graphics, not even those of a game as old as Quake 2 (published in 1996!). But using Flash 10 latests features such as the Vector and GraphicsTrianglePath classes, it is actually quite doable!

Details, pictures and a demo application right after the jump!

Quake 2's maps file format

But the really big advantage here is inside the Quake 2 maps file format itself. Each map is a *.BSP file and the BSP file format might be the best ratio in terms of ease of implementation against performance optimization. As you might have guessed, *.BSP files actually store a pre-compiled Binary Space Partitionning tree of the map itself. For those who don't already know about BSP trees, just imagine that it is actually the best known method to handle a polygon-perfect z-sorting in Flash when dealing with 3D. But computing a good BSP tree of a large and complex 3D model such as a map takes a lot of time... that is why having a pre-compiled and ready to load BSP tree is just great!

You can find a lot of good documents on the Quake 2 BSP file format. The one hosted on flipcode is quite complete and easy to understand. It is actually the one I used to create my BSP loader. Reading the file formats specifications shows that *.BSP files also include "visibility information". This data is also very important because it makes the user able to know which area of the map is visible from any other area. This way, only the areas visible from the area where the player stands have to be displayed which is an incredible performance boost!

The following application is a proof of concept of a Flash powered Quake 2 *.BSP loader :

Click on the picture to launch the demo application.

Here are some screenshots of the same application loading much more complex maps (those screenshots have been made with the Flash debugger which explains the higher memory consumption) :

actcity2

aqnitro

What it does...

• load Quake 2's *.BSP files
• build the corresponding geometry (vertices, indices and UV data) using Flash 10 native types such as Vector and GraphicsTrianglePath
• frustum culling and clipping using bounding boxes and bounding spheres
• mouse controlled "free chase" camera (click + drag'n'drop to look around, mouse wheel to zoom in/out)

What's next...

• use the embed BSP data to handle z-sorting
• use visibility information to display only visible clusters
• load WAL textures (Quake 1 and 2 textures file format, already done just have to post it )  and use them properly to display the map
• load MD2 animated 3D models (again, already done and just have to post it!)

Known issue(s)

If you zoom enough to place the camera "inside" the map geometry, you will notice a severe framerate drop down. It is very logical since frustum clipping will have to check and eventually clip each polygon of the geometry against the relevant view frustum planes. Because visibility information tells us what is actually visible or not, handling it properly should offer a huge performance boost: instead of having to clip the whole geometry, only the visible areas will be treated.