Augmented reality (AR) is a term for a live direct or indirect view of a physical real-world environment whose elements are merged with (or augmented by) virtual computer-generated imagery - creating a mixed reality.

Sounds a bit blury? Well... I'll try to make it clearer with a demo...

Demo

First, you will have to print a little black and white "marker". The AR software scans the webcam picture and look for this very marker. When it is found, its 3D position, rotation and scale are computed and used to embed a 3D object. You can found the marker here and it looks like this:

FLARToolkit Marker

Then, launch the following application and place the printed marker in front of your camera:

FLARToolkit demo application (webcam required)

What it does...

• Load a MD2 file and a JPEG texture
• Loop through the animation of the model
• Define multiple materials (press "w" to switch)
• Use the FLARToolkit library for the AR
• Translate FLARToolkit matrices to fit the 3D API

How it works

• The viewport and a dummy camera are initialized using the camera parameters and the size of the capture
• The webcam image is binarized using a threshold filter: aeras that are darker than a threshold value end up completely white on a black background. Areas that are too small or too big are rejected.
• Each remaining area is detected and its bounding box and position are computed
• For each area that "looks like" a square, the data of what should be the pattern is extracted
• This data is then compared to the actual pattern
• If the match confidence is above a certain value (80% is this case), a matrix that contains the position/rotation/scale of the center of the marker is computed
• This matrix is converted into a native Matrix3D object that can be used as the world transform for the 3D objects we want to display

What it does not do...

• Use FLARManager to handle the AR. Lee "The Flash Blog" Brimelow juste made this excellent tutorial about FLARManager: thank you Lee!

Known Issues

The FLARToolkit library is very CPU intensive! I looked into it and beside the fact that it is poorly engineered, it looks like the blob extraction algorithm is the bottleneck. Sadly, there is not much to do about it...

It was ported from a Java port of a C++ library called ARToolkit. I never read the original C++ code or its Java port, but the AS3 version seems to be just a straight forward rewriting without much thinking. AS3 native types are totally ignored, if not re-implemented, and the design clearly perspires C++ (and not only because the original code is embeded in comments all over the place...). FLARManager apparently leverages the lack of a proper API.

To put it in a nutshell, FLARToolkit is an impressive tool but it needs a big cleanup!

Credits

• Framerate counter by Mr. Doob
• Augmented reality using FLARToolkit by Saqoosha