Augmented reality (AR) is a term for a live direct or indirect view of a physical real-world environment whose elements are merged with (or augmented by) virtual computer-generated imagery - creating a mixed reality.

Sounds a bit blury? Well... I'll try to make it clearer with a demo...

Demo

First, you will have to print a little black and white "marker". The AR software scans the webcam picture and look for this very marker. When it is found, its 3D position, rotation and scale are computed and used to embed a 3D object. You can found the marker here and it looks like this:

FLARToolkit Marker

Then, launch the following application and place the printed marker in front of your camera:

FLARToolkit demo application (webcam required)

What it does...

• Load a MD2 file and a JPEG texture
• Loop through the animation of the model
• Define multiple materials (press "w" to switch)
• Use the FLARToolkit library for the AR
• Translate FLARToolkit matrices to fit the 3D API

How it works

• The viewport and a dummy camera are initialized using the camera parameters and the size of the capture
• The webcam image is binarized using a threshold filter: aeras that are darker than a threshold value end up completely white on a black background. Areas that are too small or too big are rejected.
• Each remaining area is detected and its bounding box and position are computed
• For each area that "looks like" a square, the data of what should be the pattern is extracted
• This data is then compared to the actual pattern
• If the match confidence is above a certain value (80% is this case), a matrix that contains the position/rotation/scale of the center of the marker is computed
• This matrix is converted into a native Matrix3D object that can be used as the world transform for the 3D objects we want to display

What it does not do...

• Use FLARManager to handle the AR. Lee "The Flash Blog" Brimelow juste made this excellent tutorial about FLARManager: thank you Lee!

Known Issues

The FLARToolkit library is very CPU intensive! I looked into it and beside the fact that it is poorly engineered, it looks like the blob extraction algorithm is the bottleneck. Sadly, there is not much to do about it...

It was ported from a Java port of a C++ library called ARToolkit. I never read the original C++ code or its Java port, but the AS3 version seems to be just a straight forward rewriting without much thinking. AS3 native types are totally ignored, if not re-implemented, and the design clearly perspires C++ (and not only because the original code is embeded in comments all over the place...). FLARManager apparently leverages the lack of a proper API.

To put it in a nutshell, FLARToolkit is an impressive tool but it needs a big cleanup!

Credits

• Framerate counter by Mr. Doob
• Augmented reality using FLARToolkit by Saqoosha

It's been a while since the last post. But I'm back and I decided to release the first DirectFlex demo ever

This demo is quite simple and features a 3D model extracted from the famous Half Life 1 add on Counter-Strike 1.6. The model is made of "only" 740 polygons, which is not much compared to what DirectFlex can do. Still, being able to display Half Life 1 models inside of Flash with a decent framerate brings hope for future developments. Anyway it is a good proof that DirectFlex is (way) faster than any other Flash 3D "engine" out there... but hey! live demo first and then technical stuff... (Flash 10 required!)

First Person Shooter Camera

This demo features one of the camera classes provided by DirectFlex: FPSCamera. This camera mimics the behavior of a any "classic" doom-like FPS game. Simple use the arrows/AWSD to walk forward/backward/strafe. You can also use the mouse to look around. Nothing fancy except the camera class itself which is really lightweight and easy to overload: I actually created a GhostCamera class based on the FPSCamera class by just adding 2 lines of code... All the camera classes are based on the same AbstractCamera class which makes it possible and really easy to switch from one camera type to another in real time in your application. I will release another demo to show how simple the code actually is.

3DS Loader

A good opportunity to show the 3DS loader in action. Loading and parsing are obviously really fast. The 3DS file is embed inside the SWF file which of course makes it even faster. The 3DS loader was built using this documentation.

Texture Mapping

The texture I used is the original one. I just resized it to make the final SWF file lighter. Nothing crazy about it except that it looks perfectly fine without cutting down the framerate.

Embed Assets

The texture and the 3D model files are embed inside the final SWF file. The mesh and texture class provided by DirectFlex can load such embed assets natively. It is very useful when you just want to redistribute your application without taking care of deploying the assets on a web server and dealing with security issues.

Real Time Lighting

As you can see, the back of the model is completely black. It does not mean that the texture is actually black Each polygon is skinned then lighted using a "fake" Phong algorithm. It is a very first draw and a lot of thinking has to be done to make it actually user friendly. Still, because the light and the model are not moving the "real time" part is not really obvious here... I'll try to make it more fancy in the next demo.

About performances

The framerate counter indicates 20-25 frames per second. The application does not force the framerate so the Flash Player runs it at the default framerate (which is supposed to be 24 frames per second). Then, a 20 to 25 frames per second framerate is actually as smooth as it is supposed to be It runs perfectly on my Atom 1.6Ghz processor so I guess any decent computer should run it without any performance issue of any kind.

Known issues

The mouse is not handled properly and its sensitivity will depend on the framerate of the application. If your framerate is to high, looking around might be tricky. I know how to fix that but I'm just to lazy to do it for this first demo

Z-sorting is not (yet) functional and some glitches might appear. I chose the 3D model carefully so that no major visual defects would occur.

Clipping and frustrum culling are not fully implemented. Major artifacts might appear when the model is rendered is the "back" of the camera.

What's next ?

A MD5 (Doom 3 models) loader is on its way and I will certainly try to load Half Life 2 models too. I am currently working with Alchemy v0.5a to build a fast C++ implementation of BSP trees to implement an efficient and reliable z-sorting algorithm. I will definitely post about Alchemy and my whole C/C++ to AS3 work. I will also release a very cool heightmap demo soon, with some source code to show how user friendly DirectFlex actually is. Other cool demos will follow... Finally, the official DirectFlex blog will be launched soon!

Do not hesitate to post any ideas and comments about how smooth the demo runs on your computer for example.