Augmented Reality in Flash 10
For those who have never heard of "augmented reality" (AR), here is Wikipedia's definition:
Augmented reality (AR) is a term for a live direct or indirect view of a physical real-world environment whose elements are merged with (or augmented by) virtual computer-generated imagery - creating a mixed reality.
Sounds a bit blury? Well... I'll try to make it clearer with a demo...
Demo
First, you will have to print a little black and white "marker". The AR software scans the webcam picture and look for this very marker. When it is found, its 3D position, rotation and scale are computed and used to embed a 3D object. You can found the marker here and it looks like this:
FLARToolkit Marker
Then, launch the following application and place the printed marker in front of your camera:
FLARToolkit demo application (webcam required)
What it does...
- Load a MD2 file and a JPEG texture
- Loop through the animation of the model
- Define multiple materials (press "w" to switch)
- Use the FLARToolkit library for the AR
- Translate FLARToolkit matrices to fit the 3D API
How it works
- The viewport and a dummy camera are initialized using the camera parameters and the size of the capture
- The webcam image is binarized using a threshold filter: aeras that are darker than a threshold value end up completely white on a black background. Areas that are too small or too big are rejected.
- Each remaining area is detected and its bounding box and position are computed
- For each area that "looks like" a square, the data of what should be the pattern is extracted
- This data is then compared to the actual pattern
- If the match confidence is above a certain value (80% is this case), a matrix that contains the position/rotation/scale of the center of the marker is computed
- This matrix is converted into a native Matrix3D object that can be used as the world transform for the 3D objects we want to display
What it does not do...
- Use FLARManager to handle the AR. Lee "The Flash Blog" Brimelow juste made this excellent tutorial about FLARManager: thank you Lee!
Known Issues
The FLARToolkit library is very CPU intensive! I looked into it and beside the fact that it is poorly engineered, it looks like the blob extraction algorithm is the bottleneck. Sadly, there is not much to do about it...
It was ported from a Java port of a C++ library called ARToolkit. I never read the original C++ code or its Java port, but the AS3 version seems to be just a straight forward rewriting without much thinking. AS3 native types are totally ignored, if not re-implemented, and the design clearly perspires C++ (and not only because the original code is embeded in comments all over the place...). FLARManager apparently leverages the lack of a proper API.
To put it in a nutshell, FLARToolkit is an impressive tool but it needs a big cleanup!
Credits
- Framerate counter by Mr. Doob
- Augmented reality using FLARToolkit by Saqoosha
October 21st, 2009 - 09:12
I looked into it and beside the fact that it is poorly engineered, it looks like the blob extraction algorithm is the bottleneck. Sadly, there is not much to do about it…
I think it`s all because of comments, in FLARToolkit all comments written in Japanese ですね ; )？
October 21st, 2009 - 09:53
Comments should never contain code. If you add commented code to your code, it’s an epic fail to me…
But that’s not the real problem. The main problem is how the whole thing was designed and architectured. A few things coded when some native functions/classes already exist. Plus the final API (and I’m not event sure that’s a good name in this case…) is not user friendly.
FLARManager seems to leverage the performance problem while providing a real Flash oriented API… and that just shouldn’t be possible! I’ll check it out to be 100% sure.
October 22nd, 2009 - 00:41
About architecture, I completely agree with you.
Right now I in middle of developing an color tracking / shape recognize mechanic.
So I actively read a lot of different code sources including FLARtoolkit.
IMHO Sakooshi`s port written too good / right / canonical / scholar for really effective work.
Code looks like not optimized for AS3 at all.
I think main and primitive task of such toolkit (needed in 80% of cases) – track a marker & build an camera space. Such code must be size of 300-700 lines.
But not this huge pile of interfaces and strange classes which can bee seen in FLARToolkit.
I think it not optimized till this day because of nonhummanoid – japajavarobo style of code.
I try to make an realization of primary task of AR mentioned above in simple short and effective code.
So which me a luck. If I return alive, I will publish the source code
乾杯!
October 22nd, 2009 - 10:15
Saw it last Saturday and It is pretty awesome, even if it is quite slow.
April 1st, 2010 - 17:32
what is the secret? your movie has low framerate, but it is working perfect! no model disappear, no “drunk dancing”. “excellent tutorial” you were talking about is realy good: higher framerate, easy to understand, but it has all the problems I wrote. can you upload your source code or explain key details of “how to make good AR application”?