426 lecture 4: AR Developer Tools

COSC 426: Augmented Reality

Mark Billinghurst

[email protected]

August 1st 2012

Lecture 4: AR Developer Tools

Low Level AR Libraries

Low Level AR Libraries   ARToolKit

 Marker based tracking

  FLARToolKit   Flash version of ARToolKit

  SSTT   Simple Spatial Template Tracking

  Opira   Robust Natural Feature Tracking

What is ARToolKit?

  Marker Tracking Library for AR applications   Open Source, Multi-platform (Linux, Windows, MacOS)

  Overlays 3D virtual objects on real markers   Uses single tracking marker   Determines camera pose information (6 DOF)

  ARToolKit Websites http://www.hitl.washington.edu/artoolkit/

http://artoolkit.sourceforge.net/

How it Works

ARToolKit Software   ARToolKit version: 2.65 or later   Currently two license models

  Open Source (GPL): ARToolKit 2.72   Commercial (ARToolWorks): ARToolKit 4.0

  OS: Linux, Windows, MacOS X, iPhone/Android   Programming language: C   Related software

  ARToolKit Professional: Commercial version   ARToolKitPlus: Advanced version   NyARToolkit: Java and C# version   FLARToolKit: Flash version

ARToolKit Family

ARToolKit ARToolKit NFT

ARToolKit (Symbian)

NyToolKit - Java, C#, - Android, WM

JARToolKit (Java)

FLARToolKit (Flash)

FLARManager (Flash)

ARToolKit Plus

ARToolKit Structure

  Three key libraries:   AR32.lib – ARToolKit image processing functions   ARgsub32.lib – ARToolKit graphics functions   ARvideo.lib – DirectShow video capture class

DirectShow

ARvideo.lib

Additional Software

  ARToolKit just provides tracking   For an AR application you’ll need more software

  High level rendering library  Open VRML, Open Inventor, osgART, etc

  Audio Library   Fmod, etc

  Peripheral support

What does ARToolKit Calculate?   Position of makers in the camera coordinates   Pose of markers in the camera coordinates   Output format

  3x4 matrix format to represent the transformation matrix from the marker coordinates to the camera coordinates

Coordinate Systems

From Marker To Camera   Rotation & Translation

TCM : 4x4 transformation matrix from marker coord. to camera coord.

AR Application Development

An ARToolKit Application   Initialization

  Load camera and pattern parameters   Main Loop

  Step1. Image capture and display   Step2. Marker detection   Step3. Marker identification   Step4. Getting pose information   Step5. Object Interactions/Simulation   Step6. Display virtual objects

  End Application   Camera shut down

Sample ARToolKit Applications

 Ex. 1: Simple video display  Ex. 2: Detecting a marker  Ex. 3: Using pattern  Ex. 4: Getting a 3D information  Ex. 5: Virtual object overlay

Ex 1: Simple Video Display

  Program : sample1.c  Key points

 Loop structure  Video image handling  Camera parameter handling  Window setup  Mouse and keyboard handling

Sample1.c – video initialization   Configure the video input

vconf = <video configuration string>

  Start video capture arVideoCapStart();

  In init(), open the video arVideoOpen( vconf ); arVideoInqSize(&xsize, &ysize);

  When finished, close the video path arVideoCapStop(); arVideoClose();

Changing Image Size

  For input capture "vconf = “videoWidth=320,videoHeight=240";

Note – the camera must support this image size

  For display argInit( &cparam, 1.5, 0, 0, 0, 0 );

The second parameter means zoom ratio for display image size related to input image.

Graphics handling: libARgsub

  Set up and clean up the graphics window void argInit( ARParam *cparam, double zoom, int fullFlag, int xwin, int ywin, int hmd_flag );

void argCleanup( void );

cparam: camera parameter zoom: zoom ratio fullFlag: 0: normal, 1: full screen mode Xwin, ywin: create small window for debug hmd_flag: 0: normal, 1: optical see-through mode

Sample1.c Main Function main()!{!!init();!!argMainLoop( mouseEvent, !!

!keyEvent, mainLoop); !}!

Graphics handling: libARgsub   Go into the iterative cycle

void argMainLoop( void (*mouseFunc)(int btn,int state,int x,int y), void (*keyFunc)(unsigned char key, int x, int y), void (*mainFunc)(void)

);

  Swap buffers void argSwapBuffers( void );

Sample1.c - mainLoop Function if( dataPtr = (ARUint8 *) arVideoGetImage()) == NULL ) {

arUtilSleep(2); return;

} argDrawMode2D(); argDispImage(dataPtr, 0, 0 ); arVideoCapNext(); argSwapBuffers();

Image capture: libARvideo

  Return the pointer for captured image   ARUint8 *arVideoGetImage( void );

  Pixel format and byte size are defined in config.h   #define AR_PIX_FORMAT_BGR   #define AR_PIX_SIZE 3

Graphics handling: libARgsub   Set the window for 2D drawing

void argDrawMode2D( void );

  Set the window for 3D drawing void argDrawMode3D( void );

void argDraw3dCamera( int xwin, int ywin );

  Display image void argDispImage( ARUint8 *image, int xwin, int ywin );

Ex. 2: Detecting a Marker   Program : sample2.c   Key points

  Threshold value   Important external variables   arDebug – keep thresholded image   arImage – pointer for thresholded image   arImageProcMode – use 50% image for image

processing -  AR_IMAGE_PROC_IN_FULL -  AR_IMAGE_PROC_IN_HALF

Sample2.c – marker detection

/* detect the markers in the video frame */ if(arDetectMarker(dataPtr, thresh, &marker_info, &marker_num) < 0 ) { cleanup(); exit(0); } for( i = 0; i < marker_num; i++ ) { argDrawSquare(marker_info[i].vertex,0,0); }

Sample2.c – marker_info structure typedef struct {

int area; int id; int dir; double cf; double pos[2]; double line[4][3]; double vertex[4][2];

} ARMarkerInfo; !

Ex. 3: Using a Pattern

  Program : sample3.c   Key points

  Pattern files loading   Structure of marker

information -  Region features -  Pattern Id, direction -  Certainty factor

 Marker identification

Making a pattern template   Use of utility program:

mk_patt.exe   Show the pattern   Put the corner of red line

segments on the left-top vertex of the marker

  Pattern stored as a template in a file

  1:2:1 ratio determines the pattern region used

Sample3.c – Pattern File Loading

int patt_id; char *patt_name = “Data/kanjiPatt”

/* load pattern file */ if(patt_id=arLoadPatt (patt_name) < 0) {

printf ("Pattern file load error !! \n"); exit(0);

}

Checking for known patterns /* check for known patterns */

k = -1; for( i = 0; i < marker_num; i++ ) { if( marker_info[i].id == patt_id) { /* you've found a pattern */ printf("Found pattern: %d \n",patt_id); if( k == -1 ) k = i;

else /* make sure you have the best pattern

(highest confidence factor) */ if( marker_info[k].cf < marker_info[i].cf )

k = i; } }

Ex. 4 – Getting 3D information

 Program : sample4.c  Key points

 Definition of a real marker  Transformation matrix

- Rotation component - Translation component

Sample4.c – Transformation matrix

double marker_center[2] = {0.0, 0.0}; double marker_width = 80.0; double marker_trans[3][4];

arGetTransMat(&marker_info[i], marker_center, marker_width, marker_trans);

Finding the Camera Position This function sets transformation matrix from marker

to camera into marker_trans[3][4]." arGetTransMat(&marker_info[k], marker_center,

marker_width, marker_trans);

You can see the position information in the values of marker_trans[3][4]."" Xpos = marker_trans[0][3]; Ypos = marker_trans[1][3]; Zpos = marker_trans[2][3];

ARToolKit Coordinate Frame

Ex. 5- Virtual Object Display

  Program : sample5.c   Key points

 OpenGL parameter setting  Setup of projection matrix  Setup of modelview matrix

Appending your own OpenGL code

Set the camera parameters to OpenGL Projection matrix. argDrawMode3D(); argDraw3dCamera( 0, 0 );

Set the transformation matrix from the marker to the camera to the OpenGL ModelView matrix. argConvGlpara(marker_trans, gl_para); glMatrixMode(GL_MODELVIEW); glLoadMatrixd( gl_para );

After calling these functions, your OpenGL objects are drawn in the real marker coordinates.

3D CG Model Rendering   ARToolKit does not have a function to handle

3D CG models.   3rd party CG rendering software should be

employed.  OpenVRML  OpenSceneGraph   etc

Loading Multiple Patterns

  Sample File: LoadMulti.c   Uses object.c to load

  Object Structure typedef struct { char name[256]; int id; int visible; double marker_coord[4][2]; double trans[3][4]; double marker_width; double marker_center[2]; } ObjectData_T;

Finding Multiple Transforms

  Create object list ObjectData_T *object;

  Read in objects - in init( ) read_ObjData( char *name, int *objectnum );

  Find Transform – in mainLoop( ) for( i = 0; i < objectnum; i++ ) { ..Check patterns ..Find transforms for each marker }

Drawing Multiple Objects

  Send the object list to the draw function draw( object, objectnum );

  Draw each object individually for( i = 0; i < objectnum; i++ ) { if( object[i].visible == 0 ) continue; argConvGlpara(object[i].trans, gl_para); draw_object( object[i].id, gl_para); }

Limitations of ARToolKit   Partial occlusions cause tracking failure   Affected by lighting and shadows   Tracking range depends on marker size   Performance depends on number of markers

  cf artTag, ARToolKitPlus   Pose accuracy depends on distance to marker   Pose accuracy depends on angle to marker

ARToolKit in the World

  Hundreds of projects   Large research community

FLARToolKit   Flash AS3 Version of the ARToolKit

(was ported from NyARToolkit the Java Version of the ARToolkit)

  enables augmented reality in the Browser   uses Papervision3D for as 3D Engine   available at http://saqoosha.net/   dual license, GPL and commercial license

FLARToolkit

Papervision 3D

Adobe Flash

AR Application Components

private function mainEnter(e:Event):void { /* Capture video frame*/ capture.draw(vid);

/* Detect marker */ if (detector.detectMarkerLite(raster, 80) && detector.getConfidence() > 0.5) { //Get the transfomration matrix for the current marker position detector.getTransformMatrix(trans);

//Translates and rotates the mainContainer so it looks right mainContainer.setTransformMatrix(trans);

//Render the papervision scene renderer.render(); } }

FLARToolKit Examples

Papervision 3D   http://www.papervision3d.org/   Flash-based 3D-Engine   Supports

  import of 3D Models   texturing   animation   scene graph

  alternatives: Away3d, Sandy,…

Source Packages   „Original“ FLARToolkit (Libspark, Saqoosha) (

http://www.libspark.org/svn/as3/FLARToolKit/trunk/ )

  Start-up-guides   Saqoosha (http://saqoosha.net/en/flartoolkit/start-up-guide/ )

 Miko Haapoja (http://www.mikkoh.com/blog/?p=182 )

  „Frameworks“   Squidder MultipleMarker – Example (

http://www.squidder.com/2009/03/06/flar-how-to-multiple-instances-of-multiple-markers/ )

  FLARManager (http://words.transmote.com/wp/flarmanager/ )

Other Languages   NyARToolKit

  http://nyatla.jp/nyartoolkit/wp/   AS3, Java, C#, Processing, Unity, etc

  openFrameworks   http://www.openframeworks.cc/   https://sites.google.com/site/ofauckland/examples/8-artoolkit-example

  Support for other libraries -  Kinect, Audio, Physics, etc

void testApp::update(){ //capture video and detect markers mov.update(); if (mov.isFrameNew()) { img.setFromPixels(mov.getPixels(), ofGetWidth(), ofGetHeight()); gray = img; tracker.setFromPixels(gray.getPixels()); } } //-------------------------------------------------------------- void testApp::draw(){ //draw AR objects ofSetColor(0xffffff); mov.draw(0, 0); for (int i=0; i<tracker.markers.size(); i++) { ARMarkerInfo &m = tracker.markers[i]; tracker.loadMarkerModelViewMatrix(m); ofSetColor(255, 255, 0, 100); ofCircle(0,0,25); ofSetColor(0); ofDrawBitmapString(ofToString(m.id),0,0); } }

AR Tools

experiences

applications

tools

components

Sony CSL © 2004

Building Compelling AR Experiences

Tracking, Display

Authoring

AR Authoring   Software Libraries

  osgART, Studierstube, MXRToolKit   Plug-ins to existing software

 DART (Macromedia Director), mARx, Unity,   Stand Alone

  AMIRE, BuildAR, etc   Next Generation

  iaTAR (Tangible AR)

mARx Plug-in

  3D Studio Max Plug-in   Can model and view AR content at the same time

BuildAR

  http://www.buildar.co.nz/   Stand alone application   Visual interface for AR model viewing application   Enables non-programmers to build AR scenes

Plug-ins for 3D authoring tools

  Unity - Game development tool (www.unity3d.com)   Esperient Creator – General interactive 3D authoring

(www.esperient.com)

Metaio Design   Complete commercial authoring platform

  http://www.metaio.com/software/design/  Offers viewer and editor tools  Drag and drop tools   3D model import

Total Immersion D’Fusion Studio   Complete commercial authoring platform

  http://www.t-immersion.com/  Multi-platform  Markerless tracking   Scripting   Face tracking   Finger tracking   Kinect support

Others   AR-Media

  http://www.inglobetechnologies.com/  Google sketch-up plug-in

  LinceoVR   http://linceovr.seac02.it/   AR/VR authoring package

  Libraries   JARToolKit, MXRToolKit, ARLib, Goblin XNA

OSGART Programming Library   Integration of ARToolKit with a High-Level

Rendering Engine (OpenSceneGraph) OSGART= OpenSceneGraph + ARToolKit

  Supporting Geometric + Photometric Registration

osgART:Features

  C++ (but also Python, Lua, etc).   Multiple Video Input supports:

 Direct (Firewire/USB Camera), Files, Network by ARvideo, PtGrey, CVCam, VideoWrapper, etc.

  Benefits of Open Scene Graph  Rendering Engine, Plug-ins, etc

Advanced Authoring: iaTAR (Lee 2004)

  Immersive AR Authoring   Using real objects to create AR applications

More Information •  Mark Billinghurst

– [email protected] •  Websites

– www.hitlabnz.org