Dieter Schmalsteig (TU Graz) Review of ISMAR research

ISMAR

ISMAR 2016 Research ReviewDieter Schmalstieg

Graz University of Technology, Austria

ISMARAbout me

• Professor at Graz University of Technology• 20 years of research experience in AR

• First collaborative AR system, 1996• First self‐contained AR system on PDA, 2003

• Author of Augmented Reality – Principles and Practice• http://www.augmentedrealitybook.org

• Many industrial collaborations (Qualcomm, DAQRI, …)• Long term supporter of the ISMAR academic conference

Graz through a smartphone, 2008

Dieter Schmalstieg ISMAR 2

ISMARAbout ISMAR• The IEEE International Symposium on Mixed and Augmented Reality

• Premier academic conference on MR/AR• Established in 1998• Partnership with IEEE journal

• Transactions on Visualization and Computer Graphics• Accepts work from all areas of MR/AR

• Displays (head‐mounted, projector‐based, light field…)• Computer vision (tracking, localization, SLAM…)• Computer graphics (x‐ray vision, common illumination, diminished reality…)• Human‐computer interaction (tele‐collaboration, tangible interfaces…)

• http://ismar.vgtc.orgDieter Schmalstieg ISMAR 3

ISMARTrends in Tracking

• Two directions• Better results with standard hardware

• SLAM in particular• New methods using special hardware

• Cluster of scanline cameras• ISMAR2016 best paper award


ISMARTowards KHz 6DOF Visual Tracking using an Egocentric Cluster of Rolling Shutter Camera• A. Bapat, E. Dunn, J.‐M. Frahm (UNC Chapel Hill)• 5x2 GoPros stereo rigs• Treated as line cameras• Line matching at 50K lines/sec


ISMARTrends in Displays

• Head‐mounted displays everywhere

• But many optical limitations not addressed by vendors• Controlling blur and chromatic aberration • ISMAR2016 best paper runner‐up


ISMARGaussian Light Field: Estimation of Viewpoint‐Dependent Blur for Optical See‐Through Head‐Mounted Display• Y. Itoh, T. Amano, D. Iwai, G. Klinker (TU Munich, Wakayama U., Osaka U.)• Model blur in see‐thru HMD • Fit Gauss distribution to point spread function• Index like light field (2D eye position, 2D direction)

Observed PSF Estimated PSFDieter Schmalstieg ISMAR 7

ISMARTrends in Registration

• Big remaining problem: registration in dynamic environments• Moving user

• User in moving vehicle – novel driver assistance systems• Motion capture of user and retargeting to avatar

• Moving objects• Tracking dynamic objects for correct occlusion contours


ISMARReal‐Time Augmented Reality System to See‐Through Cars• F. Rameau, H. Ha, K. Joo, J. Choi, K. Park, I. S. Kweon (KAIST)• Diminish front car by replacing with stereo video from front car







ISMAR

Retargeting Human‐Object Interaction to Virtual Avatars• Y‐ Kim, H. Park, S. Bang, S.‐H. Lee (KAIST)• Extract motion and modify it so it looks plausible on avatar







ISMAREdge Snapping Based Depth Enhancement for Dynamic Occlusion Handling in AR• C. Du, Y.‐L. Chen, M. Ye, L. Ren (U. Kentucky, Bosch)• Pixel‐precise registration between color and depth in RGB‐D video


ISMARTrends in Graphics

• Photorealistic common (virtual+real) illumination• In real time• With minimal hardware (RGB‐D sensor)• With minimal preparation and scene knowledge

• Instant high‐quality reflections• Instant lighting and relighting• Mediated reality


ISMARSingle Camera Image Based Approach for Glossy Reflections

• T. Schwandt, W. Broll (TU Illmenau)• Reflections from a single RGB‐D image• Diffuse environment map from RGB• Raytraced specular from depth map






ISMAR

Instant Mixed Reality Lighting from Casual Scanning• T. Richter‐Trummer, D. Kalkofen, J. Park, D. Schmalstieg (TU Graz, KAIST)• Extract geometry, incident light and materials from RGB‐D camera• Real‐time scene relighting and mediated reality






ISMARRealitySkins: Creating Immersive and Tactile Virtual Environments• L. Shapira, D. Freedman (Google, Microsoft)• Scan a room and generatea “themed” room in VR


ISMARTrends in Interaction

• Real haptics + immersive VR

• Selecting real objects in 3D scans

• Remote exploration of 3D scans


ISMARTactileVR: Integrating Physical Toys to Learn and Play Virtual Reality Experiences• L. Shapira, J. Amores, X. Benavides (Google, MIT)• Games for kids physically interacting with tracked objects







ISMARPixel‐Point‐Volume Segmentation for Object Referencing in Collaborative AR• K.‐C. Lien, B. Nuernberger, T. Höllerer, M. Turk (UCSB)• Identify which object the user points to in RGB‐D scan







ISMARStreaming and Exploration of Dynamically Changing Dense 3D Reconstruction in Immersive VR

• A. Mossel, M. Kroeter (TU Vienna)• Full system for RGB‐D reconstruction streaming and viewing in VR


ISMARConclusions

• ISMAR will be held in Nantes, France, October 2017• Academic paper submission deadline: 15 March 2017• Job fair• Or, get involved as sponsor, exhibitor, workshop organizer…

• Many ways to get connected!• Future will see a strong and continued collaboration with AWE

• http://ismar.vgtc.org
