Devices Synchronization for modeling « 3D plane »

Rémi Devinant DII5 / Devices synchronization for modeling 3D plane 1

Devices Synchronization for modeling « 3D plane »


Summary

Introduction Context Problem Modeling Devices Synchronization Computer vision Conclusion


Introduction More and more augmented reality applications Development of more performant technologies

Best performances Reliability tools for users


Context

Smartphones More than 100 applications 2 known techniques▪ Geolocalization and camera▪ Camera, gyro, accelerometer▪ Embedded systems constraints

« EyeTape » prototype Head up display Display information superimposed on vision


Context

The project Based on a CCD

camera▪ Taking frames in

video▪ 2 kinds of

information to▪ Static (buildings, urban

environment…)▪ Dynamic (people,

cars…)


Context Ubiquitous Computing

Mobile systems around users Interacts with the environment Devices used in parallel Mobile system▪ Video treatment▪ 2 informations flow

Embedded systems problematic Light mobile system Light CPU power Memory constraints


Context Smartphone (iPhone 4) composed of

CCD 5mPixels (video 720p à 30fps) Gyro L3G4200D Digital 3-axis Accelerometer ST Micro LIS331DLH 3-axis CPUApple A4 APL0398 (ARM Cortex A8) DMA Memory RAM 512Mo DRAM Samsung


Context

Devices synchronisation Apple iOS Apple iPhone OS Based on a BSD Kernel Mac OS X with cellphone services


Context

Video frames overlap Video’s framerate preset Purpose Reduce or increase framerate Original 30fps Smartphone motion in space

Slow motion : video frames can overlap significantly need for suppressing useless ones

Need for fusioning devices datas Allow selection in video frames


Context

2 known methods Hardware▪ Can exist if it’s built-in▪ Smartphones dont' have this characteristic

Software▪ Can be implemented in the OS▪ Require an personalized algorithm


Summary



Problem modeling

Each devices have different time delays In retrieving datas In transmitting datas

Require modeling each delays representing the time between the datas are required and the time where they are in memory


Problem modeling

Basic system architectureGyroscope

Accéléromètre

Capteur CCD

Device Controller gyro

Device Controller accéléromètre

Device Controller CCD

Buffer mémoire DMA


Problem modeling

Accelerometer 3 datas send into 1 block 3 axis acceleration sensor


Problem modeling

Accéléromètre

DeviceController Mémoir

eCPU

Data BusAdresses BusControl Bus


Problem modeling Accelerometer Retrieving datas delay

Devices bus output: fbus Datas volume: Vdata Datas acquisition delay: ∆tacqacc Bus acquisition delay (if busy): delay Writing in memory delay depending on datas

volume : ∆twrite

∆tacc = ∆tacq + (fbus / Vdata) + ∆twrite + delay


Problem modeling

Gyro 3 datas send into 1 block 3 axis rotations


Problem modeling

Gyroscope

DeviceController Mémoir

eCPU



Problem modeling Gyro Retrieving datas delay


volume : ∆twrite



Problem modeling

CCD Camera Treatment of frames

before they are placed in memory

sRGB colors 720p Red/ Green/ Blue Heavy data volume DMA


Problem modeling

CCD

DeviceController

MémoireCPU DMA



Problem modeling CCD Camera Retrieving datas delay


volume : ∆twrite



Summary



Devices synchronization

T0

T1 ∆t

Recording at 30fps

Windows of N scan in memory for finding frame

Start memory scan window



Frame Spotted at ti

- ti : date of scan

- ∆t : retrieveing image delay

- ti - ∆t = real date of the frame capture

∆t



In this window If a frame is spotted at pi S∆ = pi – (N - 1) / 2

Next window start at T1 + S∆

Window placed every time near the last image was spotted Real time and embedded constraints

context



When the window is started, creation of coordinate table Accelerometer▪ ta[] = tspotted - ∆tacc

Gyro▪ tg[] = tspotted - ∆tgyr



Coordinate are chosed tCoordonnees = tImgSpotted - ∆tImage We try to keep the nearest coordinate


Summary



Computer Vision

A frame = spatial coordinates Smartphone’s spatial localization

Spatial localization= frame plane Creates frame 3D plane


Computer Vision


Computer Vision

Modeling 3D plane

Devices Synchronization for modeling « 3D plane »

Documents

d plane2introductionmore

d plane9context2

d plane15acclr

mpixels video

ccd camerataking frames

different time delaysin

displaydisplay information

fpssmartphone motion