Sixth Sense Technology

1

Outline

Introduction

Components

Technology Behind Sixth Sense.

How It Works?

Applications

Cost

Advantages & Disadvantages

Conclusion And Future2

“Sixth Sense is a wearable gestural interface device that

augments the physical world with digital information and lets

people use natural hand gestures to interact with that

information. ”

It was developed by PRANAV MISTRY,

3

4

A pocket projector,

A mirror,

A camera.

Mobile component

Colored markers5

The Projector: projects visual information, enabling surfaces, walls and physical objects around the wearer to be used as interfaces;

The Camera And Hands: recognizes and tracks the user's hand gestures and physical objects using computer-vision based techniques.

The Software Program: processes the video stream data took by the camera and tracks the locations of the colored markers (visual tracking of colored fingers) at the tip of the user’s fingers using simple computer-vision techniques.

The maximum number of tracked fingers is only constrained by the number of unique fiducials, thus Sixth Sense also supports multi-touch and multi-user interaction.

6

Technology behind sixth sense

Hardware Setup.

Software Setup.

Events & Gestures.

System Architecture.7

Hardware setup

MOBILE

Nokia n95 smart phone is used (running symbian O.S s60

edition).

It has multitasking capability.

Built-in camera provides execution of both:

Gesture tracking engine.

Gesture enabled application.

8

PROJECTOR Pocket projector Pk101 from Optoma is used.

It augments near by surfaces.

It is a LED based projector.

Suitable for mobile usage.

Hardware setup

9

10

Software setup

Applications are implemented using JAVA 2 MICRO

edition.

Computer vision library is written in symbian c++(used in

Gesture tracking).

The software for the sixth sense prototype is developed on a

Microsoft Windows platform using C#, WPF and open CV.

11

The software works on the basis of

computer vision.

A small camera acting as an eye, connecting

us to the world of digital information.

Processing is happening in the mobile phone,

and basically works on computer vision

algorithms.

Approx 50,000 lines of code are used.

12

13

Events & Gestures

Events & Gestures are detected using colored markers

attached to fingers.

The user can zoom in or out by moving his hands/fingers

farther or nearer to each other, respectively

The user can draw on any surfaces using the movement of

the index finger used as a pen.

System also supports freehand gestures (postures). one

example is to touch both the index fingers with the opposing

thumbs, forming a rectangle or framing gesture

14

This gesture activates the photo taking application.

Another example of such gestures is the ‘Namaste’ posture

that lets the user navigate to the home screen.

System lets the user draw icons or symbols in the air using

the movement of the index finger.

15

16

Images Pictures Gestures

Info on surface

captureSend for processing

Reflect on desired surface

Project image

Send Info

17

The hardware that makes Sixth Sense work is a pendant like mobile wearable interface

It has a camera, a mirror and a projector and is connected wirelessly to a Bluetooth smart phone that can slip comfortably into one’s pocket

The camera recognizes individuals, images, pictures, gestures one makes with their hands

Information is sent to the Smartphone for processing

The downward-facing projector projects the output image on to the mirror

Mirror reflects image on to the desired surface

Thus, digital information is freed from its confines and placed in the physical world

18

19

You hold up your left hand, fingers pointing to the right.

The system recognizes that you want to make a call, and

projects a dialing pad onto your fingers. You tap the virtual

keypad with your right hand to dial the call.

20

21

22

23

Real Time Clock

24

Browsing Net

25

The current prototype system costs

approximately $350 to build, mainly due to

the micro-projector. The software may be

available for free on the model of open and

editable freeware

26

Portable

Inexpensive

Multi-sensory

Connectedness between the world and information

It is an open source

Data access directly from machine in real time

27

Software does support the ability to use real time video streams in order to produce augmented reality.

Hardware limitations of the devices, that we currently carry around with us. For example many phones will not allow the external camera feed to be manipulated in real time.

Post processing can occur however.

28

Interactive Advertisements. 3d visualizations. Solar batteries via small solar panel. Camera can act as a third eye for the blind

person

“According to researchers, after 10 years we will be here with the ultimate sixth-sense brain implant.”

29

Sixth Sense recognizes the objects around us, displaying information automatically and letting us to access it in any way we need.

The Sixth Sense prototype implements several applications that demonstrate the usefulness, viability and flexibility of the system.

Allowing us to interact with this information via natural hand gestures.

The potential of becoming the ultimate "transparent" user interface for accessing information about everything around us.

30

31

32