SIXTH SENSE TECHNOLOGY 1. INTRODUCTION Sixth Sense Technology is a mini-projector coupled with a camera and a cell phone which acts as the computer and connected to the Cloud, all the information stored on the web. Sixth Sense can also obey hand gestures. The camera recognizes objects around a person instantly, with the micro-projector overlaying the information on any surface, including the object itself or hand. Also can access or manipulate the information using fingers. Sixth Sense is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information. By using a camera and a tiny projector mounted in a pendant like wearable device. Sixth Sense sees what you see and visually augments any surfaces or objects we are interacting with. It projects information onto surfaces, walls, and physical objects around us, and lets us interact with the projected information through natural hand gestures, arm movements, or our interaction with the object itself. Sixth Sense attempts to free information from its confines by seamlessly integrating it with reality , and thus making the entire world your computer.[1] Akshay .N .Mahajan(120573116004) Page 1
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
SIXTH SENSE TECHNOLOGY
1. INTRODUCTION
Sixth Sense Technology is a mini-projector coupled with a camera and a cell phone
which acts as the computer and connected to the Cloud, all the information stored on
the web. Sixth Sense can also obey hand gestures. The camera recognizes objects
around a person instantly, with the micro-projector overlaying the information on any
surface, including the object itself or hand. Also can access or manipulate the
information using fingers.
Sixth Sense is a wearable gestural interface that augments the physical world around
us with digital information and lets us use natural hand gestures to interact with that
information. By using a camera and a tiny projector mounted in a pendant like
wearable device. Sixth Sense sees what you see and visually augments any surfaces or
objects we are interacting with. It projects information onto surfaces, walls, and
physical objects around us, and lets us interact with the projected information through
natural hand gestures, arm movements, or our interaction with the object itself. Sixth
Sense attempts to free information from its confines by seamlessly integrating it with
reality , and thus making the entire world your computer.[1]
Fig 1. Parts used in Sixth Sense Technology [1]
Akshay .N .Mahajan(120573116004) Page 1
SIXTH SENSE TECHNOLOGY
2. HOW SIXTH SENSE TECHNOLOGY WORKS
The technology in itself is nothing more than the combination of some
stunning technologies, but the idea of combining those technologies is really
great.
The technology is mainly based on hand gesture recognition, image capturing,
processing, and manipulation, etc.
The camera is used to recognize and track user’s hand gestures and physical
objects using computer-vision based techniques, while the projector is used to
project visual information on walls or on any physical thing around us. Other
hardware includes mirror and colored caps to be used for fingers
The projector: projects visual information, enabling surfaces, walls and
physical objects around the wearer to be used as interfaces;
The camera and hands: recognizes and tracks the user's hand gestures and
physical objects using computer-vision based techniques.
The software program: processes the video stream data captured by the camera
and tracks the locations of the colored markers (visual tracking of colored
fingers) at the tip of the user’s fingers using simple computer-vision
techniques. The movements and arrangements of these fiducials are
interpreted into gestures that act as interaction instructions for the projected
application interfaces.
The maximum number of tracked fingers is only constrained by the number of
unique fiducials, thus Sixth Sense also supports multi-touch and multi-user
interaction.
3. ARCHITECTURE OF SIXTH SENSE TECHNOLOGY [3]
Akshay .N .Mahajan(120573116004) Page 2
SIXTH SENSE TECHNOLOGY
Fig 2: Architecture of Sixth Sense Technology[3]
Sixth Sense, a tool that connects the physical world with the world of data.
Digital prototype called the “Sixth Sense”, which wearable hi-tech devices
that can turn anything into a touch screen. Your wrist can become a watch and
your palm a phone. The device is combination of a wearable camera, a
projector and a mirror where the camera recognizes movements of the fingers
of your hand and translates those into commands.
The hardware components are coupled in a pendant like wearable device.
Both the projector and the camera are connected to the mobile computing
device in the user’s pocket. The projector projects visual information enabling
surfaces, walls and physical objects around us to be used as interfaces; while
the camera recognizes and tracks user’s hand gestures and physical objects
using computer vision based techniques. The software program processes the
video stream data captured by the camera and tracks the locations of the
colored markers (visual tracking fiducials) at the tip of the user’s fingers using
simple computer vision techniques.
Akshay .N .Mahajan(120573116004) Page 3
SIXTH SENSE TECHNOLOGY
The movements and arrangements of these fiducials are interpreted into
gestures that act as interaction instructions for the projected application
interfaces. The maximum number of tracked fingers is only constrained by the
number of unique fiducials, thus Sixth Sense also supports multi touch and
multi user interaction.
3.1 POCKET PROJECTOR [3]
Pocket projector is an emerging technology that applies the use of a projector in a
handheld device. It is also know as Handheld projector, pocket projector or mobile
projector or Pico projector. It is a response to the emergence of compact portable
devices such as mobile phones, personal digital assistants and digital cameras. Which
have sufficient storage capacity to handle presentation materials but little space to
accommodate an attached display screen. Handheld projectors involve miniaturized
hardware and software that can project digital images onto any nearby viewing
surface, such as a wall. This entire system is compacted into one very tiny chip. An
important design characteristic of a handheld projector is the ability to project a clear
image, regardless of the physical characteristics of the viewing surface.
3.2 PARTS OF POCKET PROJECTOR [3]
The system comprises four main parts:
The electronics
The laser light sources
The combiner optic, and the
Scanning mirrors.
3.3 THE WORKING OF SIXTH SENSE DEVICE [4]
Akshay .N .Mahajan(120573116004) Page 4
SIXTH SENSE TECHNOLOGY
Fig.3 Working Of Sixth Sense [4]
First, the electronics system turns the image into an electronic signal.
Next the electronic signals drive laser light sources with different colors and
intensities down different paths.
In the combiner optic the different light paths are combined into one path
demonstrating a pallet of colors.
Finally, the mirrors copy the image pixel by pixel and can then project the
image.
3.4 APPLICATIONS [8]
Handheld projector can be used for similar applications than small conventional
projectors. Study applications that are specifically designed for handheld projectors
often using prototypes of mobile phones with an integrated projector.
Gaming: Handheld projectors, in particular projector phones, could offer new
possibilities for mobile gaming as demonstrated by the adaptation of the
Playstation3 game Little Big Planet.
Akshay .N .Mahajan(120573116004) Page 5
SIXTH SENSE TECHNOLOGY
Hand gesture recognition: Size reduction of mobile devices is often limited
by the size of the used display. Apart from the display a complete phone can
be, for example, integrated in a headset. It has been demonstrated that Pico
projectors integrated in headsets could be used as interaction devices e.g.
using additional hand and finger tracking.
Picture browsing: Projector phones allow photographs to be shared with a
larger audience. The projected display allowed viewing of pictures by all
present which is currently not possible using a single mobile device. [4]
3.5 MOBILE COMPUTING DEVICES
A mobile device (also known as cell phone device, handheld device, handheld
computer, "Palmtop" or simply handheld) is a pocket-sized computing device,
typically having a display screen with touch input or a miniature keyboard. In the case
of the personal digital assistant (PDA) the input and output are combined into a touch-
screen interface. Smart phones and PDAs are popular amongst those who require the
assistance and convenience of a conventional computer, in environments where
carrying one would not be practical.[4]
3.5.1 TYPES OF MOBILE DEVICES [4]
Mobile devices have been designed for many applications and include:
Mobile computers.
Graphing calculator.
Handheld game consoles.
Digital still camera (DSC).
Digital video camera (DVC or digital camcorder).
Portable media player.
E-book reader.
Mobile phone.
Personal navigation devices(PNDs)
Akshay .N .Mahajan(120573116004) Page 6
SIXTH SENSE TECHNOLOGY
3.5.2 USAGE
Handheld devices have become use in mobile field management situations to record
information in the field. They are used to achieve a variety of tasks for increasing
efficiency that include digitizing notes , sending and receiving invoices, assets
management, recording signatures, managing parts and scanning barcodes. Handheld
computers used at work have molded over time into a variety of form including smart
phones on the low end, handheld PDAs, Ultra Mobile PCs, tablet PCs .[4]
4. WEARABLE GESTURAL INTERFACE [5]
Information is traditionally confined to paper or digitally to a screen. In this paper,
we introduce WUW, a wearable gestural interface, which attempts to bring
information out into the tangible world. By using a tiny projector and a camera
mounted on a hat or coupled in a pendant like wearable device, WUW sees what the
user sees and visually augments surfaces or physical objects the user is interacting
with.WUW projects information onto surfaces, walls, and physical objects around us,
and lets the user interact with the projected information through natural hand gestures,
arm movements, or interaction with the object itself.
A wearable device with a projector that allows to augment reality with a pendant
projector. The device (via the web cam and coded finger-gloves) recognizes users
gestures while searching for information, taking photos, making a call and presents it
back to the user projected at any display.
The recent advent of novel sensing and display technologies has encouraged the
development of a variety of multi-touch and gesture based interactive systems. Such
systems have moved computing onto surfaces such as tables and walls and brought
the input and projection of digital interfaces into one-to-one correspondence such that
the user may interact directly with information using touch and natural hand gestures.
A parallel trend, the miniaturization of mobile computing devices permit “anywhere”
access to information, so we are always connected to the digital world. Unfortunately,
most gestural and multi-touch based interactive systems are not mobile and small
Akshay .N .Mahajan(120573116004) Page 7
SIXTH SENSE TECHNOLOGY
mobile devices fail to provide the intuitive experience of full-sized gestural systems.
Moreover, information still resides on screens or dedicated projection surfaces. There
is no link between our interaction with these digital devices and interaction with the
physical world around us.
While it is impractical to modify all physical objects and surfaces into interactive
touch-screens, and to turn everything into network enabled devices, it is possible to
augment environment around us with visual information. When digital information is
projected onto physical objects, instinctively it makes sense to interact with it in
similar patterns as with physical objects: through hand gestures.
In this paper, we present WUW, a computer-vision based wearable and gestural
information interface that augments the physical world around us with digital
information and proposes natural hand gestures as the mechanism to interact with that
information. It is not the aim of this research to present an alternative to multi-touch
or gesture recognition technology, but rather to explore the novel free-hand gestural
interaction that WUW proposes.
4.1 What is WUW? [5]
WUW is a wearable gestural interface. It consists of a camera and a small projector
mounted on a hat or coupled in a pendant like mobile wearable device. The camera
sees what the user sees and the projector visually augments surfaces or physical
objects that the user is interacting with. WUW projects information onto the surfaces,
walls, and physical objects around the user, and lets the user interact with the
projected information through natural hand gestures, arm movements, or direct
manipulation of the object itself.
The tiny projector is connected to a laptop or mobile device and projects visual
information enabling surfaces, walls and physical objects around us to be used as
interfaces; while the camera tracks user hand gestures using simple computer vision
based techniques. The current WUW prototype implements several applications that
demonstrate the usefulness, viability and flexibility of the system. The map
application(see Fig.3) lets the user navigate a map displayed on a nearby surface
Akshay .N .Mahajan(120573116004) Page 8
SIXTH SENSE TECHNOLOGY
using familiar hand gestures, letting the user zoom in, zoom out or pan using intuitive
hand movements. The drawing application (see Fig.4) lets the user draw on any
surface by tracking the fingertip movements of the user’s index finger.
The WUW prototype also implements a gestural camera that takes photos of the scene
the user is looking at by detecting the ‘framing’ gesture (see Fig.5). The user can stop
by any surface or wall and flick through the photos he/she has taken. The WUW
system also augments physical objects the user is interacting with by projecting more
information about these objects projected on them. For example, a newspaper can
show live video news or dynamic information can be provided on a regular piece of
paper (see Fig.6).The gesture of drawing a circle on the user’s wrist projects an analog
watch (see Fig.4). The WUW system also supports multiple users collaborating and
interacting with the projected digital information using their hand movements and
gestures. [5]
Fig.4: Map Application [5]
Akshay .N .Mahajan(120573116004) Page 9
SIXTH SENSE TECHNOLOGY
Fig.5: Drawing Application [5]
Fig.6: Framing Gesture [19]
Fig.7: Newspaper [5]
Akshay .N .Mahajan(120573116004) Page 10
SIXTH SENSE TECHNOLOGY
Fig.8:Analog watch [5]
4.2 WUW prototype [5]
The WUW prototype is comprised of three main hardware components: a pocket
projector (3MMPro110), a camera (Logitech Quick Cam) and a laptop computer. The
software for the WUW prototype is developed on a Microsoft Windows platform
using C#. Both the projector and the camera are mounted on a hat and connected to
the laptop computer in the user’s back pack. The prototype system uses plastic
colored-markers (e.g. caps of whiteboard markers) as the visual tracking fiducials.
Fig.9: WUW prototype system[5]
Akshay .N .Mahajan(120573116004) Page 11
SIXTH SENSE TECHNOLOGY
The movements and arrangements of these color markers are interpreted into gestures
that act as interaction mechanism for the application interfaces projected by the
projector. The current prototype only tracks 4 fingers – index fingers and thumbs.
This is largely due to the relative importance of the index finger and thumb in natural
hand gestures, tracking these WUW is able to recognize a wide variety of gestures
and postures. The maximum number of tracked fingers is only constrained by the
number of unique fiducials, thus the WUW also supports multi-user interaction.
The prototype system also implements a few object augmentation applications where
the camera recognizes and tracks physical objects such as a newspaper, a coffee cup,
or a book and the projector projects relevant visual information on them. WUW uses
computer-vision techniques in order to track objects and align projected visual
information by matching pre-printed static color markers (or patterns) on these objects
with the markers projected by the projector.
4.3 CHALLANGES
There are many challenges associated with the accuracy and usefulness of gesture
recognition software. For image-based gesture recognition there are limitations on the
equipment used and image noise. Images or video may not be under consistent
lighting, or in the same location. Items in the background or distinct features of the
users may make recognition more difficult. The variety of implementations for image-
based gesture recognition may also cause issue for viability of the technology to
general usage. For example, an algorithm calibrated for one camera may not work for
a different camera. The amount of background noise also causes tracking and
recognition difficulties, especially when occlusions (partial and full) occur.
Furthermore, the distance from the camera, and the camera's resolution and quality,
also cause variations in recognition accuracy. [5]
5. HAND GESTURE RECOGNITION [6]
Hand gesture recognition is a topic in computer science and language technology with
the goal of interpreting human gestures via mathematical algorithms. Gestures can