Can you see me now? Disability assistance in mobile devices Mark Stump Graduate Student Department of Computer Engineering Rochester Institute of Technology.

Can you see me now? Disability assistance in mobile devicesMark StumpGraduate StudentDepartment of Computer EngineeringRochester Institute of Technology

Agenda

•Introduction•Computer Vision •Android•Android/CV Fusion and Challenges•Problem•Preliminary Implementation and Results•Questions/Answers

Computer Vision

•How computers see and perceive the world

•Very diverse field with much up and coming development

Image Source:http://startupblog.files.wordpress.com/2009/10/webcam.jpg

Image Source: http://www.elexp.com/test/ping.jpg

Computer Vision•Subject Areas

▫Feature Detection Edge Detection Blob Detection Corner Detection

▫Recognition Optical Character

Recognition (OCR) Pose Estimation Gesture Recognition

•And many, many moreImage Source: http://stevehanov.ca/blog/index.php?id=62

Face Detection• Recognizes a face in an image

or video feed• Has many applications

▫Digital Cameras Autofocus

▫Security Face as a security Code

▫Advertising▫Energy Saving

Televisions detecting when someone is not looking at the screen and decreasing the brightness Image Source:

http://blog.cordiner.net/wp-content/uploads/2010/02/lenna_output.jpg

Face Detection

Image Source: http://cdn.thefrisky.com/images/uploads/domestic_violence_ad_m.jpg

Face Detection• Various algorithms developed• Find faces through various

methods▫Comparing against a large

training set▫Looking for components of a

face in a scene (eyes, mouth, etc)

• With further development, can extract other information▫Gender▫Mood

Image Source: http://www.dcviews.com/press/images/Face-detection.jpg

Face Tracking•After a face is detected

▫Follow it as it moves through a scene

•Completed many ways▫Fast face detection▫Color Analysis▫Body analysis▫Template Matching

Image Source: http://www.globalspec.com/NpaPics/38/121978_020520089775_ExhibitPic.JPG

Image Source: http://scherle.com/wp-content/uploads/2009/03/500px-android-logosvg.png

Android Introduction• Created by Google• “Software stack for mobile

devices that includes an operating system, middleware and key applications”

• Generally used in Cell Phones▫HTC Hero▫HTC Magic▫Motorola Droid

• Also beginning to branch other devices▫Dell Mini 5 Tablet PC▫Acer Aspire One D250 Netbook▫Barnes and Noble Nook eReader

Image Source: http://en.wikipedia.org/wiki/File:Android_and_cupcake.jpg

Image Source: http://developer.android.com/images/system-architecture.jpg

Android SDK•Freely available •Currently in release 2.2

(FroYo)•Provides for interfacing with

various portions of an Android device▫Screen▫Wireless (Bluetooth, Wifi)▫Motion Sensing▫GPS▫Camera▫Telephony

Image Source: http://developer.android.com/assets/images/home/froyo-android.png

Platform

•Applications primarily written in Java•Plugin available for Eclipse IDE•Runs of native “Dalvik Virtual Machine”

▫Register-based Architecture▫Used for platforms with limited resources

(like a phone)•Applications can also be written in C/C++

▫Android Native Development Kit (NDK)

Device Development (HTC Magic)• Android Phone, marketed by T-Mobile as

‘myTouch 3G’• Specifications

▫Operating System: Android 1.6▫ Processor: Qualcomm® MSM7200A™, 528 MHz▫ Memory

512 MB ROM 288 MB RAM

▫ Display: 3.2-inch TFT-LCD flat touch-sensitive screen with 320x480 HVGA resolution

▫ Connectivity Bluetooth 2.0 Wi-Fi 802.11 b/g Proprietary USB Interface (Mini-USB 2.0/Audio Jack in

one)▫ 3.2 Megapixel Camera with auto focus

Image Source: http://www.thomy.com.br/blog/uploads/htc-magic-white.jpg

Device Development (Motorola Droid)• Android Phone, marketed by Motorola on Verizon

Network• Specifications

▫Operating System: Android 2.1▫ Processor: Arm® Cortex™, 550 MHz▫ Memory

512 MB ROM 256 MB RAM

▫ Display: 3.7-inch 16:9 Widescreen touch-sensitive screen with 480x854 WVGA resolution

▫ Connectivity Bluetooth 2.1 + EDR Wi-Fi 802.11 b/g Micro USB Interface

▫ 5 Megapixel Camera with auto focus and dual-LED Flash

Image Source: http://www.softsailor.com/wp-content/uploads/2009/10/Motorola-Droid-Multimedia-Station.jpg

Image Source:http://smartphonesfans.com/wp-content/uploads/2010/03/motorola-droid.jpg

Device Development (HTC Evo)• Android Phone, marketed by HTC on Sprint

Network• Specifications

▫Operating System: Android 2.1▫ Processor: Qualcomm® Snapdragon™ 8650, 1 GHz▫ Memory

1 GB ROM 512 MB RAM

▫ Display: 4.3-inch touch-sensitive screen with 800x480 WVGA resolution

▫ Connectivity Bluetooth 2.1 + EDR Wi-Fi 802.11 b/g Micro USB Interface HDMI Output

▫ 8 Megapixel Camera with auto focus and dual-LED Flash With 1.3 MP front-facing camera

Image Source: http://nexus404.com/Blog/wp-content/uploads2/2010/03/Sprint-HTC-EVO-4G-1.jpg

Image Source:http://www.hypebeast.com/image/2010/03/htc-evo-phone-4.jpg

Android/Computer Vision

•Potential for almost any application to be developed

•Numerous Augmented Reality applications already available for platform

Image Source:http://www.uwplatt.edu/web/presentations/PennState/ar/pix/augmented-reality-hud.jpg

Google Goggles• Application created by Google Labs• “Visual Search Application”• Can take pictures of various objects and

complete searches based on them▫Landmarks▫Books▫Artwork▫Places▫Wine▫Logos▫Business Cards

Can pull phone numbers from a card to import into phone to call, store, etc.

Google Goggles - Landmark

Image Source: http://www.google.com/mobile/images/labs/goggles/goggles_landmark.jpg

Google Goggles – Business Card

Image Source: http://www.google.com/mobile/images/labs/goggles/goggles_contact.png

Google Goggles – Wine

Image Source: http://www.google.com/mobile/images/labs/goggles/goggles_wine.jpg

Scotoma (Blind Spots)•“A spot in the visual

field in which vision is absent or deficient”

•Causes▫Macular Degeneration

Degeneration of the eye▫Optic neuritis

Inflammation, Degeneration of the Optic Nerve

▫And others..Image Source: http://en.wikipedia.org/wiki/File:CentralScotoma.jpg

Scotoma Treatments•Treatments center

around stopping the spot from growing larger

•For most cases, no cure•Effects (Very Situational)

▫Impairs functional acuity▫Can affect color

perception

Image Source: http://www.incm.cnrs-mrs.fr/pperso/portraits/scotoma.png

Scotoma Effects (Continued)• ‘Eccentric Viewing’

▫Also called Preferred Retinal Loci (PRL)

▫“This is similar to looking slightly away from an object at night in order to see it better” Dan Roberts, via

Macular Generation Support, www.mdsupport.org

Image Source: http://www.acbvi.org/albums/Vision/images/112p_donut.jpg

My Goal!

•To develop a system that will aid a person in compensating with various types of scotoma▫Through the use of

mobile devices

Finally, my development•Program to help those with various scotoma

interact•Limit (and possibly eliminate) ‘eccentric

viewing’ •Goals

▫Run efficiently and quickly▫Provide easy-to-use and configurable interface▫Be implementable on a mobile device▫Implement cutting-edge face detection methods

developed in-house ‘Random Projection’-Based Methods

Current Status•Prototype Implemented on a lab machine•Relatively Inefficient Implementation•Relies heavily on OpenCV graphical libraries for

interface▫OpenCV is a freely available computer vision

library that provides many standard algorithms and aids for implementation

▫Originally launched by Intel in 1999•Uses ‘off-the-shelf’ algorithms for face

detection▫Viola-Jones Detection

Prototype Images

Prototype Images

Prototype Images

Prototype Images

Prototype Images

Prototype Images

Prototype Images

Initial Android Development•Program written to perform edge detection

on various images•Uses standard edge detection method

▫Sobel Kernel •Further extended to implement the on-

board camera on the phone

Initial Android Development

Initial Android Development

Android Emulator HTC Magic Motorola Droid

Initial Android Development

Android Emulator HTC Magic Motorola Droid

Initial Android Development

Android Emulator HTC Magic Motorola Droid

Initial Android Development

HTC Magic (Java-Based Implementation)

HTC Magic (OpenCV-Based Implementation)

Initial Android Development

Motorola Droid(Java-Based Implementation)

Motorola Droid(OpenCV-Based Implementation)

Future•Research into optimizing hand-offs between

native libraries and interface•Optimize/Port previous lab-machine code to

Android•Experiment with different interface types

for configuration and use

Questions/Answers

References/Further Information• Merriam-Webster’s Online Dictionary

▫ http://www.merriam-webster.com/dictionary• Causes of Scotoma

▫ http://www.wrongdiagnosis.com/s/scotoma/causes.htm• Scotoma – Texas School for the Blind and Visually

Impaired▫ http://www.tsbvi.edu/Education/anomalies/scotoma.htm

• Eccentric Viewing▫ http://www.mdsupport.org/library/eccentric.html

• Energy saving of TV by face detection▫ http://portal.acm.org.offcampus.lib.washington.edu/citati

on.cfm?doid=1389586.1389693• Google Goggles Information

▫ http://www.google.com/mobile/goggles/