Sensing mobile phones

1

Arpan Pal

Innovation Lab, TCS

May 17, 2015

Context-aware Mobile Phones - towards Ubiquity

Agenda

• Ubiquity and Internet-of-Things• Mobile Phone based Sensing• Smart Solutions using Mobile Phone Sensors• Ongoing Work

Ubiquity and Internet-of-Things

4

Ubiquitous Computing

“Ubiquitous computing enhances computer use by making many computers available throughout the physical environment, but making them effectively invisible to the user”

At present Ubiquity is viewed as a Consumer phenomenon – However widespread adoption of ubiquitous devices among Enterprise stakeholders will drive Enterprises to ubiquity.

5

Smart Solutions using Ubiquity

• Smartness comes from Context-awareness• Context-awareness comes from

• Sensing one’s environment (anything)• Analyzing the sensed data to make meaningful inferences• Actuate / Render value-added information from the inferred analytics

• anytime, anywhere (on Mobile, TV, PC, Tablet)

• Ubiquity CycleSense -> Analyze -> Actuate

• Need to do all the above while preserving the privacy

• Need for easy-to-use HMI to make technology hidden to background

6

Internet-of-Things (IoT) – towards Ubiquity

Internet

Sensors (Location, Audio , Camera, …)

Gateway (Stand-alone Device / Mobile Phones)

Backend Server & database

Portal

USB, CAN, Zigbee, BT, NFC, WiFi

GSM/GPRS, 3G

BAN

KIN

GB

AN

KIN

G

INSU

RA

NCE

INSU

RAN

CE

AG

RIC

ULT

UR

EA

GRI

CULT

URE

HEA

LTH

CARE

HEA

LTH

CAR

E

GO

VER

NM

ENT

GO

VER

NM

ENT

UTI

LITY

UTI

LITY

MA

NU

FACT

URI

NG

MA

NU

FACT

URI

NG

TRA

NSP

ORT

TRA

NSP

OR

T

APPLICATION SERVICESAPPLICATION SERVICES

INFRASTRUCTURE PLATFORMINFRASTRUCTURE PLATFORM

INTERNET INTERNET

GATEWAYGATEWAY

ACT

UA

TEA

CTU

ATE

SEN

SESE

NSE

AN

ALY

ZEA

NA

LYZE

Mobile Phone based Sensing

8

Mobile Phones as Sensor and Gateway

Smart Phones

InternetWiFi, 2G/GPRS, 3G

On-board sensorsAccelerometer, GPS,

CompassCamera, Microphone

Internet

Sensors (Location, Audio , Camera, …)

Gateway (Stand-alone Device / Mobile Phones)

Backend Server & database

Portal

USB, CAN, Zigbee, BT, NFC, WiFi

GSM/GPRS, 3G

Other sensor interfaces through

Bluetooth and USB

9

Mobile Phones as Sensor and Gateway

Basic and Feature PhonesInternet2G/GPRS

On-board sensorsMicrophone, Camera

Internet

Sensors (Location, Audio , Camera, …)

Gateway (Stand-alone Device / Mobile Phones)

Backend Server & database

Portal

USB, CAN, Zigbee, BT, NFC, WiFi

GSM/GPRS, 3G

10

Sensor Penetration in Indian Mobile Phones

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Bluetooth

USB

Edge

GPRS

Wifi

3G

Camera

GPS

Accelarometer

Digital Compass

Basic Phones

% Penetration for sensors and network interfaces

x100

11

Sensor Penetration in Indian Mobile Phones

% Penetration for sensors and network interfaces

0 0.2 0.4 0.6 0.8 1

Bluetooth

USB

Edge

GPRS

Wifi

3G

Camera

GPS

Accelarometer

Digital Compass

Feature Phones

x100

12

Sensor Penetration in Indian Mobile Phones

% Penetration for sensors and network interfaces

0 0.2 0.4 0.6 0.8 1

Bluetooth

USB

Edge

GPRS

Wifi

3G

Camera

GPS

Accelarometer

Digital Compass

Smart Phones

x100

13

Sensor Penetration in Indian Mobile Phones

% Penetration for sensors and network interfaces

0 20 40 60 80 100

Bluetooth

USB

Edge

GPRS

Wifi

3G

Camera

GPS

Accelarometer

Digital Compass

Consolidated Market Penetration

14

Power Consumption using Sensors

Source: Nericell: Rich Monitoring of of Road and Traffic Conditions using Mobile SmartphonesPrashant Mohan et. al., Microsoft Research, SenSys 2008, North Carolina, USA

Smart Solutions using Mobile Phone Sensors

16

Pothole Detection using Accelerometer

“The Pothole Patrol: Using a Mobile Sensor Network for Road Surface Monitoring”Jakob Ericsson et.al., MIT Labs, MobiSys 2008, Colorado, USA

17

Audio Sensing using Microphone

“SoundSense: Scalable Sound Sensing for People-centric Applications on Mobile Phones”Hong Lu et.al., Dartmouth College, MobiSys 2009, Krakow, Poland

Applications• Daily Diary• Music Classification• Honk Detection

18

Bike Sensing using external Sensors

“The BikeNet Mobile Sensing System for Cyclist Experience Mapping”Eisenman et.al., Columbia Univ, SenSys 2007, Sydney, Australlia

Applications• Fitness Sensing• Cyclist Experience• Pollution Monitoring

Ultrasound based Tracking in Smart Stores

• Ultrasound beacon transmitters used for location marks / landmarks in a retail store

• Registered Customers carrying Mobile phones can receive these beacons as a record of visiting the designated location

• The visit records are uploaded for loyalty discounts, behavior analysis etc.• The audio frequency used is in the in-audible regions around 20 KHz. • A wide range of cell phones can support this mode of communication and can

enable location sensing in smart spaces

20

Context-aware TV using Microphone and Camera

QR Code

OCR

Ultrasound Internet

Applications• Delivering TV context-aware information and ads to mobile phones / tablets

21

OCR using Mobile Phone Camera Data

“Shah, P.; Karamchandani, S.; Nadkar, T.; Gulechha, N.; Koli, K.; Lad, K.; , "OCR-based chassis-number recognition using artificial neural networks," Vehicular Electronics and Safety (ICVES), 2009 IEEE International Conference on , vol., no., pp.31-34, 11-12 Nov. 2009

Applications• Automatic scan of Car Identifier Number for Insurance Application

Edge Detection

Horizontal Segmentation

Vertical Segmentation

22

Eye Tracking using Camera

“EyePhone: Activating Mobile Phones with Your Eyes”Emiliano Mullizo et.al., Dartmouth College, MobiHeld 2010, New Delhi, India

Applications• Novel Mobile PhoneHMI

23

Brain-Computer Interaction

“NeuroPhone: Brain-Mobile Phone Interface using a Wireless EEG Headset”Campbell et.al., Dartmouth College, MobiHeld 2010, New Delhi, India

Applications• Novel Mobile Phone HMI

Ongoing Work

25

Use Case Focus

Internet of ThingsSmart Home --> Smart Car --> Smart CityBA

NKI

NG

BA

NKI

NG

INSU

RA

NCE

INSU

RAN

CE

AG

RIC

ULT

UR

EA

GRI

CULT

URE

HEA

LTH

CARE

HEA

LTH

CAR

E

GO

VER

NM

ENT

GO

VER

NM

ENT

UTI

LITY

UTI

LITY

MA

NU

FACT

URI

NG

MA

NU

FACT

URI

NG

TRA

NSP

ORT

TRA

NSP

OR

T

APPLICATION SERVICESAPPLICATION SERVICES

INFRASTRUCTURE PLATFORMINFRASTRUCTURE PLATFORM

INTERNET INTERNET

GATEWAYGATEWAY

ACT

UA

TEA

CTU

ATE

SEN

SESE

NSE

AN

ALY

ZEA

NA

LYZE

Smart City

Smart Home

Smart Buildings

Smart VehicleSmart Infra

ParticipatorySensing

Smart Stores

26

Application Focus

App Usage

Pothole / Traffic Jam Detection Smart City Municipal Service / Intelligent Transportation System

Driver Behavior Rogue Driver Detection / Dynamic Insurance

Ultrasound-based localization Smart Tourism / Smart Store

Audio Surveillance using Mobile Phones / Video Surveillance

Law Enforcement

Context-aware Advertisement Target Advertisement delivered to User

Multimodal HMI (Audio and Gesture)

Rich and easy-to-use User Interfaces

27

Platform Focus

Component Purpose

Phone Middleware Java Based Service Container and Device Management

Phone Analytics Statistical and Analytical Processing of Sensor Data – low footprint, low computation power

Privacy Privacy preservation through aggregation, anonymization and access control

Back-end Platform Device Management, Control and Sensor Data Storage

Back-end Analytics Machine Learning, Reasoning, Stream Processing and Large data Handling

Thank you

[email protected]