Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my health state

Presentacin de PowerPoint

Petia [email protected] University of Barcelona & Computer Vision Center

Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my healthstate

08:15

1

Social networks & lifeloggingFacebook. Facebook has more than 350 million active users and more than 2.5 billion photos are uploaded every month.

Flickr.com Flickr has around 44 million users.

Twitter.com More than 3 million tweets are published daily.

Youtube.Youtube is one of the Top-5 most visited websites in the world with more than 5 billion videos uploaded monthly. Instagram Instagram exceed 20 billion photos shared

2

To Log or Not to Log? Privacy Risksand Solutions for Lifelogging and Continuous Activity SharingApplications, Blaine Price, ENISA, Centre for Research in Computing.

08:15

The age of the quantified self

08:38

3Our body is radiating data

Data: loudly, continuously, honestly and individually

Heart rate, risk for cancer, events that change our mood, stress, breading, sleep, etc.

Sensors for caregivers, mothers, fermons sensors.

Sensors for behavioural modifications to be more: mindful, aware, present, human.

Lauren Constantini: Wearable tech expands human potential

Which wearables do consumers plan to buy?

Its expected to double by 2018, to 81.7 million users.

Almost 2 in 5 internet users will use wearables by 2019.08:15

4The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed 1,001 US internet users. Source: eMarketer.

Which wearables do consumers plan to buy?

The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed 1,001 US internet users. Source: eMarketer.Wearable usage will grow by nearly 60% this year.

Wearables market grows 172% in a year; 78 Million devices shipped (21 million Fitbits)IoT Daily, Connected thinking

08:40

5

Lifelogging

08:15

6

Technology for life-logging is here!

Evolution of life-logging apparatus, including wearable computer, camera, and viewfinder with wireless Internet connection. Early apparatus used separate transmitting and receiving antennas. Later apparatus evolved toward the appearance of ordinary eyeglasses in the late 1980s and early 1990s .

7

Quantified Self & life-logging MeetsInternet of Things (IOT), Mazzlan Abbas.

08:15

Wearable cameras and the life-logging trend8

Shipments of wearable computing devices worldwide by category from 2013 to 2015 (in millions)

Wearable Camera Shipments and Revenue, World Markets: 2015-2021, Source: Tractica08:15

Visual life-loggingBenefits:

A digital memory of people you met, conversations you had, places you visited, and events you participated in. This memory would be searchable, retrievable, and shareable.

A 14/7/365 monitoring of daily activities. This data could serve as a warning system and also as a personal base upon which to diagnosis illness and to prescribe medicines.

A way of organizing, shaping, and reading your own life.A complete archive of your work and play, and your work habits. Deep comparative analysis of your activities could assist your productivity, creativity, and consumptivity.

To the degree this life-log is shared, this archive of information can be leveraged to help others work, amplify social interactions, and in the biological realm, shared medical logs could rapidly advance medicine discoveries.

9The hard part is no longer deciding what to hold on to, but how to efficiently organize it, sort it, access it, and find patterns and meaning in it.08:15

10 things to know about lifelogging Social services get logging (Facebook, Twitter, Spotify)

Fitness trackers are big market

Lifelogging apps for smartphones: Saga, Instant, Narrato, OptimizeMe

There is dedicated hardware (wearable cameras)

Big tech companies are sniffing around (Sony, Samsung, Apple, etc.)

Wearables capture more data (smart watches and augmented glasses)

It really isn't a new thing (MyLifeBits since 2000s, Gordon Bell)

Lifelogging gets emotional (MIT project)

Lifelogging can be art (Alan Kwan's Bad Trip, Stephen Cartwright)

08:15

10

But!

08:15

11Scared?

Ethical guidelines for wearable camerasAnonimity and confidentiality: Researchers coding image data should:not discuss the content with anyone outside of the team,not identify anyone they recognize in the images, be aware of how sensitive the data are.

Data encryption: Confdentiality can be protected by confguring devices and using specialist viewing software to make the images accessible only to the research team (lost devices).Devices should be configured so that data can only be retrieved by the research team. It should be impossible for participants or third parties who find devices to access the images.

Data storage: Collected images should be stored securely and password-protected, according to national regulations.

1208:15

Understanding user privacy requirements and risks from emerging technologiesPeople are bad at:understanding the future value of revealing private information today,understanding the risks from technology they have not yet used or heard of.

13Wearable cameras can be very useful:

An estimated one million Russian motorists have dashboard video cameras installed in their cars.

Police officers carring video camera units and using Velcro to place these cameras in police wagons, helmet cams, ear cams, chest cams with audio capability, GPS locators, taser cams

Even with only half of the 54 uniformed patrol officers wearing cameras at any given time, a department in USA had an 88 % decline in the number of complaints filed against officers, compared with the 12 months before the study, The New York Times, 4th of July, 2013.

The worlds leading police body worn video camera deployed by over 4000 agencies in 16 countries.08:15

Benefits and potential applicationsIt will take quite some time for people to feel comfortable with always connected devices that can discreetly take photos or videos. Will the benefits outweigh the negatives?

Quantified Self & life-logging Meets Internet of Things (IOT), Dr. Mazlan Abbas, MIMOS Berhad

14

08:15

How else can be LL useful?

15

08:15

Life-logging data

16

What we have:

08:15

Wealth of life-logging dataWe propose an energy-based approach for motion-based event segmentation of life-logging sequences of low temporal resolution- The segmentation is reached integrating different kind of image features and classifiers into a graph-cut framework to assure consistent sequence treatment.Complete dataset of a day captured with SenseCam (more than 4,100 images

17Choice of devise depends on: 1) where they are set: a hung up camera has the advantage that is considered more unobtrusive for the user, or 2) their temporal resolution: a camera with a low fps will capture less motion information, but we will need to process less data.We chose a SenseCam or Narrative - cameras hung on the neck or pinned on the dress that capture 2-4 fps.

100.000 images per monthThe wealth or the hell of life-logging data08:15

Extracting semantics from egocentric imagesComputer Vision allows to process and analyze huge amount of images and extract the semantics from them.18

08:15

What technology to apply?

But in fact we are not very interested in every and each of these images but to extract semantic information or actually meanings out of them for later use.These meanings could be anything, from thinking about how I do eat or workout or even communicate with others. Studying these semantic info is an interesting subject since they can help one to observe his own life style or even helping people with memory impairment to improve their memory by reviewing events happened to them during a day.To accomplish this, a search engine is required to use existing clues to recognize different events and meanings, this procedure is called life-logging

08:15

19

Deep leearning everywhere

2008:15

Deep learning applications

2108:15

Other methods also use unsupervised pre-training to structure a neural network, making it first learn generally useful feature detectors. Then the network is trained further by supervised back-propagation to classify labeled data. The deep model of Hinton et al. (2006) involves learning the distribution of a high level representation using successive layers of binary or real-valued latent variables. It uses a restricted Boltzmann machine to model each new layer of higher level features. Each new layer guarantees an increase on the lower-bound of the log likelihood of the data, thus improving the model, if trained properly. Once sufficiently many layers have been learned the deep architecture may be used as a generative model by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations.[8] Hinton reports that his models are effective feature extractors over high-dimensional, structured data.[9]

Natural Language Processing which is used heavily in language conversion in chat rooms or processing text from where human speeches.Optical Character Recognition which is scanning of images. It's gaining traction lately to read an image and extract text out of it and correlate to the objects found on imageSpeech Recognition applications like Siri or Cortana needs no introductionArtificial Intelligence induction to different robots for automating at least a minute level of tasks a human can do. We want them to be a little smarter. Drug discovery though medical imaging-based diagnosis using deep learning. It's kind of in early stages now. Check Butterfly Network for the work they are doing.CRM needs for companies are growing day by day. There are hundreds of thousands of companies around the globe from small to big companies who wants to know their potential customers. Deep Learning has provided some outstanding results. Check for companies like RelateIQ (product) who has seen astounding success of using Machine Learning in this area.

21

From NN to CNN

22

08:15

Imagenet

23

08:15

ConvNets are everywhere

24From: Fei-Fei Li & Andrej Karpathy & Justin Johnson

08:15



08:15



08:15



08:15



08:15

Egocentric visionHow can deep learning help process egocentric images?08:45

29

InformativenessCNN

Day Lifelog

InformativeImagesEgocentric informativeness through deep learning08:15

30

Visual summary through keyframe extraction (joint work UPC & UB)

31

Results08:15

Analysis of human interaction

32

F-formations extraction from egocentric vision08:15

Two main questions?What we eat?

Automatic food recognition vs. Food diaries

And how we eat?

Automatic eating pattern extraction when, where, how, how long, with whom, in which context?

33

08:15

Automatic Food AnalysisFood detectionFood recognitionFood environment recognitionEating pattern extraction

34

08:15

Food datasets

35Food256- 100images/classClasses: 256

Food101 101.000 imagesClasses: 101Food101+FoodCAT: 146.392 (101.000+45.392)Classes: 131EgocentricFood: 5038 imagesClasses: 908:15

Food-related environment recognition

36Our localization method based on Global Average Pooling (GAP) that produces a Food Activation Map (FAM).

Examples of localization and recognition on UECFood256 (top) and EgocentricFood (bottom). Ground truth is shown in green and our method in blue.Marc Bolaos,Petia Radeva:Simultaneous Food Localization and Recognition.submitted to ICPR16,arXiv.org>cs>arXiv:1604.07953, 2016.08:15

Food environment classification

37BakeryBanquet hallBarButcher shopCafeteraIce cream parlorKitchenKitchenetteMarketPantryPicnic AreaRestaurantRestaurant KitchenRestaurant PatioSupermarketCandy storeCoffee shopDinetteDining roomFood courtGalley

Classification results:0.92- Food-related vs. Non-food-related0.68 - 22 classes of Food-related categories 08:15

Image InputFoodness MapExtractionFood Detection CNN

Food Recognition CNNFood TypeRecognitionAppleStrawberryFood recognitionResults: TOP-1 74.7%TOP-5 91.6%SoA (Bossard,2014): TOP-1 56,4%

08:1538

Demo

3908:15

What did I do today?

40Wearable cameras allows to visualize our diary.Computer Vision allows to process huge amount of data and automatically extracts the semantics from it: where, what, who, how, etc. 08:15

Will life-logging and internet of things help know us better?41

08:15

How much time I spend by day on different activities?42

08:15

My working hours43

08:15

With whom I was interacting with?44

08:15

How stressful am I in different places?45

08:15

Six Months of My Life46

by David El Achkar

08:15

Potential applications4708:15

Metabolic diseases and health48

08:15

4.2 million die of chronic diseases in Europe (diabetesor cancer) linked to lack of physical activities and unhealthy diet.Physical activitiescan increase lifespan by 1.5-3.7 years.Obesity is a chronic disease associated with huge economic, social and personal costs. Risk factors for cancers, cardiovascular and metabolic disorders and leading causes of premature mortality worldwide.

Health and medical careToday, 88% of healthcare costs are spent on medical care access to physicians, hospitals, procedures, drugs, etc.

However, medical care only accounts for approximately 10% of a persons health.

Approximately half the decline in U.S. Deaths from coronary heart disease from 1980 through 2000 may be attributable to reductions in major risk factors (systolic blood pressure, smoking, physical inactivity).

49

08:15

What are we missing in health applications?Today, automatically measuring physical activity is not a problem.

But what about food and nutrition? State of the art: Nutritional health apps are based on manual food diaries.

08:15

50SparkpeopleLoseIt!MyFitnessPalCronometerFatsecret

Towards healthy habitsTowards visualizing summarized lifestyle data to ease the management of the users healthy habits (sedentary lifestyles, nutritional activity, etc.).

Life-logging can help us to extract our nutritional habits: taking photos of our everyday life and being able to analyse what we eat, starting by the dish recognition, where we eat, with whom we eat, how we eat.

08:15

51

Teleassistance and home monitoringUsing wearable cameras, we can have a clear picture of the lifestyle the person is having during the whole day: Is he/she active or passive doing domestic duties, performing intellectual activities like reading, coping with daily functions, etc.

08:15

52

Life-logging for cognitive treatment of people with amnesia53

Program based on life-logging captured by a Wearable Camera recording specific autobiographical episodes for stimulating posteriorly episodic memory function. Using wearable cameras and looking at their autobiographic experiences, people with amnesia improved their memory and cognitive facilities by using episodic images.

Claire, a 49-year-old former nurse, six years ago suffered brain damage due to a rare viral infection called herpes encephalitis. Now an amnesiac who is unable to recognize faces, Claire lives in a world in which even her lifelong friends appear as strangers.08:15

Life-logging for MCI treatment54Goal: using episodic images to develop cognitive exercises and tools for memory reforcing of MCI and Alzheimer people.

To explore the association between changes in cognitive, functional and emotional outcomes.

08:15

Application of Lifelogging to MigraineObjective Biomarkers in Migraine - how to predict a migraine attackHow does the brain interact with the environment - the cues that ensure adaptation55

Lifelogging can assure individualized tools to detect which are the triggers of the migraine for an individual.

08:15

Lifelogging and wearable camerasLifelogging technology is here

Computer Vision and Machine Learning can help process big image data

How to get profit of it to help improve our health?! 08:32

56

Thank you!5708:15

57

Deep learning applications

5808:15

Medical applications - there are tremendous advances in robotic surgery that relies on extremely sensitive tactile equipment. However, if a doctor can advise a robot to "move a fraction of a millimeter to the left of the clavicle" they could potentially gain more control by directing the robot via full understood voice control.Automotive - we are already seeing self driving cars; deep learning will possibly integrate into automated driving systems to detect and interpret sights and sounds that might be beyond the capacity of humans.Military - drones are particularly well suited to deep learning.Surveillance - here too drones will play a role, but the idea of computers that are able to sense and interpret with a human-like degree of accuracy will change the way in which surveillance is done.

58

Abstract

Using daily Facebook, Twitter or Instagram, people having conscience or not about it, are immersed into a rising trend call lifelogging. The practice of lifelogging refers to logging or registering humans daily life with technical tools and services. Lifelogging allows to people continuously recording their everyday experiences, typically via wearable sensors including accelerometers and cameras, among others. When the visual signal is the only one recorded, typically by a wearable camera, it is referred to as visual lifelogging. Nowadays, wearable cameras are very small devices that can be worn all-day long and automatically record the everyday activities of the wearer in a passive fashion, from a first-person point of view. Big amounts of data acquired over long periods of time, offer considerable potential for inferring knowledge about behavior patterns, habits or lifestyle of the user and hence enable many real applications. In this talk, we will discuss how visual lifelogging can provide a disruptive technology to help people with obesity, overweight, diabetes, or cardiovascular disease improve their healthy nutritional habits; or people with mild cognitive impairment to be involved in a new cognitive training framework and thus delay the progress of the Alzheimer disease.

08:15

59