iSenseStress: Assessing Stress Through Human-Smartphone Interaction Analysis

iSenseStress: Assessing Stress Through Human-Smartphone

Interaction AnalysisMatteo Ciman1, Katarzyna Wac2 and Ombretta Gaggi1

1 University of Padua Padua, Italy

2 University of Geneva and University of Copenhagen Geneva, Switzerland and Copenhagen, Denmark

PervasiveHealth 2015, Istanbul, Turkey /26

Stress Experience

• Stress is mental condition experienced every day

• Long exposure can lead to anxiety, depression etc. => increase of healthcare costs

• In 2013 American teens reported stress experienced at unhealthy levels (and at increasing lower ages) [http://www.apa.org/news/press/releases/stress/2013/teenstress.aspx]

2

• Early assessment of stress condition can help to provide feedback to improve health state of individuals

http://www.apa.org/news/press/releases/stress/2013/teenstress.aspx


Stress Assessment - State of the Art

• Stress assessment using wearable and ubiquitous devices can increase individuals’s acceptance without interfering with their life

• MouStress: project for stress assessment considering computer mouse movements or keyboard [1]

• Usage smartphone sensors (WiFi, GPS, Bluetooth, calls, SMS) [2]

3

[1] D. Sun, P. Paredes, and J. Canny, “Moustress: Detecting stress from mouse motion,” in SIGCHI Conference on Human Factors in Computing Systems, 2014, pp. 61–70. [2] G. Bauer and P. Lukowicz, “Can smartphones detect stress-related changes in the behaviour of individuals?” in PERCOM Workshops, 2012.


The idea

4


Our Approach

• No external devices used, just smartphone (less expensive, more usable)

• No privacy-related information (i.e., calls, messages, location etc.)

• Possible to run a phone background service all the day long

• Based on human-smartphone interaction analysis

• Limitation: an interaction with the smartphone is required to make an assessment

5


Human-Smartphone Interaction

6

Tap

Scroll

Swipe

Text WritingDouble

Tap

Rotate

Zoom

Pinch

Long press


Tasks Definition

• Search Task:

• Scroll, swipe and tap

• Write Task:

• Tap, Text Writing

7

Tap Scroll Swipe

Tap Text Writing


Search Task

• Find inside a 21x15 grid the right icon (s)

• Scroll and Swipe to inspect all the icons

• Tap to select the right icon

8


Search Task Features

• Tap {min, max, average} pressure / length / size

• Scroll, Swipe

• {min, max, average} speed / time length / acceleration / pixels length / pressure

• Linearity

• D(interaction, center), D(interaction, top_left_screen)

9


Write Task

• Paragraph writing as text message

• Keyboard without autocorrection or word suggestion

• English as text language

10


Write Task Features

• Tap {min, max, average} pressure, length, size

• Tap movement and duration

• Writing:

• Speed

• # errors

• Back digits

11


Protocol

12

Initial Relax (5’)

Relaxed Tasks (~30’)

Stressor (5’-10’)

Stressed Tasks (~10’)

Self Assessment

Negative ValenceLow energyNot Stressed

Positive ValenceHigh Energy

Stressed

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

ESM 1 ESM 2 ESM 3 ESM 4 ESM 5


Protocol (II)

13

Initial Relax (5’)

Relaxed Tasks (~30’)

Stressor (5’-10’)

Stressed Tasks (~10’)


How to Stress People

• Most common used stressors for tests

• Mathematical problems

• Timing pressure

• Social evaluation

• Repetition

• Uncontrollability

14

We used these


Stressor Task: Math

15

Large prime number


Stressor Task: Math (II)

16

Progress bar + tic-tac sound

Random decrease Digits back to 0

every time

Each wrong answer annoying sound and going back


Search Task - Stressed

17


Sound + vibration


Write Task - Stressed

18



User Study

• 13 Participants (7M, 6F), average age 26,38 (± 2,53)

• Own phone, no constraints for the type to use (Android OS)

• Different English literacy level

• Average protocol duration: 1 hour

• Cover story: New Google interface analysis

19


Stress Induction Analysis

20

Initial Relax

Relaxed Tasks

Stressor Stressed TasksESM 1 ESM 2 ESM 3 ESM 4 ESM 5

TEST t(13) p-value

ESM 3 VS ESM 4 1.99 0,007 *

ESM 3 VS ESM 5 -2.84 0,009 *

ESM 4 VS ESM 5 -2.74 0.5

Participants were stressed

Different stress level at the end of tasks

Kept stressed during stress tasks


Features Evaluation

• Statistical analysis for significance evaluation

• Stress prediction model using Decision Tree (DT), k-Nearest Neighbourhood (kNN), Bayes Network (BN), Support Vector Machine (SVM) and Neural Networks (NN)

• User and global model (evaluated using 10-Fold cross validation and leave-one-out)

21


Search Task - Statistical Correlation• Only weak correlation between our features

• Global Model

• Average swipe pressure (p-value = 0,09)

• Scroll distance from center (p-value = 0,065)

• Scroll distance from top left (p-value = 0,07)

22

• User model

• Scroll interaction length (strong correlation for 61% of users)

• Scroll delta (strong correlation for 40% of users)

• Scroll linearity (strong correlation for 45% of users)


Search Task - Prediction Model

23

F-measure for Scroll interaction models

MODEL DT KNN SVM NN BN

USER (AVERAGE) 0.79 0.80 0.81 0.80 0.77

GLOBAL (AVERAGE) 0.73 0.71 0.78 0.74 0.67

F-measure for Swipe interaction modelsMODEL DT KNN SVM NN BN

USER (AVERAGE) 0.86 0.86 0.79 0.87 0.85

GLOBAL (AVERAGE) 0.92 0.75 0.81 0.82 0.77


Write Task - Statistical Correlation

• User Model

• Digits size (64% of users with strong correlation)

• Pressure/Size ratio (55% of users with strong correlation)

• Global Model

• Wrong Words / Total words ratio (p-value = 0,028)

• Digits time distance (p-value = 0,012)

• Digit duration (p-value = 0,08)

24


Conclusions• Stress assessment using data from non-intrusive

devices can increase people’ acceptance

• Human-smartphone interaction analysis can be leveraged to assess stress state in users

• Scroll and Swipe: F-measure of stress prediction between 79% and 85% for user models, and between 70% and 80% for global model.

• Text writing: several features showed strong correlation

25


Future works

• Real-time background service for stress assessment

• Behaviour suggestion implementation

• Stress assessment in the wild (ongoing study, 29 participants)

26

• My PhD Thesis :)

iSenseStress: Assessing Stress Through Human-Smartphone

Interaction AnalysisMatteo Ciman, Katarzyna Wac and Ombretta Gaggi

{mciman,gaggi}@math.unipd.it [email protected]

[email protected]

http://math.unipd.it

mailto:[email protected]

iSenseStress: Assessing Stress Through Human-Smartphone Interaction Analysis

Education

stress experience stress

stress assessment state

art stress assessment

stressrelated changes

size tap movement

task paragraph writing

search task features

pressure length size