Activity And Emotion Recognition to Support Early Diagnosis of Psychiatric Diseases

Tampere, January 30thB. Arnrich

Activity and Emotion Recognition to Support Early Diagnosis of Psychiatric Diseases

Activity and Emotion Recognition to Support Early

Diagnosis of Psychiatric Diseases

Pervasive Health ConferencePervasive Health ConferenceTampere, 30Tampere, 30thth January 2008January 2008



David Tacconi, Oscar MayoraCREATE-NET

Paul LukowiczUniversity of Passau

Bert Arnrich, Cornelia Setz, Gerhard TrösterETH Zurich

Christian HaringPSYCHIATRIC STATE HOSPITAL TIROL

Paper ContributorsPaper Contributors

PSHTPSYCHIATRIC STATE HOSPITAL TIROL



OutlineOutlineIntroduction and Motivation

Bipolar disorder

Pervasive computing to support diagnosis of Bipolar Disorder

A proposed System Architecture

Discussion and Future Work



IntroductionIntroductionGlobal Burden of Disease

Mental illness accounts for over 15% of the burden of diseases in established market economies1

Disability Adjusted Life Years (DALYs) measure the lost years of healthy life due to premature death or disability

Depressionis the most common psychiatric disorder, accounting for 50.8 million DALYs or 10.7% of the global burden of disease It is ranked fourth among all causes of DALYs and is the leading nonfatal condition globally

Mental disorders like the bipolar disorderaccount for another 14.1 million (3.0%) DALYs

1World Health Organization, World Bank, Harvard University



MotivationMotivationExtend psychotherapy beyond the therapy hour

State of the Art: computer-aided between-session therapy

Online questionnairesAutomatic scheduling

Our proposal: Activity and Emotion Recognition as a specific contribution to therapy



ChallengesChallengesFew technological solutions exist to aid people affected by mental illness

Obvious reasons are:people affected by mental illness are more likely to have problems dealing with complex technologyproviding behavioral assistance is much more difficult than providing physical assistancesolutions require considerable amount of domain specific knowledge



Bipolar DisorderBipolar DisorderCharacterization

repeated relapses of depression and maniaRecurrence rates are high at around 50% to 70%

Treatment of Bipolar disorderMain: PharmacotherapyAlternative: teach the patients to recognize and manage Early Warning Signs (EWS)

Diagnosis through patient questionnaires Depression: Hamilton Depression Scale (HAMD)Mania: Bech-Rafaelsen Mania Scale (BRMS)Both contain a series of questions related to patient’s state, activities and feelings



Bipolar Disorder: the HAMDBipolar Disorder: the HAMD

1. Depressed Mood2. Feelings of Guilty3. Suicide4. Insomnia (early)5. Insomnia (middle)6. Insomnia (late)7. Work and Activities8. Retardation:

Psychomotor9. Agitation10. Anxiety (Psychological)

11. Anxiety Somatic12. Somatic Symptoms

(Gastrointestinal)13. Somatic Symptoms

General14. General Symptoms15. Hypocondriasis16. Loss of Weight17. Insight18. Diurnal Variation19. Depersonalization and

Derealization20. Paranoid Symptoms



Bipolar Disorder: the BRMSBipolar Disorder: the BRMS

1. Motor activity

2. Verbal activity

3. Flight of thoughts

4. Voice/Noise level

5. Hostility

6. Mood and feelings of well-being

7. Self-esteem

8. Contact

9. Sleep

10. Sexual interest and activity

11. Work level



ContributionsContributionsWe identify Bipolar Disorder as a condition that can realistically benefit from behavioral monitoring

We identify support in early detection of imminent transitions between normal, manic and depressed states as the specific contribution to therapy

We identify specific behaviors that need to be detected by the proposed system, using the HAMD and the BRMS

Based on literature study and previous work by the authors, we argue that detecting these specific behaviors is feasible

We propose an appropriate system architecture based on existing devices and previous systems implemented by the authors groups



Insomnia and Sleep disordersInsomnia and Sleep disordersHAMD 4-6, BRMS 9

”Gold standard” (laboratory settings):polysomnographic monitoring of sleep timephysiological parameters (e.g. respiration, heart rate variability) and sleep motion

Alternative On-body sensorsunobtrusively embedded into biomedical clothes or mattressesallow to obtain preliminary diagnosis and to perform more frequent tests under real-life conditions

Alternative sensor mats placed under the mattressthin film, dynamic quasi-piezoelectric sensorscapacitive pressure sensor mat



Verbal activities and ConversationsVerbal activities and ConversationsBRMS 2, 4 and BRMS 8

Spoken messages convey non-textual characteristics like intonation, speaking rate or emotional state

Automatic speech character identification would allow to extract features describing contextual side information

Emotion recognition can give the therapists information about variation of the patient’s mental state



Emotion RecognitionEmotion RecognitionRestriction to a set of basic emotional states

Feasibility study: 10 subjects6 emotionsRecognition rates comparable to humans



Activity RecognitionActivity RecognitionHAMD 7-11 and BRMS 1

Several past works on activity recognition

Based on previous experience, we target systematic real life trials to:

quantify Work and Activities (HAMD 7 and BRMS 1)detect Agitation (HAMD 9) and Anxiety (HAMD 10, 11)measure Psychomotoric Retardation (HAMD 8)



Activity recognitionActivity recognitionMain sources of activity information

Worn combination of accelerometer and microphoneLocation information

Previous experimentsSpotting complex activities is feasibleRecognition directly on wrist worn device



System Architecture System Architecture Challenges to be considered:

Patients are likely to reject pervasive computing technology in principleTarget devices should be as less obtrusive as possiblePatients cannot be asked to perform any training of devices

Activity and emotion recognition is targeted to medium and long term behavior

Higher errors in single activity recognition are allowed Focus on average behaviors rather than in instantaneous activitypattern or emotionsBehaviors that are repeated in time and that can be symptoms of disease’s relapse



System ArchitectureSystem ArchitectureThe User Interfaces module:•present persuasive feedback to the users for motivating healthier patients’ behavior

The User Model includes all patient’s characteristics, disease’s peculiarities and his preferences. Information stored in:•User Profile (UP)•Disease Description (DD)•Patient Description (PD)

The Context Acquisition module gathers data from Sensors and is driven by:•Emotion Recognition Manager that selects sensors for emotion recognition•Activity Recognition Manager that selects sensors for recognizing user’s activity•User model manager gives proper inputs

The Content Manager module is responsible:•For uploading the data to the EMR through the Data Upload module•For presenting information to the patient through the Feedback Manager module



Conclusion and Future workConclusion and Future workConclusion

Concept on applying existing pervasive computing techniques to support the early diagnosis of bipolar disorder Proposal of a system architecture designed to monitor patient’s behavior

Future work:Integrating the currently available technologyLaboratory testingField test at the Psychiatric Hospital in Tirol, Austria



Thank you for your attention.Thank you for your attention.

Activity And Emotion Recognition to Support Early Diagnosis of Psychiatric Diseases

Health & Medicine

support early

support early

emotion recognition

arnrich tampere

arnrich tampere

arnrich tampere

selects sensors

patients