Top Banner

Click here to load reader

Thrust 2 07-aug-2013_v3 (1)

Jan 12, 2015




  • 1. Thrust 2: Interaction Modeling (2013-14 academic years) Defined last year by Shri: 1. Analyze and model spatio-temporal context surrounding (leading to, caused by) behavioral events. 2. Quantitative description and understanding of the underlying structure (e.g., timing, phasing of salient events) and emergence and evolution of interactions.

2. Progress since Last Thrust 2 Con Call: (April 4, 2013) 1. CVPR paper 2. Nine Expeditions-related papers (or more?) in Interspeech 2013 (Lyon, France) (some are Thrust 2) 3. Synchronization issues for multimodal RABC data-base (all but Kinect). 3. Thrust 1 / Thrust 2 dependencies Thrust 1 is concerned with extracting high/low-level features for further analysis. Thrust 1s raison dtre is to serve Thrust 2 with descriptors. Thrust 2s obligation is to convey its needs to Thrust 1. We have already developed single-mode processing to analyze behavior; now it is time to use multi-mode methods. 4. Thrust 1 / Thrust 2 dependencies We need a comprehensive list of Thrust 1 deliverables. 1. Speech and Audio examples: Voicing intervals Diarization (who is talking when with cross-talk) Word locations (KWS) Non-linguistic events (laughter, whining, crying, sighs) Prosodic features Spectral/articulatory features 2. Q-sensors EDA Temperature Accelerometers Statistical measures associated with various states and timings 5. Thrust 1 / Thrust 2 dependencies We need a comprehensive list of Thrust 1 deliverables. 3. Vision . (many) 4. Gaze --- (many) 5. Kinect --- (many) Thrust 1: Please fill this out for us. 6. Thrust 2 projects using RABC (all multi-modal) 1. Engagement score analysis (revisited) We have engagement scores on all 157 sessions (only 43 are fully annotated). Develop optimal categorization using ground truth in the annotated sessions. Test using non-annotated sessions with Thrust 1 primitives. 7. Thrust 2 projects using RABC (all multi-modal) 2. Parsing of stages (revisited) The only attempt was using KWS (~80% correct) Obvious extension uses Kinect object tracking Others include EDA, para-linguistic detection, gaze, posture, etc. 8. Thrust 2 projects using RABC (all multi-modal) 3. Response to name: Every RABC session starts with greeting. After formal session, childs name is uttered twice (once from side, once from front) a. What responses are predictable? (All modes) b. Can we ascertain when name was called from childs reactions w/o audio cues? c. This will require gaze, head-orientation, EDA, posture, etc. 9. Multimodal Retrieval We currently have the ability to retrieve interesting regions of RABC session through key- word spotting and diarization. Easy extension using any of the other modalities. Challenging extensions would be: Find when child is unhappy (requires multi-modes) Find when child is not responding. 10. Multimodal Retrieval Examples: Is a child making eye contact while an examiner talking? (gaze and diarization) Where is a child looking at while an examiner talking? (voice active detector + gaze tracker + head orientation). 11. Multimodal Linguistic Analysis Classify examiners speech into the four sentence types: 1. Statement 2. Question (Are you ready to play with new toys?, where is the yellow duck?, etc.) 3. Exclamation (Its a hat! Its on my head!) 4. Command (Look at my book) Useful for retrieval and parsing. Childs responses and turn-taking to each type can be tracked. 12. What Happens During Cross-Talk? 1) terminal overlaps: a speaker assumes the other speaker has or is about to finish their turn. Can include head- nodding and gestures. 2) continuers: examples of the continuers phrases are mm hm or uh huh. (speech and para-linguistics) 3) conditional access to the turn: the current speaker yields their turn or invite another speaker to interject in the conversation, usually as collaborative effort. (all sorts of gestures) 4) chordal: a non-serial occurrence of turns, such as laughter, smiling, and gesturing. 13. Example of Multi-Modal Retrieval/Analysis During the ball/book play in RABC, what is a child response when a ball/book is presented? Is a child looking at the ball/book and the examiner back and forth? (word spotting + gaze tracker + head orientation). Is a child making vocalizations? If so, is it a positive or negative response? (word spotting + emotion classifier) Does a child smile when a ball/book is presented? (word spotting + smile detector) 14. Previous Project Example Uni-modal paralinguistic event detection Detection of laughter in childrens speech Training data from FAU Aibo Emotion Corpus (AEC) consists of spontaneous vocalizations/verbalizations by adolescents Testing on ~10 Rapid ABC sessions yielded accuracy of 70.58% Robust laughter detector with good generalization properties that can be used for paralinguistic event diarization 15. Current Project Example Multi-modal paralinguistic event detection Laughter detection using audio Smile detection using Omron. These have different time scales and frame rates. Fusion of confidence scores is likely to improve accuracy. Might EDA also be useful for this? 16. Basic Science in Required for Multi-Modal Analyses: Fusion Example from USC 17. Basic Science in Required for Multi-Modal Analyses: Fusion Example from GT: binary fusion method with different analysis lengths 18. Opportunistic use of other data as it may become available BU childrens diagnostic data. (Helen Tager-Flusberg) Includes audio and video of at-risk ASD children with more vocalizations. RABC Floor time (natural but very uncontrolled) ADOS (only available to USC) CFD table sessions. 19. Thrust 2 action items Set up blog like Thrust 1 Identify who is participating Monthly con-call for everyone. Set up time-table for new / revisited challenges.