Collaborative Annotation, Archival and Visualization in a Biofeedback Rehabilitation system
Hari SundaramArts Media and Engineering
Arizona State University
Memex Seminar, April 10, 2023 2
Introduction
Motivation: Every 45 seconds, someone in the United
States suffers a stroke. It results in functional deficits of neuropsychological and physical functions in post-stroke survivors.
Up to 85% of patients have a sensorimotor deficit in the arm, such as muscle weakness, abnormal muscle tone, abnormal movement synergies, and lack of coordination during voluntary movement
Goal Design a real time multimodal biofeedback
system for stroke patient rehabilitation. Archival / annotation and information
visualization to provide insight.
The Biofeedback system
[To appear in acm mm 2006]
Memex Seminar, April 10, 2023 4
System Overview
The Biofeedback system system situates participants in a multi-sensory engaging environment, where physical actions of the right arm are closely coupled with digital feedback.
The Biofeedback system integrates five computational subsystems. Motion capture Motion analysis Audio feedback Visual feedback Database for archival and annotation
All five subsystems are synchronized with respect to a universal time clock.
Memex Seminar, April 10, 2023 5
Action Analysis Arm Representation
11 labeled markers on arm and torso 3 labeled markers on the back of chair
Feature Extraction 3D hand trajectory / 3D hand trajectory relative
to the predefined straight line Shoulder / Elbow extension Hand Orientation Shoulder rotation / abduction/elevation Trunk flexion / rotation / lean and shoulder
trajectory Wrist extension
Multi-goal Framework Reaching Opening Flow
Memex Seminar, April 10, 2023 6
Coupling Action to Feedback Engagement
Aesthetically attractive, easy to use and intuitive.
Message and Mapping Reaching - visual target, an image
completion/reassembly task, and an accompanying musical progression.
Flow - pointalistic sound clouds in the main musical line, flowing particles in the visuals
Opening - a rich, resonant musical accompaniment.
Environment Introduction ( visual ) Abstract I (visual+audio) Abstract II (visual+audio), more
variation
Memex Seminar, April 10, 2023 7
Audio Feedback Dynamic mapping of the normalized
distance to target along the z coordinate to harmonic progression.
Map the hand trajectory velocity in the z direction to event density.
Joint Synchrony and Harmonic Progression. Shoulder - woodwind sounds (flute,
clarinet, bassoon) through the progression
Elbow - string sounds (a violin section of tremolo, a violin section, and a pizzicato violincello section).
Mapping of Shoulder and Elbow Extensions Midi velocity (Mv)
Duration (td)
The probability of an octave doubling (Pd)
Memex Seminar, April 10, 2023 8
Visual Feedback
Transition Environment 3D virtual environment Physical movement will control
the virtual environment.
Abstract Environment A picture in a frame Explosion Turbulence Horizontal and Vertical Pull
Memex Seminar, April 10, 2023 9
Validation Offline Segmentation
Reaction Reaching Grasping Returning
Spatial Error Target-Hand Distance Hand Orientation
Arm Openness Should Openness Elbow Openness
Reaching Duration Flow Error
Zero crossing number Polynomial curve fitting
error Consistency
Reaction
Acceleration Deceleration
Adjustment Return
t10 t2 t3 t4 Tt5
Han
d S
peed
Real Cup Virtual CupAbstract I
Abstract II Real Cup
Trial Index
Spa
tial E
rror
Memex Seminar, April 10, 2023 10
Results
Real Cup Virtual CupAbstract I
Abstract II Real Cup
Trial Index
Spa
tial E
rror
Real CupVirtual Cup
Abstract I
Abstract IIReal Cup
Trial Index
Arm
Ope
nnes
s E
rror
Real Cup
Virtual Cup Abstract IAbstract II
Real Cup
Trial Index
Zer
o C
ross
ing
Num
ber
Reaching - our visual-audio feedback design can guide the normal subject to do the reaching as accurately as they did in real world.
Openness - our audio feedback design for the abstract environment can help subjects with more openness.
Flow - the smooth of speed curve means three things: Subjects are clearer the goal and they need not hesitant what will happen. Subjects are clearer about the feedback cue. Based on the current feedback and their
memory, they can easily find the way to reach the target. Subjects start following the rhythm, that is mapped in the audio feedback.
Archival Sub-system
Memex Seminar, April 10, 2023 12
Overview
Challenges: Continuous data streams and large datasets Real-time annotation has high cognitive load We are integrating an archival subsystem into a
team with different domain experts.
Our Approach: Continuous multimodal archival Real-time collaborative annotation Offline information visualization
Memex Seminar, April 10, 2023 13
Archival Subsystem Design
Part of our overall Biofeedback system
Manage multimodal data streams Different data
transport rates (total: 1.89MBps)
Scalable Multicast Network
raises synchronization problem
Memex Seminar, April 10, 2023 14
Continuous Multimodal Archival
We split computational and storage resources into two archival subsystems
1. Archiving parametric system models Raw motion capture data Motion analysis parameters Audio-visual synthesis parameters Data was multicast
2. Contextual media capture Seven channels
• Actual audio-visual feedback data• Three microphones• Video camera
Hardware: soundboard, microphones, VGA monitor scan-converter, video camera, mpeg hardware encoder, due-core server
max / msp graphical program
Memex Seminar, April 10, 2023 15
Database design
Indexing Scheme The patient / session / set / trial hierarchy Universal time stamp of synchronized subsystems
Structural DB tables Motion capture and analysis parameters
categorization Group audio-visual data by feedback semantics
We first stream parametric data into a multi-buffered queue, then write to DB using bulk insert in parallel.
We keep reference of multimedia data. Privacy issue
Memex Seminar, April 10, 2023 16
Real-Time Collaborative Annotation
Why emphasize “real-time collaborative”? Annotations are time critical
Each trial is short (~5 sec.) – there can be many unexpected events in this
period – can cause cognitive overload. Team is focused on the experiment!
Design goals of the annotation tool Distributive Personalized (Multi-disciplinary team) Collaborative
Memex Seminar, April 10, 2023 17
Annotation Interface
Design Elements Dynamic experiment
progression indicators Domain specific checklist Collaborative annotation
sharing• We multicast annotations
from one client to others Random query, retrieval
and modification
User feedback is very positive
Dynamic Trial Progress Bar: red = current trial; green – trials finished; blue - trials not started
yet; pink: trials interrupted by query
Calibration Indicator
Hierarchical Indexing Scheme
Therapist Personalized Checklist
Post Annotation
Receive Annotation
In Real-Time Annotation
In Query/Modification
Memex Seminar, April 10, 2023 18
Information Visualization
Offline visualization for review / annotation of archived data.
Our design goals Hierarchical and selectable
motion parameters / evaluation metrics navigation
Synchronized contextual information playback
Facilitates annotation modification
Helps domain experts share information and improve their subsystems
Memex Seminar, April 10, 2023 19
Visualization Prototype
Features: Allows navigation
through our trial hierarchy on motion analysis results
Contextual media playback with parametric motion analysis visualization
Provides offline annotation facilities
Evaluation: Spatial Error
Evaluation: Reaching Duration
Evaluation: Flow Error
Evaluation: Shoulder Extension
Evaluation: Openness Error
Hierarchy 1: The Entire Biofeedback History
Hierarchy 2: The Session
Hierarchy 4: The Trial
ToolTip Content:(a) session/set/trial(b) evaluation value(c) annotations
Evaluated Session
Annotations appeared as vertically aligned and colored dots
Media Playback
Curves with Mouse Over Highlighted as Red
Selection on Evaluation Metric
Selection on Motion Analysis Parameter
Direct Playback and Annotation Context Menu
Evaluation Metric Indicator
MA Data, Annotation Dots and Text Aligned Exactly
Hierarchy Indicator
Synchronized Motion Parameter Playback with Media Playback; the Scan Line
Multi-Domain Annotation Presentation and Edit Box
Jump From any Hierarchical Level; Direct Context Playback of the Trial
Hierarchy 3: The Set
Hierarchy 5: The Frame
Patient Selection
Memex Seminar, April 10, 2023 20
Open Issues
Event Model Event definition by domain experts Event detection Event Network Modeling
Pre-emptive Annotation Show events with high priority Event log
SenseCam Integration SenseCam pictures can be integrated into our
visualization framework
thanks
Team: Weiwei Xu, Yinpeng Chen, Richard Wallis, Thanassis Rikakis, Hari Sundaram, Todd Ingalls, Loren Olson, Jiping He,
Sharon Liu