Collaborative Annotation, Archival and Visualization in a Biofeedback Rehabilitation system Hari Sundaram Arts Media and Engineering Arizona State University.

Collaborative Annotation, Archival and Visualization in a Biofeedback Rehabilitation system

Hari SundaramArts Media and Engineering

Arizona State University

Memex Seminar, April 10, 2023 2

Introduction

Motivation: Every 45 seconds, someone in the United

States suffers a stroke. It results in functional deficits of neuropsychological and physical functions in post-stroke survivors.

Up to 85% of patients have a sensorimotor deficit in the arm, such as muscle weakness, abnormal muscle tone, abnormal movement synergies, and lack of coordination during voluntary movement

Goal Design a real time multimodal biofeedback

system for stroke patient rehabilitation. Archival / annotation and information

visualization to provide insight.

The Biofeedback system

[To appear in acm mm 2006]


System Overview

The Biofeedback system system situates participants in a multi-sensory engaging environment, where physical actions of the right arm are closely coupled with digital feedback.

The Biofeedback system integrates five computational subsystems. Motion capture Motion analysis Audio feedback Visual feedback Database for archival and annotation

All five subsystems are synchronized with respect to a universal time clock.


Action Analysis Arm Representation

11 labeled markers on arm and torso 3 labeled markers on the back of chair

Feature Extraction 3D hand trajectory / 3D hand trajectory relative

to the predefined straight line Shoulder / Elbow extension Hand Orientation Shoulder rotation / abduction/elevation Trunk flexion / rotation / lean and shoulder

trajectory Wrist extension

Multi-goal Framework Reaching Opening Flow


Coupling Action to Feedback Engagement

Aesthetically attractive, easy to use and intuitive.

Message and Mapping Reaching - visual target, an image

completion/reassembly task, and an accompanying musical progression.

Flow - pointalistic sound clouds in the main musical line, flowing particles in the visuals

Opening - a rich, resonant musical accompaniment.

Environment Introduction ( visual ) Abstract I (visual+audio) Abstract II (visual+audio), more

variation


Audio Feedback Dynamic mapping of the normalized

distance to target along the z coordinate to harmonic progression.

Map the hand trajectory velocity in the z direction to event density.

Joint Synchrony and Harmonic Progression. Shoulder - woodwind sounds (flute,

clarinet, bassoon) through the progression

Elbow - string sounds (a violin section of tremolo, a violin section, and a pizzicato violincello section).

Mapping of Shoulder and Elbow Extensions Midi velocity (Mv)

Duration (td)

The probability of an octave doubling (Pd)


Visual Feedback

Transition Environment 3D virtual environment Physical movement will control

the virtual environment.

Abstract Environment A picture in a frame Explosion Turbulence Horizontal and Vertical Pull


Validation Offline Segmentation

Reaction Reaching Grasping Returning

Spatial Error Target-Hand Distance Hand Orientation

Arm Openness Should Openness Elbow Openness

Reaching Duration Flow Error

Zero crossing number Polynomial curve fitting

error Consistency

Reaction

Acceleration Deceleration

Adjustment Return

t10 t2 t3 t4 Tt5

Han

d S

peed

Real Cup Virtual CupAbstract I

Abstract II Real Cup

Trial Index

Spa

tial E

rror


Results

Real Cup Virtual CupAbstract I

Abstract II Real Cup

Trial Index

Spa

tial E

rror

Real CupVirtual Cup

Abstract I

Abstract IIReal Cup

Trial Index

Arm

Ope

nnes

s E

rror

Real Cup

Virtual Cup Abstract IAbstract II

Real Cup

Trial Index

Zer

o C

ross

ing

Num

ber

Reaching - our visual-audio feedback design can guide the normal subject to do the reaching as accurately as they did in real world.

Openness - our audio feedback design for the abstract environment can help subjects with more openness.

Flow - the smooth of speed curve means three things: Subjects are clearer the goal and they need not hesitant what will happen. Subjects are clearer about the feedback cue. Based on the current feedback and their

memory, they can easily find the way to reach the target. Subjects start following the rhythm, that is mapped in the audio feedback.

Archival Sub-system


Overview

Challenges: Continuous data streams and large datasets Real-time annotation has high cognitive load We are integrating an archival subsystem into a

team with different domain experts.

Our Approach: Continuous multimodal archival Real-time collaborative annotation Offline information visualization


Archival Subsystem Design

Part of our overall Biofeedback system

Manage multimodal data streams Different data

transport rates (total: 1.89MBps)

Scalable Multicast Network

raises synchronization problem


Continuous Multimodal Archival

We split computational and storage resources into two archival subsystems

1. Archiving parametric system models Raw motion capture data Motion analysis parameters Audio-visual synthesis parameters Data was multicast

2. Contextual media capture Seven channels

• Actual audio-visual feedback data• Three microphones• Video camera

Hardware: soundboard, microphones, VGA monitor scan-converter, video camera, mpeg hardware encoder, due-core server

max / msp graphical program


Database design

Indexing Scheme The patient / session / set / trial hierarchy Universal time stamp of synchronized subsystems

Structural DB tables Motion capture and analysis parameters

categorization Group audio-visual data by feedback semantics

We first stream parametric data into a multi-buffered queue, then write to DB using bulk insert in parallel.

We keep reference of multimedia data. Privacy issue


Real-Time Collaborative Annotation

Why emphasize “real-time collaborative”? Annotations are time critical

Each trial is short (~5 sec.) – there can be many unexpected events in this

period – can cause cognitive overload. Team is focused on the experiment!

Design goals of the annotation tool Distributive Personalized (Multi-disciplinary team) Collaborative


Annotation Interface

Design Elements Dynamic experiment

progression indicators Domain specific checklist Collaborative annotation

sharing• We multicast annotations

from one client to others Random query, retrieval

and modification

User feedback is very positive

Dynamic Trial Progress Bar: red = current trial; green – trials finished; blue - trials not started

yet; pink: trials interrupted by query

Calibration Indicator

Hierarchical Indexing Scheme

Therapist Personalized Checklist

Post Annotation

Receive Annotation

In Real-Time Annotation

In Query/Modification


Information Visualization

Offline visualization for review / annotation of archived data.

Our design goals Hierarchical and selectable

motion parameters / evaluation metrics navigation

Synchronized contextual information playback

Facilitates annotation modification

Helps domain experts share information and improve their subsystems


Visualization Prototype

Features: Allows navigation

through our trial hierarchy on motion analysis results

Contextual media playback with parametric motion analysis visualization

Provides offline annotation facilities

Evaluation: Spatial Error

Evaluation: Reaching Duration

Evaluation: Flow Error

Evaluation: Shoulder Extension

Evaluation: Openness Error

Hierarchy 1: The Entire Biofeedback History

Hierarchy 2: The Session

Hierarchy 4: The Trial

ToolTip Content:(a) session/set/trial(b) evaluation value(c) annotations

Evaluated Session

Annotations appeared as vertically aligned and colored dots

Media Playback

Curves with Mouse Over Highlighted as Red

Selection on Evaluation Metric

Selection on Motion Analysis Parameter

Direct Playback and Annotation Context Menu

Evaluation Metric Indicator

MA Data, Annotation Dots and Text Aligned Exactly

Hierarchy Indicator

Synchronized Motion Parameter Playback with Media Playback; the Scan Line

Multi-Domain Annotation Presentation and Edit Box

Jump From any Hierarchical Level; Direct Context Playback of the Trial

Hierarchy 3: The Set

Hierarchy 5: The Frame

Patient Selection


Open Issues

Event Model Event definition by domain experts Event detection Event Network Modeling

Pre-emptive Annotation Show events with high priority Event log

SenseCam Integration SenseCam pictures can be integrated into our

visualization framework

thanks

Team: Weiwei Xu, Yinpeng Chen, Richard Wallis, Thanassis Rikakis, Hari Sundaram, Todd Ingalls, Loren Olson, Jiping He,

Sharon Liu

Collaborative Annotation, Archival and Visualization in a Biofeedback Rehabilitation system Hari Sundaram Arts Media and Engineering Arizona State University.

Documents

memex seminar

biofeedback system system

archival subsystem slide

visualaudio feedback

variation slide

archival annotation

p d slide

opening flow slide