STARDUST – Speech Training And Recognition for Dysarthric Users of Assistive Technology Mark Hawley et al Barnsley District General Hospital and University.

STARDUST – Speech Training And Recognition for Dysarthric Users of

Assistive Technology

Mark Hawley et alBarnsley District General Hospital and

University of Sheffield

STARDUST

• To develop speech-driven environmental control and voice output communication devices for people with dysarthria

– To develop a reliable small vocabulary speech recogniser for dysarthric speakers

– To develop a computer training program to help to stabilise the speech of dysarthric speakers

Research Team

• Department of Medical Physics and Clinical Engineering, Barnsley District General Hospital (Mark Hawley, Simon Brownsell, Stuart Cunningham)

• Institute of General Practice and Primary Care, University of Sheffield (Pam Enderby, Mark Parker, Rebecca Palmer)

• Department of Computer Science, University of Sheffield (Phil Green, Nassos Hatzis, James Carmichael)

• Project funded by Dept of Health New and Emerging Applications of Technology (NEAT) programme

Dysarthria

A neurological motor speech impairment characterised by slow, weak, imprecise and/or uncoordinated

movements of the speech musculature.

Speech is often difficult to understand (unintelligible) and variable (inconsistent)

Frequently associated with other physical disabilities

Severe = <40% intelligible

Speech-input writing programmes

• Normal speech - with recognition training can get >90% recognition rates (Rose and Galdo, 1999)

• Mild dysarthric speech - 10-15% lower recognition rates (Ferrier, 1992)

• Recognition declines as speech deteriorates - by 30-50% for single words (Thomas-Stonell, 1998, Hawley 2002)

Performance of a commercial speaker-dependent recogniser

(in ‘ideal’ conditions)

Recognition rateN=6

Dysarthric subject 1 60%

Dysarthric subject 2 80%

‘Normal’ control 98%

Difficult speech recognition problem

• Dysarthric speech– different to ‘normal’ models– more variable than ‘normal’ speech, both

between and within speakers– difficult to collect large corpus of speech

STARDUST

• To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria

• To develop a reliable small vocabulary speech recogniser for dysarthric speakers

• To develop a computer training program to help to stabilise the speech of dysarthric speakers(ie improve consistency) and improve recognition

Intelligibility and Consistency

• ‘Normal’ speech will be almost 100% intelligible and with few articulatory differences over time (consistent).

• ‘Severe’ dysarthria may be completely unintelligible to the naïve listener and will show high variability (inconsistent)– but may show consistency of key elements which will

make it more intelligible to the familiar listener.

• STARDUST is concerned with consistency

Training program

• Visual feedback to improve consistency at word level– Quantitative – Real time

• To be used by the client alone or with carer or therapist

• Training tool records speech - used to build recogniser

Training program set-up

• Record 10 examples of each word to be trained• Program builds models of words based on

examples• For each word, program selects example that best

matches its model (the best-fit recording)• Program feeds back a measure of the match

between last utterance and model

Recogniser

Score calculation

Microphone

PCSwitch

Outcome of speech training(preliminary data)

0 2 4 6 8 10 12 14 16 18 20 22-69

-68

-67

-66

-65

-64

Session Number

Mean L

og P

robabili

ty

0 1 2 3 4 5 6 7 8 9-74

-73

-72

-71

-70

-69

-68

-67

-66

-65

-64

Session Number

Mean L

og P

robabili

ty

In a group of 5 users, 3 showed an upward trend, 2 showed no upward trend

STARDUST

• To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria

• To develop a reliable small vocabulary speech recogniser for dysarthric speakers

• To develop a computer training program to help to stabilise the speech of dysarthric speakers

Recognition technology

• Small vocabulary

• Speaker dependent

• uses hidden Markov models• based on HTK (University of Cambridge)

STARDUST recogniser performance (N=number of words used for training)

IntelligibilitySingle words-sentences

STARDUSTrecogniser

N=6

STARDUSTrecogniser

N=20

STARDUSTrecogniser

N=28

CommercialspeechrecognitionECSN=6

Subject 1 0% - 0% 64% 80% 85% 60%

Subject 2 22% - 34% 100% 100% 100% 80%

Control 100% 100% 100% 100% 98%

STARDUST

• To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria

• To develop a reliable small vocabulary speech recogniser for dysarthric speakers

• To develop a computer training program to help to stabilise the speech of dysarthric speakers

Recogniser

Look-up table

Speech

synthesiser or

recording

Microphone

PCSwitch

Vocabulary mapping

• One to one (word to phrase) mapping– ‘Want’ = I need something, could you help me?

• Pseudo-grammatical combinations– ‘Want ... drink’ = Could I have a drink, please?

• Coding– ‘3…6…4’ = I went to Spain for my holidays

– nm possible combinations, where n is no of words in vocab, m is length of vocabulary string

Recogniser

Look-up table

Microphone

PC

Infra-red

Switch

Work in progress

• Test systems in home-based field trials– acceptability– usability (eg speed of access)– accuracy– reliability– practicality

Work in progress

• Remove switch activation of recogniser

• Increase vocabularies to test limits of recogniser

• Develop tools for clinicians to build and test individual configurations

STARDUST - conclusions

• Recogniser that recognises severely dysarthric speech

• Computer-based training program to improve recognition and consistency – word level (and sub-word level in future)

– collects lots of speech data for recogniser

• Developed demonstrators of environmental control and voice-output device– next step to test in real usage