Sheep, Goats, Lambs and Wolves, An Analysis of Individual …simson.net/ref/1998/Sheep_Goats_Lambs_and_Wolves.pdf · Sheep, Goats, Lambs and Wolves An Analysis of Individual Differences

Sheep, Goats, Lambs and WolvesAn Analysis of Individual Differences in

Speaker Recognition Performance11/23/98

These are the slides of the NIST coordinated presentation for the ICSLP '98 Conference in Sidney,Australia, November 1998. They are not part of the official Proceedings of this Conference.

Click here to start

Table of Contents

Sheep, Goats, Lambs and Wolves, AnAnalysis of Individual Differences in SpeakerRecognition Performance

The Hypothetical Menagerie

The Question

The Task and the Data

Speaker Recognition Performance

The Goat Test Statistic, nonparametricKruskal-Wallis rank sum test


Lamb/Wolf Test Statistics

Statistical Significance

Distribution of Errors versus Animal Rankings

Can a Lamb be also a Wolf?

Can a Sheep be also a Wolf?

Author: George R. Doddington

Home Page: http://www.nist.gov/speech

Sheep, Goats, Lambs and Wolves, An Analysis of Individual Differences in Speaker Recognition Performance

http://www.nist.gov/speech/tests/spk/1998/icslp_98/index.htm (1 of 2) [6/23/2002 11:01:32 PM]

http://www.nist.gov/speech

Can a Sheep be also a Lamb?

Conclusion: It’s a ZOO out there!

Sheep, Goats, Lambs and Wolves, An Analysis of Individual Differences in Speaker Recognition Performance

http://www.nist.gov/speech/tests/spk/1998/icslp_98/index.htm (2 of 2) [6/23/2002 11:01:32 PM]

Slide 1 of 14

Sheep, Goats, Lambs and Wolves

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld001.htm [6/23/2002 11:01:41 PM]


Slide 2 of 14




Slide 14 of 14

Conclusion: It’s a out there!



Sheep, Goats, Lambs and Wolves AnAnalysis of Individual Differences inSpeaker Recognition Performance

George Doddington1,2,3,5, Walter Liggett1,Alvin Martin1, Mark Przybocki1, DouglasReynolds3,4

1 National Institute of Standards andTechnology 2 The Johns Hopkins University 3U.S. Department of Defense 4 MIT LincolnLaboratory 5 SRI International

Next slide Back to first slide View graphic version

Sheep, Goats, Lambs and Wolves

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld001.htm [6/23/2002 11:01:45 PM]

Slide 3 of 14

The Question




Typical speakers: The well-behaved majority.Sheep: Speakers who exhibit good true speaker acceptance.❍

●

Problem speakers: The troublesome minorities.Goats: Speakers who are exceptionally unsuccessful at being accepted.❍

Lambs: Speakers who are exceptionally vulnerable to impersonation by others.❍

Wolves: Speakers who are exceptionally successful at impersonating others.❍

●

Previous slide Next slide Back to first slide View graphic version



Slide 4 of 14




The Question

Do these animals really exist? That is, do thespeakers in the population of speakers trulyexhibit individual differences?

●

– OR –

Are the observed performance differencesmerely a result of statistical variance (due tohandset, message content, acousticenvironment, etc.)?

●


The Question


Slide 5 of 14





Text-independent speaker verification: acceptspeaker if score ³ threshold (if x ³ q).

30-second test segments (3400 segments, total)❍

Conversational telephone speech❍

200 men and 220 women (no cross-sex trials)❍

Different handsets used in training and test❍

Data restricted to electret handsets❍

For details, refer to the 1998 Speaker RecognitionEvaluation Plan at URL ftp://jaguar.ncsl.nist.gov/evaluations/speaker/feb98/plans/current_plan.htm




Slide 6 of 14

The Goat Test Statistic







Slide 7 of 14




The Goat Test Statistic nonparametricKruskal-Wallis rank sum test

Each model speaker, Mi, contributes ni testsegments (ni ³ 5).

●

Each of these segments is scored against itsrespective model.

●

Test Procedure

Replace each score by its rank among allscores: xij Þ rij.

●

Sum the segment ranks for each model speaker:●

Compute the Kruskal-Wallis test statistic:●


The Goat Test Statistic


Slide 8 of 14





Typical speakers: The well-behaved majority.Sheep: A speaker who exhibits good true speaker acceptance.❍

●

Problem speakers: The troublesome minorities.Goat: A speaker who is exceptionally unsuccessful at being accepted.❍

Lamb: A model speaker who is exceptionally vulnerable to impersonation by others.❍

Wolf: A segment speaker who is exceptionally successful at impersonating others.❍

●




Slide 9 of 14





Each segment speaker, Si, contributes ni testsegments

●

Each of these segments is scored against allmodels

●

Test Procedure

Compute average impostor scores, xij, for all Siand Mj.

●

Replace each score by its rank among allscores.

●

Sum the ranks for each putative animal:Lamb: Row sum❍

Wolf: Column sum❍

●




Slide 10 of 14

Distribution of Errors




Goatish, lambish and wolfish behaviors were alldemonstrated with confidence > 99%




Slide 11 of 14




Distribution of Errors versus AnimalRankings

Cumulative Errors

Misses for Model Speakers False Alarms for Model Speakers False Alarms for Segment Speakers

Cumulative Trials

ordered by Goat/Lamb/Wolf rank


Distribution of Errors


Slide 12 of 14




Can a Lamb be also a Wolf?Wolf Scores versus Lamb Scores

Wolf Average

Impostor scores

for a segment speaker

averaged over

the best scoring 10%

of model speakers

Lamb Average

Impostor scores for a model speaker averaged over

the best scoring 10% of segment speakers

R2 = 0.63

R2 = 0.34




Slide 13 of 14




Can a Sheep be also a Wolf?Wolf Average

Impostor scores

for a segment speaker averaged over the best scoring 10% of model speakers

Model Average

True Speaker scores for a model speaker

averaged over all model speaker segments

Wolf Scores versus Model Scores

R2 = 0.31

R2 = 0.03




Can a Sheep be also a Lamb?Lamb Average

Impostor scores

for a model speaker averaged over the best scoring 10% of segment speakers

Model Average

True Speaker scores for a model speaker

averaged over all model speaker segments

Lamb Scores versus Model Scores

R2 = 0.01

R2 = 0.09





Goats, Lambs and Wolves exist!

Recommendation:

Direct research toward the minimization of theseundesirable animals.

●

Include measurement of speaker differences aspart of performance evaluation.

An html copy of this presentation is available at URL http://www.nist.gov/speech/icslp_98

●

Previous slide Back to first slide View graphic version



Sheep, Goats, Lambs and Wolves, An Analysis of Individual …simson.net/ref/1998/Sheep_Goats_Lambs_and_Wolves.pdf · Sheep, Goats, Lambs and Wolves An Analysis of Individual Differences

Documents