Top Banner

Click here to load reader

Sheep, Goats, Lambs and Wolves, An Analysis of Individual · PDF fileSheep, Goats, Lambs and Wolves An Analysis of Individual Differences in Speaker Recognition Performance 11/23/98

Mar 03, 2019

ReportDownload

Documents

dinhduong

Sheep, Goats, Lambs and WolvesAn Analysis of Individual Differences in

Speaker Recognition Performance11/23/98

These are the slides of the NIST coordinated presentation for the ICSLP '98 Conference in Sidney,Australia, November 1998. They are not part of the official Proceedings of this Conference.

Click here to start

Table of Contents

Sheep, Goats, Lambs and Wolves, AnAnalysis of Individual Differences in SpeakerRecognition Performance

The Hypothetical Menagerie

The Question

The Task and the Data

Speaker Recognition Performance

The Goat Test Statistic, nonparametricKruskal-Wallis rank sum test

The Hypothetical Menagerie

Lamb/Wolf Test Statistics

Statistical Significance

Distribution of Errors versus Animal Rankings

Can a Lamb be also a Wolf?

Can a Sheep be also a Wolf?

Author: George R. Doddington

Home Page: http://www.nist.gov/speech

Sheep, Goats, Lambs and Wolves, An Analysis of Individual Differences in Speaker Recognition Performance

http://www.nist.gov/speech/tests/spk/1998/icslp_98/index.htm (1 of 2) [6/23/2002 11:01:32 PM]

http://www.nist.gov/speech

Can a Sheep be also a Lamb?

Conclusion: Its a ZOO out there!

Sheep, Goats, Lambs and Wolves, An Analysis of Individual Differences in Speaker Recognition Performance

http://www.nist.gov/speech/tests/spk/1998/icslp_98/index.htm (2 of 2) [6/23/2002 11:01:32 PM]

Slide 1 of 14

Sheep, Goats, Lambs and Wolves

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld001.htm [6/23/2002 11:01:41 PM]

http://www.nist.gov/speech

Slide 2 of 14

The Hypothetical Menagerie

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld002.htm [6/23/2002 11:01:43 PM]

http://www.nist.gov/speech

Slide 14 of 14

Conclusion: Its a out there!

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld014.htm [6/23/2002 11:01:45 PM]

http://www.nist.gov/speech

Sheep, Goats, Lambs and Wolves AnAnalysis of Individual Differences inSpeaker Recognition Performance

George Doddington1,2,3,5, Walter Liggett1,Alvin Martin1, Mark Przybocki1, DouglasReynolds3,4

1 National Institute of Standards andTechnology 2 The Johns Hopkins University 3U.S. Department of Defense 4 MIT LincolnLaboratory 5 SRI International

Next slide Back to first slide View graphic version

Sheep, Goats, Lambs and Wolves

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld001.htm [6/23/2002 11:01:45 PM]

Slide 3 of 14

The Question

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld003.htm [6/23/2002 11:01:47 PM]

http://www.nist.gov/speech

The Hypothetical Menagerie

Typical speakers: The well-behaved majority.Sheep: Speakers who exhibit good true speaker acceptance.

Problem speakers: The troublesome minorities.Goats: Speakers who are exceptionally unsuccessful at being accepted.

Lambs: Speakers who are exceptionally vulnerable to impersonation by others.

Wolves: Speakers who are exceptionally successful at impersonating others.

Previous slide Next slide Back to first slide View graphic version

The Hypothetical Menagerie

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld002.htm [6/23/2002 11:01:48 PM]

Slide 4 of 14

The Task and the Data

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld004.htm [6/23/2002 11:01:50 PM]

http://www.nist.gov/speech

The Question

Do these animals really exist? That is, do thespeakers in the population of speakers trulyexhibit individual differences?

OR

Are the observed performance differencesmerely a result of statistical variance (due tohandset, message content, acousticenvironment, etc.)?

Previous slide Next slide Back to first slide View graphic version

The Question

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld003.htm [6/23/2002 11:01:50 PM]

Slide 5 of 14

Speaker Recognition Performance

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld005.htm [6/23/2002 11:01:52 PM]

http://www.nist.gov/speech

The Task and the Data

Text-independent speaker verification: acceptspeaker if score threshold (if x q).

30-second test segments (3400 segments, total)

Conversational telephone speech

200 men and 220 women (no cross-sex trials)

Different handsets used in training and test

Data restricted to electret handsets

For details, refer to the 1998 Speaker RecognitionEvaluation Plan at URL ftp://jaguar.ncsl.nist.gov/evaluations/speaker/feb98/plans/current_plan.htm

Previous slide Next slide Back to first slide View graphic version

The Task and the Data

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld004.htm [6/23/2002 11:01:55 PM]

Slide 6 of 14

The Goat Test Statistic

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld006.htm [6/23/2002 11:01:57 PM]

http://www.nist.gov/speech

Speaker Recognition Performance

Previous slide Next slide Back to first slide View graphic version

Speaker Recognition Performance

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld005.htm [6/23/2002 11:01:58 PM]

Slide 7 of 14

The Hypothetical Menagerie

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld007.htm [6/23/2002 11:01:59 PM]

http://www.nist.gov/speech

The Goat Test Statistic nonparametricKruskal-Wallis rank sum test

Each model speaker, Mi, contributes ni testsegments (ni 5).

Each of these segments is scored against itsrespective model.

Test Procedure

Replace each score by its rank among allscores: xij rij.

Sum the segment ranks for each model speaker:

Compute the Kruskal-Wallis test statistic:

Previous slide Next slide Back to first slide View graphic version

The Goat Test Statistic

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld006.htm [6/23/2002 11:02:00 PM]

Slide 8 of 14

Lamb/Wolf Test Statistics

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld008.htm [6/23/2002 11:02:01 PM]

http://www.nist.gov/speech

The Hypothetical Menagerie

Typical speakers: The well-behaved majority.Sheep: A speaker who exhibits good true speaker acceptance.

Problem speakers: The troublesome minorities.Goat: A speaker who is exceptionally unsuccessful at being accepted.

Lamb: A model speaker who is exceptionally vulnerable to impersonation by others.

Wolf: A segment speaker who is exceptionally successful at impersonating others.

Previous slide Next slide Back to first slide View graphic version

The Hypothetical Menagerie

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld007.htm [6/23/2002 11:02:03 PM]

Slide 9 of 14

Statistical Significance

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld009.htm [6/23/2002 11:02:05 PM]

http://www.nist.gov/speech

Lamb/Wolf Test Statistics

Each segment speaker, Si, contributes ni testsegments

Each of these segments is scored against allmodels

Test Procedure

Compute average impostor scores, xij, for all Siand Mj.

Replace each score by its rank among allscores.

Sum the ranks for each putative animal:Lamb: Row sum

Wolf: Column sum

Previous slide Next slide Back to first slide View graphic version

Lamb/Wolf Test Statistics

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld008.htm [6/23/2002 11:02:06 PM]

Slide 10 of 14

Distribution of Errors

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld010.htm [6/23/2002 11:02:07 PM]

http://www.nist.gov/speech

Statistical Significance

Goatish, lambish and wolfish behaviors were alldemonstrated with confidence > 99%

Previous slide Next slide Back to first slide View graphic version

Statistical Significance

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld009.htm [6/23/2002 11:02:08 PM]

Slide 11 of 14

Can a Lamb be also a Wolf?

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld011.htm [6/23/2002 11:02:09 PM]

http://www.nist.gov/speech

Distribution of Errors versus AnimalRankings

Cumulative Errors

Misses for Model Speakers False Alarms for Model Speakers False Alarms for Segment Speakers

Cumulative Trials

ordered by Goat/Lamb/Wolf rank

Previous slide Next slide Back to first slide View graphic version

Distribution of Errors

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld010.htm [6/23/2002 11:02:10 PM]

Slide 12 of 14

Can a Sheep be also a Wolf?

http://www.nist.gov/speech/tests/spk/1998/icslp_98/sld012.htm [6/23/2002 11:02:12 PM]

http://www.nist.gov/speech

Can a Lamb be also a Wolf?Wolf Scores versus Lamb Scores

Wolf Average

Impostor scores

for a segment speaker

averaged over

the best scoring 10%

of model speakers

Lamb Average

Impostor scores for a model speaker averaged over

the best scoring 10% of segment speakers

R2 = 0.63

R2 = 0.34

Previous slide Next slide Back to first slide View graphic version

Can a Lamb be also a Wolf?

http://www.nist.gov/speech/tests/spk/1998/icslp_98/tsld011.htm [6/23/2002 11:02:12 PM]

Slide 13 of 14

Can a Sheep be also a Lamb?

http://www.nist.gov/speech/tests/spk/1998/ics

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.