Cerias talk on testing and evaluation

BIOMETRICS – TESTING

AND EVALUATION

• Traditional biometric testing

• Algorithm testing

• Well established metrics

• Well understood testing methodologies

• Operational testing

• Harder to do

• Access to environments

• Test methodologies dependent in some cases on the test

BIOMETRIC TESTING

•Essentially trying to understand how a system

performs

• Maybe more fundamentally – who or what is

causing the errors?

TESTING AND EVALUATION

PROBABLY NOT MISSING,

BUT HARD TO DO

• There have been several papers on the contribution of individual error on performance

• What causes these errors?

• There are some papers that examine meta-data and the contribution of variables (age) and examines the training of algorithms

GAPS IN BIOMETRIC TESTING

•Training

• How do users get accustomed to devices?

• Can they remember how to use them?

• How do we provide good training to the users that has a consistent message?

GAPS IN BIOMETRIC TESTING

• Accessibility

• How many people know people who have problems with interacting with a biometric systems?

• How do we deal with accessibility and usability issues?

• Hearing and sight issues

GAPS IN BIOMETRIC TESTING

• Human Factors

• Testing and evaluating biometric systems by looking at how the users interact with the system

• Are performance results different in an operational environment than collect in a lab?

• Are these performance results due to the environment?

GAPS IN BIOMETRIC TESTING

• Is the error always subject centric?

• The role of the device?

• The role of the operator?

GAPS IN BIOMETRIC TESTING

FILLING IN THOSE GAPSA WORK IN PROGRESS

• Devices are:

• Now more complex

• Variety of form factors, deployments, applications

• In the hands of more customers

• They become more demanding

• Technology development cycle is short

WORK IN PROGRESS

• What are biometrics?

• Why should we test?

• Complexities of testing today

• Examples of testing

• Operational / Scenario / Hybrid

• Our approach to testing

• Upcoming tests

TODAY – AN UPDATE TO TESTING AND

EVALUATION

•How do we identify people?

•As we move around, the need to identify

individuals increases

INITIAL CONCEPTS

•Biometric – of or having to do with biometrics

•Biometrics – automated recognition of

individuals based on their behavioral and

biological characteristics

TWO MAIN DEFINITIONS

•Biometrics appear in a number of different

environments:

• Large scale government programs

• Individual identification on a laptop

INITIAL CONCEPTS

•Border control – US VISIT and Global Entry

•Apple fingerprint sensors

TWO EXAMPLES

•How do we test and evaluate these products to

make sure that we have good performance?

TWO VERY DIFFERENT SCENARIOS

• International Center for Biometric Research, and its predecessors have been testing and evaluating biometric products for over 14 years.

• The devices have changed over time, but the needs of testing and evaluation haven’t

• Companies want to test and evaluate products to prove that they will work in the marketplace

ICBR HISTORY

•Anyone know what this

is?

BIOMETRIC DEVICES AND

MODALITIES

• Iris camera

•Now what year was this

launched?

BIOMETRIC DEVICES AND

MODALITIES

• Iris camera

•2001

BIOMETRIC DEVICES AND

MODALITIES

• We have tested mobile iris, eye vein, voice, signature, face, palm, and two-factor authentications, all since 2001

• However, today, testing and evaluation is a lot more complex

• Biometrics has a huge market opportunity, but to deploy successfully, we need to test

SO MOBILE ISNT NEW!

TESTING AND EVALUATION

– THINGS TO CONSIDER

• What is the purpose of the test?

• To get an idea of the performance of the biometric

• To get an idea of the usability

• To calculate the throughput of an ABC gate

• To understand the differences of whether there is a difference in one group or another

• Take the example of the phone versus the border gate – what are we interested in?

WHAT ARE WE EVALUATING

•Understanding the problem statement is key

for a good test protocol

•Tests are expensive and time consuming, and

we want to be able to collect the data

efficiently, accurately, and within budget

WHAT ARE WE EVALUATING

• Discuss with the clients the different aspects of evaluation that you can do

• Such as straight performance through to understanding the issues relating to performance

• Sample size – the complexity of the test will drive the sample size

WHAT ARE WE EVALUATING

•Scenario

•Technology

•Operational

DIFFERENT TYPES OF TESTS

•The matching algorithms are then tested on a

sequestered dataset

TECHNOLOGY EVALUATIONS

•Takes place in a testing facility

•Matchers are installed in an “office

environment” and biometric devices are tested

SCENARIO EVALUATIONS

•Volunteers are recruited

•Group uses the system to collect over time

•This creates a set of databases that can be

used for technology evaluations later on

SCENARIO EVALUATIONS

• From Mansfield and Wayman:

• Provide a framework for developing and fully describing test protocols

• Help avoid systematic bias due to incorrect data collection or analytic procedures in evaluation

• To help testers achieve the best possible estimate of field performance while expanding the minimum effort in conducting their evaluation

• To improve the understanding of the limits of applicability of test results and test methods

SCENARIO EVALUATIONS

•This type of evaluation is used to determine

the performance of a biometric system in a

real-world environment

OPERATIONAL EVALUATION

• A combination of technology and scenario testing

• We want to solve these additional problems of what are these biometric system errors, and how can we fix them

• So we do:

• test protocol development, human subject testing, many different modalities

• Usability testing

• Surveys

• Focus groups

OUR TESTING

USABILITY TESTING

HUMAN BIOMETRIC SENSOR

INTERACTION

HBSI MODEL

•ABC gates are important, but these are

complicated from the perspective of biometric

testing

HBSI MODEL ADVANCES

•Token- the passport

•People traveling- potential incorrect passport

•Throughput issues- how to measure

CHALLENGES

TOKEN HBSI MODEL

Token HBSI Model

TOKEN HBSI MODEL

TOKEN HBSI MODEL

TOKEN HBSI MODEL

• HBSI and Border Gates

• Building a border gate in the center to test various technologies, including iris, fingerprint, documents

• Throughput research

• Usability using the Kinect

• Novel performance metrics such as the Stability Score Index

TESTING AND EVALUATION RESEARCH

• Mobile devices are important as well, and we continue to work in this area:

• Wild testing

• Illumination / Noise / Controlled environments

• Voice, face, signature, palm, and multi-factor authentication

TESTING AND EVALUATION RESEARCH

QUESTIONS