19-gamut-chi-slides.pdf - Fred Hohman

Steven Drucker Microsoft Research

A Design Probe to Understand How Data Scientists Understand Machine Learning Models

Gamut

CHI 2019

Andrew Head UC BerkeleyFred Hohman

@fredhohman Georgia Tech Rob DeLine

Microsoft Research

Rich Caruana Microsoft Research

!2

While building and deploying ML models is now an increasingly common practice, interpreting models is not.

!3

What is interpretability?


Human understanding of a system’s…



internals e.g., components [Gilpin, 2018]

operations e.g., math [Biran, 2017]

data mapping e.g., input to output [Montavon, 2017]

representation in an explanation [Ribeiro, 2016]



internals e.g., components [Gilpin, 2018]

operations e.g., math [Biran, 2017]

data mapping e.g., input to output [Montavon, 2017]

representation in an explanation [Ribeiro, 2016]

No formal, agreed upon definition [Lipton, 2016]

!5

GDPR (General Data Protection Regulation)

!5

Chapter 3 Section 4

“Automated individual decision-making, including profiling”


Article 22

Right to explanation⚖

!5

Chapter 3 Section 4

“Automated individual decision-making, including profiling”


Article 22

!6

Capabilities of interpretability

Design Probe embodying capabilities

Evaluation & Investigation of probe & emerging practice of interpretability w/ real users

Gamut Contributions

1.

2.

3.

Can we operationalize interpretability?

!7

Contribution 1: Interpretability Capabilities


Formative research with professional data scientists @• 4 senior ML researchers• 5 ML practitioners

!7



Formative research with professional data scientists @• 4 senior ML researchers• 5 ML practitioners

Prompt: In a perfect world, given a machine learning model, what questions would you ask it to help you interpret both the model and its predictions?

!7


Explainable ML Interface QuestionsFrom formative research


Why does this house cost that much?



What is the difference between these two?

$250,000 $250,000




$250,000 $350,000




What if I added…

$ + $

=1 bedroom




What if I added…

What are similar homes?


What if I added…

What are similar homes? ML



Where is it wrong?


What if I added…




Where is it wrong?


What if I added…




Where is it wrong?



What is most important?


What if I added…


Where is it wrong?ML





What if I added…


Where is it wrong?ML





What if I added…


Where is it wrong?


Explainable ML InterfaceWhy does this house cost that much?



What if I added…


Where is it wrong?

Capabilities

Local instance explanations

Feature importance

Instance explanation comparisons

Counterfactuals

Nearest neighbors

Regions of error

C1

C2

C3

C4

C5

C6

From formative research

Explainable ML InterfaceWhy does this house cost that much?



What if I added…


Where is it wrong?

Capabilities

Local instance explanations

Feature importance

Instance explanation comparisons

Counterfactuals

Nearest neighbors

Regions of error

C1

C2

C3

C4

C5

C6

From formative research

Definitions + examples in the paper!

How to test our capabilities?

!30

Contribution 2: Design Probe


!30


Goal: understand emerging practice of model interpretability

Design probe: “instrument that is deployed to find out about the unknown—returning with useful or interesting data.” Balance of design, social science, engineering


[Hutchinson, 2003]

!30


Goal: understand emerging practice of model interpretability

!31

How does our design probe support our capabilities?

House 550$190,606

32

House 550$190,606

BedroomAbvGrLotFrontageBsmtFullBathFullBathBsmtHalfBathtotal

LotAreaSecondFlrSFGrLivAreaKitchenAbvGrYearBuiltPoolAreaOverallQualBsmtFinSFOneWoodDeckSFOpenPorchSFBsmtUnfSFEnclosedPorchTotalBsmtSFMiscValThreeSsnPorchFirstFlrSFGarageYrBltGarageAreaYrSoldLowQualFinSFTotRmsAbvGrdBsmtFinSFTwoMSSubClassOverallCondMoSoldYearRemodAddMasVnrAreaScreenPorchHalfBathGarageCarsFireplaces

33

House 550$190,606

BedroomAbvGr

LotFrontageBsm

tFullBathFullBathBsm

tHalfBathtotal

LotAreaSecondFlrSFGrLivAreaKitchenAbvGrYearBuiltPoolAreaOverallQualBsm

tFinSFOneWoodDeckSF

OpenPorchSFBsm

tUnfSFEnclosedPorchTotalBsm

tSFMiscVal

ThreeSsnPorchFirstFlrSFGarageYrBltGarageAreaYrSoldLowQualFinSFTotRm

sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea

ScreenPorchHalfBathGarageCarsFireplaces

Intercept

020k60k

100k140k180k

House features

Prediction

34

House 550$190,606

BedroomAbvGr

LotFrontageBsm


tHalfBathtotal


tFinSFOneWoodDeckSF

OpenPorchSFBsm


tSFMiscVal


sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea


Intercept

020k60k

100k140k180k

Intercept

020k60k

100k140k180k House 798

$188,620

35

BedroomAbvGr

LotFrontageBsm


tHalfBathtotal


tFinSFOneWoodDeckSF

OpenPorchSFBsm


tSFMiscVal


sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea


Intercept

020k60k

100k140k180k

Intercept

020k60k

100k140k180k

House 550$190,606

House 798$188,620

36

BedroomAbvGr

LotFrontageBsm


tHalfBathtotal


tFinSFOneWoodDeckSF

OpenPorchSFBsm


tSFMiscVal


sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea


Intercept

020k60k

100k140k180k

Intercept

020k60k

100k140k180k

House 550$190,606

House 798$188,620

36

BedroomAbvGr

LotFrontageBsm


tHalfBathtotal


tFinSFOneWoodDeckSF

OpenPorchSFBsm


tSFMiscVal


sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea


Intercept

020k60k

100k140k180k

Intercept

020k60k

100k140k180k

House 550$190,606

House 798$188,620

36

BedroomAbvGr

LotFrontageBsm


tHalfBathtotal


tFinSFOneWoodDeckSF

OpenPorchSFBsm


tSFMiscVal


sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea


Intercept

020k60k

100k140k180k

Intercept

020k60k

100k140k180k

House 550$190,606

House 798$188,620

37

BedroomAbvGr

LotFrontageBsm


tHalfBathtotal


tFinSFOneWoodDeckSF

OpenPorchSFBsm


tSFMiscVal


sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea


Intercept

020k60k

100k140k180k

Intercept

020k60k

100k140k180k

House 550$190,606

House 798$188,620

37

BedroomAbvGr

LotFrontageBsm


tHalfBathtotal


tFinSFOneWoodDeckSF

OpenPorchSFBsm


tSFMiscVal


sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea


Intercept

020k60k

100k140k180k

Intercept

020k60k

100k140k180k

House 550$190,606

House 798$188,620

38

BedroomAbvGr

LotFrontageBsm


tHalfBathtotal


tFinSFOneWoodDeckSF

OpenPorchSFBsm


tSFMiscVal


sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea


Intercept

020k60k

100k140k180k

Intercept

020k60k

100k140k180k

OverallQual: 8Contribution: +22,295

OverallQual: 6Contribution: -14,340

House 550$190,606

House 798$188,620

38

OverallQual

2 4 6 8 10-20k

020k40k60k80k

House 550$190,606

House 798$188,620

BedroomAbvGr

LotFrontageBsm


tHalfBathtotal


tFinSFOneWoodDeckSF

OpenPorchSFBsm


tSFMiscVal


sAbvGrdBsm

tFinSFTwoMSSubClass

OverallCondMoSold

YearRemodAdd

MasVnrArea


Intercept

020k60k100k140k180k

Intercept

020k60k100k140k180k

OverallQual: 8Contribution: +22,295

OverallQual: 6Contribution: -14,340

39

OverallQual

2 4 6 8 10-20k

020k40k60k80k

40

Global explanation Easy to understand:

Average math skills Average graphicacy

High accuracy, realistic

Generalized Additive Model (GAM)

OverallQual

2 4 6 8 10-20k

020k40k60k80k

40

Global explanation Easy to understand:

Average math skills Average graphicacy

High accuracy, realistic

Generalized Additive Model (GAM)

OverallQual

2 4 6 8 10-20k

020k40k60k80k

41

OverallQual

2 4 6 8 10-20k

020k40k60k80k

!42

User Study

!43

Contribution 3: Evaluation and Investigation

User Study

12 data scientists, ~1.5 hours each

!43


User Study


Think-aloud + answering questions:1. data & model questions they wrote before seeing Gamut2. prepared questions by us

!43


User Study


Think-aloud + answering questions:1. data & model questions they wrote before seeing Gamut2. prepared questions by us

Tutorial → Study → Interview

!43


Research Questions

!44

What we want to investigate using Gamut

Research Questions

RQ1. Reasons for Model InterpretabilityWhy do data scientists need interpretability and how do they use it in Gamut?

"

!44


Research Questions


RQ2. Global v. Local ExplanationsHow do data scientists use different explanation paradigms?

"

#

!44


Research Questions


RQ2. Global v. Local ExplanationsHow do data scientists use different explanation paradigms?

RQ3. Interactive ExplanationsHow does interactivity play a role in explainable machine learning interfaces?

"

#

⚡

!44


RQ1. Interpretability Needs and Usage "

“…figure out what you want emphasize and what you want to minimize. Know your audience and purpose.”

Communication is a spectrum.

!45



“I want to understand bit by bit how the dataset features work with each other, influence each other.”

Model building and debugging to boost accuracy.

!46



“This would help me get to valuable nuggets of information, which is what [my stakeholders] are ultimately interested in.”

Data understanding > model deployment.

!47



But… eager to rationalize explanations; troublesome without healthy skepticism.

Hypothesis generation to help build trust.

!48


RQ2. Global v. Local Explanations #

!49



!49



!49

Global features + model



!49


Local single instances



!49



ML novice [1-3 years]



!49




ML familiars [3-5 years]



!49




ML familiars [3-5 years]

ML experts [5+ years]


RQ3. Interactive Explanations ⚡

!50



Primary mechanism for exploring,comparing, and explaining predictions

!50




Converse with a model

!50




Converse with a model

Could not conceive of non-interactive

!50


Takeaways

!51

Consider interpretability capabilities for your interfaces Interpretability is not a singular, rigid concept

Takeaways

!51

"


Takeaways

!51

"

# Tailor explanations for specific audiences Balance simplicity and completeness


Takeaways

!51

"

#

⚡

Tailor explanations for specific audiences Balance simplicity and completeness

Design and integrate effective interaction Interaction key to realizing interpretability & solidify model understanding [Weld & Bansal, 2018]

Fred Hohman @fredhohman Georgia Tech

A Design Probe to Understand How Data Scientists Understand Machine Learning Models

bit.ly/gamut-chiGamut

Steven Drucker Microsoft Research

Andrew Head UC Berkeley

Rob DeLine Microsoft Research

Rich Caruana Microsoft Research

Thanks!

% paper & video ' blog ( slides

extra slides

!53

!56

General Linear Model

!56


!56


!56


!56


!56


!56


!56


Generalized Additive Model

!56



!56



!56

nonlinear, or “bendy” [Jones & Almond, 1992]



!56

19-gamut-chi-slides.pdf - Fred Hohman

Documents