ROC & AUC, LIFT

ROC & AUC, LIFT

רוזנפלד" אבי ר ד

Introduction to ROC curves

• ROC = Receiver Operating Characteristic• Started in electronic signal detection theory

(1940s - 1950s)• Has become very popular in biomedical

applications, particularly radiology and imaging• מידע בכריית בשימוש גם

False Positives / Negatives

P NP 20 10N 30 90

Predicted

Actu

al

Confusion matrix 1

P NP 10 20N 15 105

Predicted

Actu

al

Confusion matrix 2

FN

FP

Precision (P) = 20 / 50 = 0.4Recall (P) = 20 / 30 = 0.666F-measure=2*.4*.666/1.0666=.5

4

Different Cost Measures• The confusion matrix (easily generalize to multi-class)

• Machine Learning methods usually minimize FP+FN • TPR (True Positive Rate): TP / (TP + FN) = Recall• FPR (False Positive Rate): FP / (TN + FP) = Precision

Predicted classYes No

Actual class

Yes TP: True positive

FN: False negative

No FP: False positive

TN: True negative

Specific Example

Test Result

People with disease

People without disease

Test Result

Call these patients “negative”

Call these patients “positive”

Threshold

Test Result

Call these patients “negative”

Call these patients “positive”

without the diseasewith the disease

True Positives

Some definitions ...

Test Result

Call these patients “negative”

Call these patients “positive”

without the diseasewith the disease

False Positives

Test Result

Call these patients “negative”

Call these patients “positive”

without the diseasewith the disease

True negatives

Test Result

Call these patients “negative”

Call these patients “positive”

without the diseasewith the disease

False negatives

Test Result

without the diseasewith the disease

‘‘-’’

‘‘+’’

Moving the Threshold: left

True

Pos

itive

Rat

e

(Re

call)

0%

100%

False Positive Rate (1-specificity)

0%

100%

ROC curve

ה שינוי של הגרף THRESHOLDההשפעה על

Figure 5.2 A sample ROC curve.

של שונים גרפים ROCסוגים

Area under ROC curve (AUC)

כללי • מדדלגרך • מתחת ROCהשטח

•0.50 , רנדומאלי מחירה .1.0הוא מושלם הוא

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0%

100%

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0%

100%

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0%

100%

AUC = 50%

AUC = 90% AUC =

65%

AUC = 100%

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0%

100%

AUC for ROC curves

Lift Charts

• X axis is sample size: (TP+FP) / N• Y axis is TP

40% of responses for 10% of costLift factor = 4

80% of responses for 40% of costLift factor = 2Model

Random

Lift factor

00.51

1.52

2.53

3.54

4.55 15 25 35 45 55 65 75 85 95

Lift

Sample Size

Lift

Valu

e

המדדים בין הקשר

התרגיל ...לקראת

ואז מודל על ימני לחצןCost / Benefit Analysis for Wood

ה את לראות וגם הסף את לשנות אפשרCONFUSION MATRIX

ה את גם לראות וגם Liftאפשרמחיר השפעת