8/6/2019 Scoring Model
1/22
1
Issues in Credit Scoring
Model Development and Validation
Dennis Glennon
Risk Analysis Division
Economics Department
The Office of the Comptroller of the Currency
The opinions expressed are those of the author and do not necessarily reflect those of the Office of the
Comptroller of the Currency.
8/6/2019 Scoring Model
2/22
2
Model Development and ValidationModel Development and Validation
Outline
1. Credit Risk vs. Model Risk2. Model Risk Analysis
3. Model Purpose
i. Classification
ii. Prediction
8/6/2019 Scoring Model
3/22
3
Model Review
Scope of a Review
I. Credit Risk The risk to earnings or capital of an obligor's failure to meet the
terms of any contract with the bank or otherwise fail to perform as
agreed.
II. Model Risk
Although model risk contributes to the overall portfolio or creditrisk, it represents a conceptually distinct exposure that emerges
from an overly broad interpretation or application of a model
beyond that for which it was developed.
8/6/2019 Scoring Model
4/22
4
Model Review
Scope of a Review
I. Credit Risk Analysis
i. evaluate strategiesii. assess current portfolio performance
II. Model Risk Analysisi. evaluate model validity, reliability, and
accuracy
8/6/2019 Scoring Model
5/22
5
Model Review
Model Risk Analysis
I. Are the models developed using valid statistical or
industry-accepted methods?
i. Appropriate sample design
a. truncated/censored samplesb. over-sample
8/6/2019 Scoring Model
6/22
6
Model Review
Model Risk Analysis (continued)
ii. Valid model design
a. satisfy minimum statistical requirements
b. in-sample performance (including holdout sample)
c. out-of-time performance
8/6/2019 Scoring Model
7/22
7
Model Review
Model Risk Analysis (continued)
II. Are the models used in ways that are consistent with the
original purpose for which the model was developed?
i. Model purpose
a. classificationb. prediction
8/6/2019 Scoring Model
8/22
8
Model Purpose
Model Purpose
The underlying objective of a classification-based
model is different from that of a prediction model.
As such, a model should be evaluated within thescope of its primary objective.
8/6/2019 Scoring Model
9/22
9
Model Purpose
Models as Classification Tools
Banks are developing or purchasing models that are designed
as classification tools. That is, the models are developed forthe purpose of partitioning populations or portfolios into
groups by their expected relativeperformance.
Modeling Objective: Maximize the divergence or separation betweenthe distributions of good and bad accounts.
8/6/2019 Scoring Model
10/22
10
K-S = 64.0
P erform ance D istribution
0
2000
4000
6000
150
170
190
210
230
250
270
290
score
0
5 0 0 0 0
1 0 0 0 0 01 5 0 0 0 0
2 0 0 0 0 0
2 5 0 0 0 0
b a d s g o o d s
Performance Distribution
0100020003000400050006000
150
170
190
210
230
250
270
290
score
0
50000
100000150000
200000
250000
bads goods
K-S = 26.5
Classification Design: Example
A Comparison of Model PerformanceA Comparison of Model Performance
8/6/2019 Scoring Model
11/22
11
Model Purpose
Classification Objective
Interpretation: If, for example, the good/bad odds ratio
associated with the score interval between 200-210 is 30:1,then the odds ratio for the intervals above (below) 200-210will be greater (less) than 30:1.
Result: A model that maintains its ability to rank-orderperformance is considered to be reliable.
8/6/2019 Scoring Model
12/22
12
Model Purpose
Classification-Based Models
Valid Purpose: models developed under this
criteria are valid as decision tools if the objective isto simply identify segments of the population that,as a group, perform poorly.
Appropriate for identifying and excluding specificsegments of the population -- a strategy that, inpractice, often improves average portfolio performancerelative to a random-selection method.
8/6/2019 Scoring Model
13/22
13
Classification Model
Log Odds Curve
0
1
2
3
4
5
6
7
644 653 665 675 684 693 706 715 725 739 753
Score Bands
ln(good/bad)
Development (K-S = 32.1)
Validation (K-S = 34.3)
ln(20/1) = 3
bad rate = .05
ln(20/1) = 3
bad rate = .05
ln(4/1) = 1.39
bad rate = .20
ln(4/1) = 1.39
bad rate = .20
8/6/2019 Scoring Model
14/22
14
Model Purpose
Alternative Purpose: Predicting Performance.
Banks want models for risk-based pricing/re-pricing
and profitability analysis -- models that are designed
specifically to address the issue of trading risk for
margin (i.e., return).
For that purpose, banks need models that are accurate
predictors of performance.
8/6/2019 Scoring Model
15/22
15
Model Selection: Which model is better?
0
1
y
Score (quintiles)
y
1
020 40 80600 10010 30 50 70 90
010
20 40 60 80 10030 7050 90
Score (quintiles)
9 7 5 3 111 6 5 2 1
1 74 11
951 5 4
3
[0.1][0.08] [0.45] [0.44] [0.67] [0.92]
[0.3] [0.5] [0.7] [0.9]
K-S = 48 K-S = 48
[#B / (#G + #B)]
[bad rate][bad rate]
obs. bad (B) - y=1obs. good (G) - y=0
Model 2Model 1
8/6/2019 Scoring Model
16/22
16
Model Purpose: Prediction
Models as Prediction Tools
Purpose: to predict the expected frequency at
which accounts with similar attributes perform(e.g., respond, attrite, default). For example,predict the probability of default.
Modeling Objective: Minimize the difference betweenthe predicted and actual percentage of defaults withineach score range (i.e., maximize the goodness-of-fit).
8/6/2019 Scoring Model
17/22
17
Model Purpose: Prediction
Prediction-Based Models
Interpretation: If within the interval 200-210 the risk
model predicts a probability of default of .04, then for every100 account that score within that range, four shoulddefault.
A model that satisfies this condition is considered to be
accurate.
This is a much stronger condition than that associated with aclassification objective (i.e., reliable).
8/6/2019 Scoring Model
18/22
18
Model Purpose: Prediction
Prediction-Based Models
Valid Purpose: models developed under this approachare valid as actuarial tools; as such, they are appropriate in
situations in which the actual, not just the relative,
measure of performance is required.
8/6/2019 Scoring Model
19/22
19
Model Purpose: Prediction
Limitations of a Prediction-Based Model
The model-development process is significantly more
complex especially when data across all aspects of thebehavior decision (i.e., individual, market, and industry)
are limited.
8/6/2019 Scoring Model
20/22
20
Model Purpose
Conclusion: Models are developed for different purposes --
e.g., classification or prediction. As such, the choices of:
- sample design,- modeling technique, and
- validation procedures
are driven by the intended purpose for which the model will
ultimately be used.
8/6/2019 Scoring Model
21/22
21
Model Purpose
Observation: The choice of modeling objective isimportant not only because it defines how we assess
its validity, but also because it defines a full set oftechnical estimation procedures that are used toselect the best model under the chosen objective.
8/6/2019 Scoring Model
22/22
22
Issues in Credit Scoring
Model Development and Validation
The End
Model Development and ValidationModel Development and Validation
The EndThe End