Page 1
The validation processLiterature review
Methodological proposals
The validation of Credit Rating and ScoringModels
Raffaella Calabrese
[email protected] of Milano-Bicocca
Swiss Statistics MeetingGeneva, SwitzerlandOctober 29th, 2009
Raffaella Calabrese Validation of internal rating systems
Page 2
The validation processLiterature review
Methodological proposals
Outline
1 The validation process
2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
3 Methodological proposalsCurve of Classification Error Costs and Error Costs
Raffaella Calabrese Validation of internal rating systems
Page 3
The validation processLiterature review
Methodological proposals
Outline
1 The validation process
2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
3 Methodological proposalsCurve of Classification Error Costs and Error Costs
Raffaella Calabrese Validation of internal rating systems
Page 4
The validation processLiterature review
Methodological proposals
Outline
1 The validation process
2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
3 Methodological proposalsCurve of Classification Error Costs and Error Costs
Raffaella Calabrese Validation of internal rating systems
Page 5
The validation processLiterature review
Methodological proposals
Preliminary definitions
Credit rating and scoring models estimate the creditobligor’s worthiness and provide an assessment of theobligor’s future status.
The discriminatory power of a rating or scoring modeldenotes its ability to discriminate ex ante betweendefaulting and non-defaulting borrowers.
The validation process assesses the discriminatory powerof a rating or scoring model
Raffaella Calabrese Validation of internal rating systems
Page 6
The validation processLiterature review
Methodological proposals
Preliminary definitions
Credit rating and scoring models estimate the creditobligor’s worthiness and provide an assessment of theobligor’s future status.
The discriminatory power of a rating or scoring modeldenotes its ability to discriminate ex ante betweendefaulting and non-defaulting borrowers.
The validation process assesses the discriminatory powerof a rating or scoring model
Raffaella Calabrese Validation of internal rating systems
Page 7
The validation processLiterature review
Methodological proposals
Preliminary definitions
Credit rating and scoring models estimate the creditobligor’s worthiness and provide an assessment of theobligor’s future status.
The discriminatory power of a rating or scoring modeldenotes its ability to discriminate ex ante betweendefaulting and non-defaulting borrowers.
The validation process assesses the discriminatory powerof a rating or scoring model
Raffaella Calabrese Validation of internal rating systems
Page 8
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Each borrower is characterized by two random variables:
the score S assigned to the borrower is a continuous r. v.with support (−∞,∞)
the Bernoulli r.v. B represents the borrower’s state at theend of a fixed time-period
B =
{
1, the borrower’s state is default (d );0, the borrower’s state is non default (n).
The conditional distribution functions of S given a value of Bare denoted respectively by Fd(·) and Fn(·).
Raffaella Calabrese Validation of internal rating systems
Page 9
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Each borrower is characterized by two random variables:
the score S assigned to the borrower is a continuous r. v.with support (−∞,∞)
the Bernoulli r.v. B represents the borrower’s state at theend of a fixed time-period
B =
{
1, the borrower’s state is default (d );0, the borrower’s state is non default (n).
The conditional distribution functions of S given a value of Bare denoted respectively by Fd(·) and Fn(·).
Raffaella Calabrese Validation of internal rating systems
Page 10
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Each borrower is characterized by two random variables:
the score S assigned to the borrower is a continuous r. v.with support (−∞,∞)
the Bernoulli r.v. B represents the borrower’s state at theend of a fixed time-period
B =
{
1, the borrower’s state is default (d );0, the borrower’s state is non default (n).
The conditional distribution functions of S given a value of Bare denoted respectively by Fd(·) and Fn(·).
Raffaella Calabrese Validation of internal rating systems
Page 11
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
The distribution function of the score S is
F (s) = pFd(s) + (1 − p)Fn(s)
where p is the probability of default p = P[B = d ].
The accuracy (AC) is
AC = pFd(s) + (1 − p)[1 − Fn(s)] = 2pFd(s) − F (s) + (1 − p)
Raffaella Calabrese Validation of internal rating systems
Page 12
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
The distribution function of the score S is
F (s) = pFd(s) + (1 − p)Fn(s)
where p is the probability of default p = P[B = d ].
The accuracy (AC) is
AC = pFd(s) + (1 − p)[1 − Fn(s)] = 2pFd(s) − F (s) + (1 − p)
Raffaella Calabrese Validation of internal rating systems
Page 13
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Actual default Actual non defaultPredicted default True Default (TD) False Default (FD)(below s) Type II errorPredicted non default False Non Default (FN) True Non Default(above s) Type I error (TN)
Nd Nn
hit rate F̂d(s) =TDNd
false alarm rate F̂n(s) =FDNn
F̂ (s) = p̂F̂d(s) + (1 − p̂)F̂n(s) =TD + FDNd + Nn
where p̂ =Nd
Nd + Nn
Raffaella Calabrese Validation of internal rating systems
Page 14
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Actual default Actual non defaultPredicted default True Default (TD) False Default (FD)(below s) Type II errorPredicted non default False Non Default (FN) True Non Default(above s) Type I error (TN)
Nd Nn
hit rate F̂d(s) =TDNd
false alarm rate F̂n(s) =FDNn
F̂ (s) = p̂F̂d(s) + (1 − p̂)F̂n(s) =TD + FDNd + Nn
where p̂ =Nd
Nd + Nn
Raffaella Calabrese Validation of internal rating systems
Page 15
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Actual default Actual non defaultPredicted default True Default (TD) False Default (FD)(below s) Type II errorPredicted non default False Non Default (FN) True Non Default(above s) Type I error (TN)
Nd Nn
hit rate F̂d(s) =TDNd
false alarm rate F̂n(s) =FDNn
F̂ (s) = p̂F̂d(s) + (1 − p̂)F̂n(s) =TD + FDNd + Nn
where p̂ =Nd
Nd + Nn
Raffaella Calabrese Validation of internal rating systems
Page 16
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Actual default Actual non defaultPredicted default True Default (TD) False Default (FD)(below s) Type II errorPredicted non default False Non Default (FN) True Non Default(above s) Type I error (TN)
Nd Nn
hit rate F̂d(s) =TDNd
false alarm rate F̂n(s) =FDNn
F̂ (s) = p̂F̂d(s) + (1 − p̂)F̂n(s) =TD + FDNd + Nn
where p̂ =Nd
Nd + Nn
Raffaella Calabrese Validation of internal rating systems
Page 17
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
curve: CAP(u) = Fd [F−1(u)], u ∈ (0, 1)
Raffaella Calabrese Validation of internal rating systems
Page 18
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
curve: CAP(u) = Fd [F−1(u)], u ∈ (0, 1)
Raffaella Calabrese Validation of internal rating systems
Page 19
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
synthetic index (BCBS, 2005):
AR =aR
aR + aQAR ∈ [0, 1]
optimal cut-off score (Hong, 2009): the intersection of theCAP curve and the iso-performance tangent line
Fd(s) =1
2p[F (s) + AC + p − 1]
drawbacks:- dependence on the sample relative frequency of defaulted
borrowers;- the type II error and the costs of wrong classification are
ignored.
Raffaella Calabrese Validation of internal rating systems
Page 20
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
synthetic index (BCBS, 2005):
AR =aR
aR + aQAR ∈ [0, 1]
optimal cut-off score (Hong, 2009): the intersection of theCAP curve and the iso-performance tangent line
Fd(s) =1
2p[F (s) + AC + p − 1]
drawbacks:- dependence on the sample relative frequency of defaulted
borrowers;- the type II error and the costs of wrong classification are
ignored.
Raffaella Calabrese Validation of internal rating systems
Page 21
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
synthetic index (BCBS, 2005):
AR =aR
aR + aQAR ∈ [0, 1]
optimal cut-off score (Hong, 2009): the intersection of theCAP curve and the iso-performance tangent line
Fd(s) =1
2p[F (s) + AC + p − 1]
drawbacks:- dependence on the sample relative frequency of defaulted
borrowers;- the type II error and the costs of wrong classification are
ignored.
Raffaella Calabrese Validation of internal rating systems
Page 22
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
synthetic index (BCBS, 2005):
AR =aR
aR + aQAR ∈ [0, 1]
optimal cut-off score (Hong, 2009): the intersection of theCAP curve and the iso-performance tangent line
Fd(s) =1
2p[F (s) + AC + p − 1]
drawbacks:- dependence on the sample relative frequency of defaulted
borrowers;- the type II error and the costs of wrong classification are
ignored.
Raffaella Calabrese Validation of internal rating systems
Page 23
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Receiver Operating Characteristic (ROC) Curve andArea Under the Curve (AUC)
Curve: ROC(u) = Fd [F−1(u)], u ∈ (0, 1)
Raffaella Calabrese Validation of internal rating systems
Page 24
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Receiver Operating Characteristic (ROC) Curve andArea Under the Curve (AUC)
Curve: ROC(u) = Fd [F−1(u)], u ∈ (0, 1)
Raffaella Calabrese Validation of internal rating systems
Page 25
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Synthetic index (BCBS, 2005):
AUC =
∫ 1
0Fd [Fn(s)]dFn(s) AUC ∈ [0.5, 1]
Optimal cut-off score (Hong, 2009): the intersection of theROC curve and the iso-performance tangent line
Fd(s) =1 − p
pFn(s) +
1p
(AC + p − 1).
drawbacks: the costs of wrong classification are ignored.
Raffaella Calabrese Validation of internal rating systems
Page 26
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Synthetic index (BCBS, 2005):
AUC =
∫ 1
0Fd [Fn(s)]dFn(s) AUC ∈ [0.5, 1]
Optimal cut-off score (Hong, 2009): the intersection of theROC curve and the iso-performance tangent line
Fd(s) =1 − p
pFn(s) +
1p
(AC + p − 1).
drawbacks: the costs of wrong classification are ignored.
Raffaella Calabrese Validation of internal rating systems
Page 27
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Synthetic index (BCBS, 2005):
AUC =
∫ 1
0Fd [Fn(s)]dFn(s) AUC ∈ [0.5, 1]
Optimal cut-off score (Hong, 2009): the intersection of theROC curve and the iso-performance tangent line
Fd(s) =1 − p
pFn(s) +
1p
(AC + p − 1).
drawbacks: the costs of wrong classification are ignored.
Raffaella Calabrese Validation of internal rating systems
Page 28
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Curve of Classification Error Costs (CEC) and ErrorCosts (EC)
Curve:
C[u] =CFN
2{1 − Fd [F−1(u)]} +
CFD
2Fn[F−1(u)] u ∈ (0, 1)
Synthetic index:
EC∗ =
∫ 1
0C[F (s)]dF (s)
EC =EC∗ − EC∗
R
EC∗
P − EC∗
REC ∈ [0, 1]
where EC∗
R and EC∗
P are respectively the error costs of therandom and perfect models.
Raffaella Calabrese Validation of internal rating systems
Page 29
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Curve of Classification Error Costs (CEC) and ErrorCosts (EC)
Curve:
C[u] =CFN
2{1 − Fd [F−1(u)]} +
CFD
2Fn[F−1(u)] u ∈ (0, 1)
Synthetic index:
EC∗ =
∫ 1
0C[F (s)]dF (s)
EC =EC∗ − EC∗
R
EC∗
P − EC∗
REC ∈ [0, 1]
where EC∗
R and EC∗
P are respectively the error costs of therandom and perfect models.
Raffaella Calabrese Validation of internal rating systems
Page 30
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Curve of Classification Error Costs (CEC) and ErrorCosts (EC)
Optimal cut-off score: the value s that satisfies
mins
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
= maxs
[
Fd(s)
CFD−
Fn(s)
CFN
]
Point measure (Zenga, 2007): U(c) =
−
µ (c)+µ (c)
where c ∈ (−∞,+∞) and
−
µ (c) =1
F (c)
∫ c
−∞
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
dF (s)
+µ (c) =
11 − F (c)
∫
+∞
c
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
dF (s)
Raffaella Calabrese Validation of internal rating systems
Page 31
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Curve of Classification Error Costs (CEC) and ErrorCosts (EC)
Optimal cut-off score: the value s that satisfies
mins
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
= maxs
[
Fd(s)
CFD−
Fn(s)
CFN
]
Point measure (Zenga, 2007): U(c) =
−
µ (c)+µ (c)
where c ∈ (−∞,+∞) and
−
µ (c) =1
F (c)
∫ c
−∞
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
dF (s)
+µ (c) =
11 − F (c)
∫
+∞
c
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
dF (s)
Raffaella Calabrese Validation of internal rating systems
Page 32
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Bibliography
Basel Committee on Banking Supervision (2005). Studies on theValidation of Internal Rating Systems. Working paper 14. Basel,BIS.
Hong C. S. (2009). Optimal Threshold from ROC and CAPCurves. Communications in Statistics, 38, 2060-2072.
Zenga M. (2007). Inequality curve and inequality index based onthe ratio between lower and upper arithmetic means. Statistica &Applicazioni, V, 3-27.
Raffaella Calabrese Validation of internal rating systems