Top Banner
Modeling Reserve Modeling Reserve Variability Variability November 13, 2007 November 13, 2007 Mark R. Shapland, FCAS, ASA, Mark R. Shapland, FCAS, ASA, MAAA MAAA Louise A. Francis, FCAS, MAAA Louise A. Francis, FCAS, MAAA
75

Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Jan 04, 2016

Download

Documents

Austen Hardy
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Modeling Reserve Modeling Reserve VariabilityVariabilityNovember 13, 2007November 13, 2007

Mark R. Shapland, FCAS, ASA, Mark R. Shapland, FCAS, ASA, MAAAMAAA

Louise A. Francis, FCAS, MAAALouise A. Francis, FCAS, MAAA

Page 2: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Definitions of TermsDefinitions of Terms

Ranges vs. DistributionsRanges vs. Distributions

Methods vs. ModelsMethods vs. Models

Types of Methods/ModelsTypes of Methods/Models

Advantages of DistributionsAdvantages of Distributions

Diagnostic TestingDiagnostic Testing

OverviewOverview

Page 3: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

ReserveReserve – an amount carried in the liability – an amount carried in the liability section of a risk-bearing entity’s balance sheet for section of a risk-bearing entity’s balance sheet for claims incurred prior to a given accounting date.claims incurred prior to a given accounting date.

LiabilityLiability – the actual amount that is owed and will – the actual amount that is owed and will ultimately be paid by a risk-bearing entity for ultimately be paid by a risk-bearing entity for claims incurred prior to a given accounting date.claims incurred prior to a given accounting date.

Loss LiabilityLoss Liability – the – the expected valueexpected value of all of all estimated future claim estimated future claim paymentspayments..

RiskRisk ( (from the “risk-bearers” point of viewfrom the “risk-bearers” point of view) – the ) – the uncertainty (deviations from expected) in both uncertainty (deviations from expected) in both timing and amount of the future claim timing and amount of the future claim paymentpayment stream.stream.

Definitions of TermsDefinitions of Terms

Page 4: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Process RiskProcess Risk – the randomness of future – the randomness of future outcomes given a outcomes given a knownknown distribution of distribution of possible outcomes.possible outcomes.

Parameter RiskParameter Risk – the potential error in the – the potential error in the estimated parameters used to describe the estimated parameters used to describe the distribution of possible outcomes, assuming distribution of possible outcomes, assuming the process generating the outcomes is the process generating the outcomes is knownknown..

Model RiskModel Risk – the chance that the model – the chance that the model (“process”) used to estimate the distribution (“process”) used to estimate the distribution of possible outcomes is incorrect or of possible outcomes is incorrect or incomplete.incomplete.

Definitions of TermsDefinitions of Terms

Page 5: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

RiskRisk – unknown outcomes with – unknown outcomes with quantifiablequantifiable probabilities. probabilities.

UncertaintyUncertainty – unknown outcomes that – unknown outcomes that cannot be cannot be estimatedestimated or or quantifiedquantified..

All entrepreneurship involves both All entrepreneurship involves both riskrisk (which can be transferred) and (which can be transferred) and uncertaintyuncertainty (which cannot be (which cannot be transferred).transferred).

Definitions of TermsDefinitions of Terms

Source: Knight, Frank H. 1921. “Risk, Uncertainty, and Profit.” Houghton Mifflin.

Page 6: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Variance, standard deviation, kurtosis, Variance, standard deviation, kurtosis, average absolute deviation, Value at Risk, average absolute deviation, Value at Risk, Tail Value at Risk, Tail Value at Risk, etcetc. which are measures . which are measures of dispersion. of dispersion.

Other measures useful in determining Other measures useful in determining “reasonableness” could include: mean, “reasonableness” could include: mean, mode, median, pain function, mode, median, pain function, etcetc..

The choice for measure of risk will also be The choice for measure of risk will also be important when considering the important when considering the “reasonableness” and “materiality” of the “reasonableness” and “materiality” of the reserves in relation to the capital position. reserves in relation to the capital position.

Definitions of TermsDefinitions of Terms

Measures of Risk from Statistics:Measures of Risk from Statistics:

Page 7: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

A “Range” is not the same as a “Distribution”A “Range” is not the same as a “Distribution”

A Range of Reasonable EstimatesA Range of Reasonable Estimates is a range is a range of estimates that could be produced by of estimates that could be produced by appropriate actuarial methods or alternative appropriate actuarial methods or alternative sets of assumptions that the actuary judges to sets of assumptions that the actuary judges to be reasonable.be reasonable.

A A DistributionDistribution is a statistical function that is a statistical function that attempts to quantify probabilities of all attempts to quantify probabilities of all possible outcomes.possible outcomes.

Ranges vs. DistributionsRanges vs. Distributions

Page 8: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Ranges vs. DistributionsRanges vs. Distributions

A Range, by itself, creates problems:A Range, by itself, creates problems:

A range can be misleading to the A range can be misleading to the layperson – it can give the impression layperson – it can give the impression that any number in that range is equally that any number in that range is equally likely. likely.

A range can give the impression that as A range can give the impression that as long as the carried reserve is “within long as the carried reserve is “within the range” anything is reasonable.the range” anything is reasonable.

Page 9: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Ranges vs. DistributionsRanges vs. Distributions

A Range, by itself, creates problems:A Range, by itself, creates problems:

There is currently no specific guidance There is currently no specific guidance on how to consistently determine a on how to consistently determine a range within the actuarial community range within the actuarial community (e.g., +/- X%, +/- $X, using various (e.g., +/- X%, +/- $X, using various estimates, etc.). estimates, etc.).

A range, in and of itself, needs some A range, in and of itself, needs some other context to help define it (e.g., other context to help define it (e.g., how to you calculate a risk margin?)how to you calculate a risk margin?)

Page 10: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Ranges vs. DistributionsRanges vs. Distributions

A Distribution provides:A Distribution provides:

Information about “all” possible Information about “all” possible outcomes. outcomes.

Context for defining a variety of other Context for defining a variety of other measures (e.g., risk margin, measures (e.g., risk margin, materiality, risk based capital, etc.)materiality, risk based capital, etc.)

Page 11: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Ranges vs. DistributionsRanges vs. Distributions

Should we use the same:Should we use the same:

Criteria for judging the quality of a Criteria for judging the quality of a range vs. a distribution? range vs. a distribution?

Basis for determining materiality? risk Basis for determining materiality? risk margins?margins?

Selection process for which numbers Selection process for which numbers are “reasonable” to chose from?are “reasonable” to chose from?

Page 12: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

A A MethodMethod is an algorithm or recipe is an algorithm or recipe – a series of steps that are followed – a series of steps that are followed to give an estimate of future to give an estimate of future payments.payments.

The well known chain ladder (CL) The well known chain ladder (CL) and Bornhuetter-Ferguson (BF) and Bornhuetter-Ferguson (BF) methods are examples.methods are examples.

The search for the “best” pattern.The search for the “best” pattern.

Methods vs. ModelsMethods vs. Models

Page 13: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

A A ModelModel specifies statistical specifies statistical assumptions about the loss process, assumptions about the loss process, usually leaving some parameters to be usually leaving some parameters to be estimated.estimated.

Then estimating the parameters gives Then estimating the parameters gives an estimate of the ultimate losses and an estimate of the ultimate losses and some statistical properties of that some statistical properties of that estimate.estimate.

The search for the “best” distribution.The search for the “best” distribution.

Methods vs. ModelsMethods vs. Models

Page 14: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Methods vs. ModelsMethods vs. Models

Many good probability models have been Many good probability models have been built using “Collective Risk Theory” built using “Collective Risk Theory”

Each of these models make assumptions Each of these models make assumptions about the processes that are driving claims about the processes that are driving claims and their settlement valuesand their settlement values

None of them can ever completely None of them can ever completely eliminate “model risk”eliminate “model risk”

““All models are wrong. Some models are All models are wrong. Some models are useful.”useful.”

Page 15: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Types of ModelsTypes of Models

Individual Claim ModelsTriangle Based Models vs.

Conditional Models Unconditional Modelsvs.

Single Triangle Models Multiple Triangle Modelsvs.

Parametric Models Non-Parametric Modelsvs.

Diagonal Term No Diagonal Termvs.

Fixed Parameters Variable Parametersvs.

Page 16: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Types of ModelsTypes of Models

Processes used to calculate liability Processes used to calculate liability ranges or distributions can be grouped ranges or distributions can be grouped into four general categories: into four general categories:

1) Multiple Projection Methods,1) Multiple Projection Methods,

2) Statistics from Link Ratio Models,2) Statistics from Link Ratio Models,

3) Incremental Models, and3) Incremental Models, and

4) Simulation Models4) Simulation Models

Page 17: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

$11M $16M

Page 18: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

We could define a “reasonable” We could define a “reasonable” range based on probabilities of the range based on probabilities of the distribution of possible outcomes.distribution of possible outcomes.

This can be translated into a range This can be translated into a range of liabilities that correspond to of liabilities that correspond to those probabilities.those probabilities.

Advantages of Advantages of DistributionsDistributions

Premise:Premise:

Page 19: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

The “risk” in the data defines the range.The “risk” in the data defines the range.

Adds context to other statistical measures.Adds context to other statistical measures.

A “reserve margin” can be defined more A “reserve margin” can be defined more precisely.precisely.

Can be related to risk of insolvency and Can be related to risk of insolvency and materiality issues.materiality issues.

Others can define what is reasonable for Others can define what is reasonable for them.them.

Advantages of Advantages of DistributionsDistributions

A probability range has several advantages:A probability range has several advantages:

Page 20: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

150150100100909011511510010097975050thth to 75 to 75thth PercentilePercentile

12012010010080801201201001008080Expected +/- 20%Expected +/- 20%

HighHighEVEVLowLowHighHighEVEVLowLowMethodMethod

More Volatile More Volatile LOBLOB

Relatively Stable Relatively Stable LOBLOB

Advantages of Advantages of DistributionsDistributions

Comparison of “Reasonable” Reserve RangesComparison of “Reasonable” Reserve Rangesby Methodby Method

Page 21: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

Comparison of “Normal” vs. “Skewed” Liability Distributions

Page 22: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

Comparison of Aggregate Liability Distributions

A

B

C

Added(100% Correlation)

Correlated(0% Correlation)

Page 23: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

Comparison of Aggregate Liability Distributions

Value

Pro

bab

ility

Value

Pro

bab

ility

Before CorrelationBefore Correlation After CorrelationAfter Correlation

Capital = $1,000M Capital = $600M

Page 24: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

Reasonable & Prudent Margin

Reasonable & Conservative Margin

Page 25: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

1%1%16016015%15%12012040%40%808050%50%100100

Prob. Prob. Of Ins.Of Ins.

AmouAmountnt

Prob. Prob. Of Ins.Of Ins.

AmouAmountnt

Prob. Prob. Of Ins.Of Ins.

AmouAmountntProb. Prob.

AmouAmountnt

Situation CSituation CSituation BSituation BSituation ASituation ALoss ReservesLoss Reserves

Corresponding Surplus Depending on SituationCorresponding Surplus Depending on Situation

1%1%15015015%15%11011040%40%707075%75%110110

1%1%14014015%15%10010040%40%606090%90%120120

Advantages of Advantages of DistributionsDistributions

Comparison of “Reasonable” Reserve RangesComparison of “Reasonable” Reserve Rangeswith Probabilities of Insolvencywith Probabilities of Insolvency

“Low” Reserve Risk

Page 26: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

10%10%12012040%40%808060%60%404090%90%140140

10%10%14014040%40%10010060%60%606075%75%120120

10%10%16016040%40%12012060%60%808050%50%100100

Prob. Prob. Of Ins.Of Ins.

AmouAmountnt

Prob. Prob. Of Ins.Of Ins.

AmouAmountnt

Prob. Prob. Of Ins.Of Ins.

AmouAmountntProb. Prob.

AmouAmountnt

Situation CSituation CSituation BSituation BSituation ASituation ALoss ReservesLoss Reserves

Corresponding Surplus Depending on SituationCorresponding Surplus Depending on Situation

Advantages of Advantages of DistributionsDistributions

Comparison of “Reasonable” Reserve RangesComparison of “Reasonable” Reserve Rangeswith Probabilities of Insolvencywith Probabilities of Insolvency

“Medium” Reserve Risk

Page 27: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

20%20%606050%50%202080%80%-20-2090%90%200200

20%20%11011050%50%707080%80%303075%75%150150

20%20%16016050%50%12012080%80%808050%50%100100

Prob. Prob. Of Ins.Of Ins.

AmouAmountnt

Prob. Prob. Of Ins.Of Ins.

AmouAmountnt

Prob. Prob. Of Ins.Of Ins.

AmouAmountntProb. Prob.

AmouAmountnt

Situation CSituation CSituation BSituation BSituation ASituation ALoss ReservesLoss Reserves

Corresponding Surplus Depending on SituationCorresponding Surplus Depending on Situation

Advantages of Advantages of DistributionsDistributions

Comparison of “Reasonable” Reserve RangesComparison of “Reasonable” Reserve Rangeswith Probabilities of Insolvencywith Probabilities of Insolvency

“High” Reserve Risk

Page 28: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Principle of Greatest Common InterestPrinciple of Greatest Common Interest – the “largest amount” considered – the “largest amount” considered “reasonable” when a variety of constituents “reasonable” when a variety of constituents share a common goal or interest, such that share a common goal or interest, such that all common goals or interests are met; and all common goals or interests are met; and thethe

Principle of Least Common InterestPrinciple of Least Common Interest – – the “smallest amount” considered the “smallest amount” considered “reasonable” when a variety of constituents “reasonable” when a variety of constituents share a common goal or interest, such that share a common goal or interest, such that all common goals or interests are met.all common goals or interests are met.

Advantages of Advantages of DistributionsDistributions

Satisfying Different Constituents:Satisfying Different Constituents:

Page 29: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

$11M $16M

Page 30: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

$11M $16M

Page 31: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

$11M $16M

Page 32: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

$11M $16M

Page 33: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

$11M $16M

Page 34: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

Reasonable Range

Reasonable Distribution

Page 35: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Advantages of Advantages of DistributionsDistributions

A Distribution can be used for:A Distribution can be used for:Risk-Based / Economic CapitalRisk-Based / Economic Capital

Reserve RiskReserve RiskPricing RiskPricing RiskALM RiskALM Risk

Pricing / ROEPricing / ROEReinsurance AnalysisReinsurance Analysis

Quota ShareQuota ShareAggregate ExcessAggregate ExcessStop LossStop LossLoss Portfolio TransferLoss Portfolio Transfer

Dynamic Risk Modeling (DFA)Dynamic Risk Modeling (DFA)Strategic Planning / ERMStrategic Planning / ERM

Allocated Capital

Risk Transfer

Page 36: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

The Need for TestingThe Need for Testing

Actuaries Have Built Many Actuaries Have Built Many Sophisticated Models Based on Sophisticated Models Based on Collective Risk Theory Collective Risk Theory

All Models Make Simplifying All Models Make Simplifying AssumptionsAssumptions

How do we Evaluate Them?How do we Evaluate Them?

Page 37: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

How Do We “Evaluate”?How Do We “Evaluate”?

$11M $16M

(Point Estimates)(Point Estimates)

Page 38: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Liability Estimates

Pro

bab

ility

How Do We “Evaluate”?How Do We “Evaluate”?(Multiple Distributions)(Multiple Distributions)

Page 39: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Liability Estimates

Pro

bab

ility

How Do We “Evaluate”?How Do We “Evaluate”?(Competing Distributions)(Competing Distributions)

Page 40: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Ranges vs. DistributionsRanges vs. Distributions

Page 41: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Ranges vs. DistributionsRanges vs. Distributions

Reasonable Range

Reasonable Distribution

Page 42: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

The Need for TestingThe Need for Testing

Actuaries Have Built Many Actuaries Have Built Many Sophisticated Models Based on Sophisticated Models Based on Collective Risk Theory Collective Risk Theory

All Models Make Simplifying All Models Make Simplifying AssumptionsAssumptions

How do we Evaluate Them?How do we Evaluate Them?

Page 43: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Fundamental QuestionsFundamental Questions

How Well Does the Model How Well Does the Model Measure and Reflect the Measure and Reflect the Uncertainty Inherent in the Uncertainty Inherent in the Data?Data?

Does the Model do a Good Job of Does the Model do a Good Job of Capturing and Replicating the Capturing and Replicating the Statistical Features Found in the Statistical Features Found in the Data? Data?

Page 44: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Modeling GoalsModeling Goals

Is the Goal to Minimize the Range Is the Goal to Minimize the Range (or Uncertainty) that Results from (or Uncertainty) that Results from the Model?the Model?

Goal of Modeling is Goal of Modeling is NOTNOT to to Minimize Process Uncertainty!Minimize Process Uncertainty!

Goal is to Find the Best Statistical Goal is to Find the Best Statistical Model, While Minimizing Model, While Minimizing Parameter and Model Uncertainty. Parameter and Model Uncertainty.

Page 45: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Liability Estimates

Pro

bab

ility

How Do We “Evaluate”?How Do We “Evaluate”?

Page 46: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Diagnostic TestsDiagnostic Tests

Model Selection CriteriaModel Selection Criteria

Model Reasonability ChecksModel Reasonability Checks

Goodness-of-Fit & Prediction Goodness-of-Fit & Prediction ErrorsErrors

Page 47: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Model Selection CriteriaModel Selection Criteria

Criterion 1Criterion 1: : Aims of the AnalysisAims of the Analysis

Will the Procedure Achieve the Will the Procedure Achieve the Aims of the Analysis?Aims of the Analysis?

Criterion 2Criterion 2: : Data AvailabilityData Availability

Access to the Required Data Access to the Required Data Elements?Elements?

Unit Record-Level Data or Unit Record-Level Data or Summarized “Triangle” Data?Summarized “Triangle” Data?

Page 48: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Model Selection CriteriaModel Selection Criteria

Criterion 3Criterion 3: : Non-Data Specific Non-Data Specific Modeling Technique EvaluationModeling Technique Evaluation

Has Procedure been Validated Has Procedure been Validated Against Historical Data? Against Historical Data?

Verified to Perform Well Against Verified to Perform Well Against Dataset with Similar Features?Dataset with Similar Features?

Assumptions of the Model Plausible Assumptions of the Model Plausible Given What is Known About the Given What is Known About the Process Generating this Data?Process Generating this Data?

Page 49: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Model Selection CriteriaModel Selection Criteria

Criterion 4Criterion 4:: Cost/Benefit Cost/Benefit ConsiderationsConsiderations

Can Analysis be Performed Using Can Analysis be Performed Using Widely Available Software?Widely Available Software?

Analyst Time vs. Computer Time?Analyst Time vs. Computer Time?

How Difficult to Describe to How Difficult to Describe to Junior Staff, Senior Management, Junior Staff, Senior Management, Regulators, Auditors, etc.?Regulators, Auditors, etc.?

Page 50: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Model Reasonability Model Reasonability ChecksChecks

Criterion 5Criterion 5: : Coefficient of Coefficient of Variation by YearVariation by Year

Should be Largest for Oldest Should be Largest for Oldest (Earliest) Year(Earliest) Year

Criterion 6Criterion 6: : Standard Error by Standard Error by YearYear

Should be Smallest for Oldest Should be Smallest for Oldest (Earliest) Year (on a Dollar Scale)(Earliest) Year (on a Dollar Scale)

Page 51: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Model Reasonability Model Reasonability ChecksChecks

Criterion 7Criterion 7:: Overall Coefficient of Overall Coefficient of VariationVariation

Should be Smaller for All Years Should be Smaller for All Years Combined than any Individual YearCombined than any Individual Year

Criterion 8Criterion 8:: Overall Standard Overall Standard ErrorError

Should be Larger for All Years Should be Larger for All Years Combined than any Individual YearCombined than any Individual Year

Page 52: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Model Reasonability Model Reasonability ChecksChecks

Accident Yr MeanStandard

ErrorCoeffient of

Variation1996 26,416 37,927 143.6%1997 26,216 38,774 147.9%1998 50,890 54,508 107.1%1999 90,705 74,824 82.5%2000 148,110 99,986 67.5%2001 186,832 117,230 62.7%2002 418,461 183,841 43.9%2003 638,082 268,578 42.1%2004 607,107 477,760 78.7%2005 1,521,202 1,017,129 66.9%Total 3,714,020 1,299,184 35.0%

Page 53: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Model Reasonability Model Reasonability ChecksChecks

Accident Yr MeanStandard

ErrorCoeffient of

Variation1996 25,913 37,956 146.5%1997 25,708 38,846 151.1%1998 50,043 54,780 109.5%1999 89,071 74,987 84.2%2000 145,388 100,373 69.0%2001 183,864 118,502 64.5%2002 411,367 185,211 45.0%2003 628,347 271,722 43.2%2004 1,113,073 229,923 20.7%2005 1,263,550 253,596 20.1%Total 3,936,326 599,048 15.2%

Page 54: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Model Reasonability Model Reasonability ChecksChecks

Criterion 9Criterion 9:: Correlated Standard Correlated Standard Error & Coefficient of VariationError & Coefficient of Variation

Should Both be Smaller for All Should Both be Smaller for All LOBs Combined than the Sum of LOBs Combined than the Sum of Individual LOBsIndividual LOBs

Criterion 10Criterion 10:: Reasonability of Reasonability of Model Parameters and Model Parameters and Development PatternsDevelopment Patterns

Is Loss Development Pattern Is Loss Development Pattern Implied by Model Reasonable?Implied by Model Reasonable?

Page 55: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Model Reasonability Model Reasonability ChecksChecks

Criterion 11Criterion 11:: Consistency of Consistency of Simulated Data with Actual DataSimulated Data with Actual Data

Can you Distinguish Simulated Can you Distinguish Simulated Data from Real Data?Data from Real Data?

Criterion 12Criterion 12:: Model Model Completeness and ConsistencyCompleteness and Consistency

Is it Possible Other Data Elements Is it Possible Other Data Elements or Knowledge Could be Integrated or Knowledge Could be Integrated for a More Accurate Prediction?for a More Accurate Prediction?

Page 56: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Criterion 13Criterion 13: : Validity of Link Validity of Link RatiosRatios

Link Ratios are a Form of Link Ratios are a Form of Regression and Can be Tested Regression and Can be Tested StatisticallyStatistically

Exercise: Test the Validity of Exercise: Test the Validity of the 12 to 24 month age-to-age the 12 to 24 month age-to-age factors for the Mack data.factors for the Mack data.

Page 57: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Graph the ratios of the 12 to 24 Graph the ratios of the 12 to 24 month factors (12 month values month factors (12 month values on X axis and 24 month values on X axis and 24 month values on Y axis).on Y axis).

Page 58: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Page 59: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Calculate the linear regression of Y Calculate the linear regression of Y given X values, without an given X values, without an intercept.intercept.

Add a regression line to the Graph Add a regression line to the Graph of age-to-age ratios.of age-to-age ratios.

What is the slope of the regression?What is the slope of the regression?

Was it a good fit?Was it a good fit?

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Page 60: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Page 61: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Calculate the linear regression of Calculate the linear regression of Y given X values, with an Y given X values, with an intercept.intercept.

Add another regression line to the Add another regression line to the Graph of age-to-age ratios.Graph of age-to-age ratios.

What is the intercept & slope of What is the intercept & slope of the regression?the regression?

Was it a good fit?Was it a good fit?

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Page 62: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Page 63: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Criterion 13Criterion 13: : Validity of Link RatiosValidity of Link Ratios

Link Ratios are a Form of Regression Link Ratios are a Form of Regression and Can be Tested Statisticallyand Can be Tested Statistically

Criterion 14Criterion 14: : Standardization of Standardization of ResidualsResiduals

Standardized Residuals Should be Standardized Residuals Should be Checked for Normality, Outliers, Checked for Normality, Outliers, Heteroscedasticity, etc.Heteroscedasticity, etc.

Page 64: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Standardized Standardized ResidualsResiduals

The residual equals the actual value The residual equals the actual value for a variable minus its expected for a variable minus its expected valuevalue

In general, a standardized residual In general, a standardized residual equals the residual divided by the equals the residual divided by the standard deviation of the variablestandard deviation of the variable

Sometimes residuals for a weighted Sometimes residuals for a weighted model (for instance WAD model) is model (for instance WAD model) is divided by the square root of the divided by the square root of the weightweight

Page 65: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Standardized ResidualsStandardized Residuals

Plot of Residuals against Predicted

-2.0000

-1.5000

-1.0000

-0.5000

0.0000

0.5000

1.0000

1.5000

5.0000 6.0000 7.0000 8.0000

Predicted

Res

idua

ls

Page 66: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Standardized ResidualsStandardized Residuals

Plot of Residuals against Predicted

-0.8000

-0.6000

-0.4000

-0.2000

0.0000

0.2000

0.4000

0.6000

0.8000

4.0000 5.0000 6.0000 7.0000 8.0000

Predicted

Res

idua

ls

Page 67: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Criterion 15Criterion 15: : Analysis of Analysis of Residual PatternsResidual Patterns

Check Against Accident, Check Against Accident, Development and Calendar Development and Calendar PeriodsPeriods

Page 68: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Standardized ResidualsStandardized Residuals

Plot of Residuals against Predicted

-0.8000

-0.6000

-0.4000

-0.2000

0.0000

0.2000

0.4000

0.6000

0.8000

4.0000 5.0000 6.0000 7.0000 8.0000

Predicted

Res

idu

als

Plot of Residuals against Development Period

-0.8000

-0.6000

-0.4000

-0.2000

0.0000

0.2000

0.4000

0.6000

0.8000

Development Period

Res

idua

ls

Plot of Residuals against Accident Period

-0.8000

-0.6000

-0.4000

-0.2000

0.0000

0.2000

0.4000

0.6000

0.8000

0 2 4 6 8 10 12

Accident Period

Res

idua

l

Plot of Residuals against Payment Period

-2.5000

-2.0000

-1.5000

-1.0000

-0.5000

0.0000

0.5000

1.0000

0 2 4 6 8 10 12

Payment Period

Res

idu

al

Page 69: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Standardized ResidualsStandardized ResidualsPlot of Residuals against Development Period

-600.00

-400.00

-200.00

0.00

200.00

400.00

600.00

800.00

Development Period

Res

idua

ls

Plot of Residuals against Accident Period

-600.00

-400.00

-200.00

0.00

200.00

400.00

600.00

800.00

Accident Period

Res

idua

l

Plot of Residuals against Payment Period

-600.00

-400.00

-200.00

0.00

200.00

400.00

600.00

800.00

Payment Period

Res

idua

l

Plot of Residuals against Predicted

-600.00

-400.00

-200.00

0.00

200.00

400.00

600.00

800.00

0 200000 400000 600000 800000 1000000 1200000 1400000

Predicted

Res

idua

ls

Page 70: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Criterion 15Criterion 15: : Analysis of Residual Analysis of Residual PatternsPatterns

Check Against Accident, Check Against Accident, Development and Calendar PeriodsDevelopment and Calendar Periods

Criterion 16Criterion 16:: Prediction Error and Prediction Error and Out-of-Sample DataOut-of-Sample Data

Test the Accuracy of Predictions on Test the Accuracy of Predictions on Data that was Not Used to Fit the Data that was Not Used to Fit the ModelModel

Page 71: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Criterion 17Criterion 17: : Goodness-of-Fit Goodness-of-Fit MeasuresMeasures

Quantitative Measures that Enable Quantitative Measures that Enable One to Find Optimal Tradeoff Between One to Find Optimal Tradeoff Between Minimizing Model Bias and Predictive Minimizing Model Bias and Predictive VarianceVarianceAdjusted Sum of Squared Errors Adjusted Sum of Squared Errors (SSE)(SSE)

Akaike Information Criterion (AIC)Akaike Information Criterion (AIC)Bayesian Information Criterion (BIC)Bayesian Information Criterion (BIC)

Page 72: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Goodness-of-Fit & Goodness-of-Fit & Prediction ErrorsPrediction Errors

Criterion 18Criterion 18: : Ockham’s Razor Ockham’s Razor and the Principle of Parsimonyand the Principle of Parsimony

All Else Being Equal, the Simpler All Else Being Equal, the Simpler Model is PreferableModel is Preferable

Criterion 19Criterion 19:: Model Validation Model Validation

Systematically Remove Last Systematically Remove Last Several Diagonals and Make Several Diagonals and Make Same Forecast of Ultimate Values Same Forecast of Ultimate Values Without the Excluded DataWithout the Excluded Data

Page 73: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Questions? Questions?

Page 74: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Suggested BibliographySuggested Bibliography

[3][3] Barnett, Glen and Ben ZehnwirthBarnett, Glen and Ben Zehnwirth. . 2000. 2000. Best Best Estimates for ReservesEstimates for Reserves. PCAS LXXXVII: 245-321. . PCAS LXXXVII: 245-321.

[15][15] The Analysis and Estimation of Loss & ALAE The Analysis and Estimation of Loss & ALAE Variability: A Summary ReportVariability: A Summary Report. 2005. . 2005. CAS Working CAS Working Party on Quantifying Variability in Reserve EstimatesParty on Quantifying Variability in Reserve Estimates. . CAS CAS ForumForum (Fall): 29-146. (Fall): 29-146.

[25][25] Shapland, Mark RShapland, Mark R. 2007.. 2007. Loss Reserve Estimates: A Loss Reserve Estimates: A Statistical Approach for Determining ‘Reasonableness’Statistical Approach for Determining ‘Reasonableness’. . VarianceVariance 1, no. 1 (Spring): 120-148. 1, no. 1 (Spring): 120-148.

[27][27] Venter, Gary GVenter, Gary G. 1998. . 1998. Testing the Assumptions of Age-Testing the Assumptions of Age-to-Age Factorsto-Age Factors. PCAS LXXXV: 807-47.. PCAS LXXXV: 807-47.

[30][30] Zehnwirth, BenZehnwirth, Ben. 1994. . 1994. Probabilistic Development Probabilistic Development Factor Models with Applications to Loss Reserve Factor Models with Applications to Loss Reserve Variability, Prediction Intervals and Risk Based CapitalVariability, Prediction Intervals and Risk Based Capital. . CAS CAS ForumForum 2 (Spring): 447-605. 2 (Spring): 447-605.

Page 75: Modeling Reserve Variability November 13, 2007 Mark R. Shapland, FCAS, ASA, MAAA Louise A. Francis, FCAS, MAAA.

Contact InformationContact Information

Mark R. Shapland, FCAS, ASA, MAAAMark R. Shapland, FCAS, ASA, MAAAE-Mail:E-Mail:[email protected]@milliman.comOffice:Office: (636) 273-6428(636) 273-6428

Louis A. Francis, FCAS, MAAALouis A. Francis, FCAS, MAAAE-Mail:E-Mail: [email protected][email protected]:Office: (215) 923-1567(215) 923-1567