This material is not for general distribution, and its contents should not be quoted, extracted for publication, or otherwise copied or distributed without prior coordination with the Department of the Army, ATTN: ETF. UNCLASSIFIED / FOUO UNCLASSIFIED / FOUO UNCLASSIFIED / FOUO UNCLASSIFIED / FOUO National Guard Black Belt Training Module 25 Measurement System Analysis (MSA) Attribute Data
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
This material is not for general distribution, and its contents should not be quoted, extracted for publication, or otherwisecopied or distributed without prior coordination with the Department of the Army, ATTN: ETF.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
National GuardBlack Belt Training
Module 25
Measurement System Analysis (MSA)
Attribute Data
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
CPI Roadmap – Measure
Note: Activities and tools vary by project. Lists provided here are not necessarily all-inclusive.
TOOLS
•Process Mapping
•Process Cycle Efficiency/TOC
•Little’s Law
•Operational Definitions
•Data Collection Plan
•Statistical Sampling
•Measurement System Analysis
•TPM
•Generic Pull
•Setup Reduction
•Control Charts
•Histograms
•Constraint Identification
•Process Capability
ACTIVITIES• Map Current Process / Go & See
• Identify Key Input, Process, Output Metrics
• Develop Operational Definitions
• Develop Data Collection Plan
• Validate Measurement System
• Collect Baseline Data
• Identify Performance Gaps
• Estimate Financial/Operational Benefits
• Determine Process Stability/Capability
• Complete Measure Tollgate
1.Validate the
Problem
4. Determine Root
Cause
3. Set Improvement
Targets
5. Develop Counter-
Measures
6. See Counter-MeasuresThrough
2. IdentifyPerformance
Gaps
7. Confirm Results
& Process
8. StandardizeSuccessfulProcesses
Define Measure Analyze ControlImprove
8-STEP PROCESS
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3Measurement System Analysis - Attribute
Learning Objective
Understand how to conduct and interpret a measurement system analysis with Attribute Data
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4Measurement System Analysis - Attribute
Attribute Measurement Systems
Most physical measurement systems use measurement devices that provide continuous data
For continuous data Measurement System Analysis we can use control charts or Gage R&R methods
Attribute/ordinal measurement systems utilize accept/reject criteria or ratings (such as 1 - 5) to determine if an acceptable level of quality has been attained
Kappa and Kendall techniques can be used to evaluate these Attribute and Ordinal Measurement Systems
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5Measurement System Analysis - Attribute
Are You Really Stuck With Attribute Data?
Many inspection or checking processes have the ability to collect continuous data, but decide to use attribute data to simplify the task for the person taking and recording the data
Examples:
On-time Delivery can be recorded in 2 ways:
a) in hours late or
b) whether the delivery was on-time or late
Many functional tests will evaluate a product on a continuous scale (temperature, pressure drop, voltage drop, dimensional, hardness, etc) and record the results as pass/fail
Strive to get continuous data!
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6Measurement System Analysis - Attribute
Attribute and Ordinal Measurements
Attribute and Ordinal measurements often rely on subjective classifications or ratings
Examples include:
Rating different features of a service as either good or bad, or on a scale from 1 to 5 with 5 being best
Rating different aspects of employee performance as excellent, satisfactory, needs improvement
Rating wine on a) aroma, b) taste, and c) after taste
Should we evaluate these measurement systems before using them to make decisions on our CPI project?
What are the consequences of not evaluating them?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7Measurement System Analysis - Attribute
MSA – Attribute Data
What methodologies are appropriate to assess Attribute Measurement Systems?
Attribute Systems – Kappa technique which treat all misclassifications equally
Ordinal Systems – Kendall‟s technique which considers the rank of the misclassification
For example, if we are judging an advertising service on a scale from 1 to 5 and Inspector A rates the service a „1‟ while Inspector B rates it a „5.‟ That is a greater misclassification than Inspector A rating it a „4‟ while Inspector B rates it a „5.‟
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
8Measurement System Analysis - Attribute
Data Scales Nominal: Contains numbers that have no basis on which to arrange
in any order or to make any assumptions about the quantitative difference between them. These numbers are just names or labels. For example:
In an organization: Dept. 1 (Accounting), Dept. 2 (Customer Service), Dept. 3 ( Human Resources)
In an insurance co.: Business Line 1, Line 2, Line 3
Ordinal: Contains numbers that can be ranked in some natural sequence. This scale, however, cannot make an inference about the degree of difference between the numbers. Examples:
On service performance: excellent, very good, good, fair, poor
Salsa taste test: mild, hot, very hot, makes me suffer
Kappa is appropriate for non-quantitative systems such as:
Good or bad
Go/No Go
Differentiating noises (hiss, clank, thump)
Pass/fail
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
10Measurement System Analysis - Attribute
Kappa Techniques
Kappa for Attribute Data:
Treats all misclassifications equally
Does not assume that the ratings are equally distributed across the possible range
Requires that the units be independent and that the persons doing the judging or rating make their classifications independently
Requires that the assessment categories be mutually exclusive
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
11Measurement System Analysis - Attribute
Operational Definitions
There are some quality characteristics that are either difficult or very time consuming to define
To assess classification consistency, several units must be classified by more than one rater or judge
If there is substantial agreement among the raters, there is the possibility, although no guarantee, that the ratings are accurate
If there is poor agreement among the raters, the usefulness of the rating is very limited
Poor attribute measurement systems can almost always be traced to poor operational definitions
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
12Measurement System Analysis - Attribute
Consequences?
What are the important concerns?
What are the risks if agreement within and between raters is not good?
Are bad items escaping to the next operation in the process or to the external customer?
Are good items being reprocessed unnecessarily?
What is the standard for assessment?
How is agreement measured?
What is the Operational Definition for assessment?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
13Measurement System Analysis - Attribute
What Is Kappa? “K”
P observed
Proportion of units on which both Judges agree = proportion both Judges agree are good + proportion both Judges agree are bad
P chance (expected)
Proportion of agreements expected by chance = (proportion Judge A says good * proportion Judge B says good) + (proportion Judge A says bad * proportion B says bad)
Note: equation applies to a two category analysis, e.g., good or bad
chance
chanceobserved
P
PPK
1
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
14Measurement System Analysis - Attribute
Kappa
For perfect agreement, P observed = 1 and K=1
As a rule of thumb, if Kappa is lower than 0.7, the measurement system is not adequate
If Kappa is 0.9 or above, the measurement system is considered excellent
The lower limit for Kappa can range from 0 to -1
For P observed = P chance (expected), then K=0
Therefore, a Kappa of 0 indicates that the agreement is the same as would be expected by random chance
chance
chanceobserved
P
PPK
1
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
15Measurement System Analysis - Attribute
Attribute MSA Guidelines
When selecting items for the study consider the following:
If you only have two categories, good and bad, you should have a minimum of 20 good and 20 bad
As a maximum, have 50 good and 50 bad
Try to keep approximately 50% good and 50% bad
Have a variety of degrees of good and bad
If only good items are chosen for the study, what might happen to P-chance (expected)?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
16Measurement System Analysis - Attribute
Attribute MSA Guidelines (Cont.)
If you have more than two categories, with one of the categories being good and the other categories being different error modes, you should have approximately 50% of the items being good and a minimum of 10% of the items in each of the error modes
You might combine some of the error modes as “other”
The categories should be mutually exclusive or, if not, they should also be combined
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
17Measurement System Analysis - Attribute
Within Rater/Repeatability Considerations
Have each rater evaluate the same item at least twice
Calculate a Kappa for each rater by creating separate Kappa tables, one for each rater
If a Kappa measurement for a particular rater is small, that rater does not repeat well within self
If the rater does not repeat well within self, then they will not repeat well with the other raters and this will hide how good or bad the others repeat between themselves
Calculate a between-rater Kappa by creating a Kappa table from the first judgment of each rater
Between-rater Kappa will be made as pairwise comparisons (A to B, B to C, A to C)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
18Measurement System Analysis - Attribute
Example: Data Set = Attribute Ordinal.mtw
An educational testing organization is training five new appraisers for the written portion of the twelfth-grade standardized essay test
The appraisers‟ ability to rate essays consistent with the standards needs to be assessed
Each appraiser rated fifteen essays on a five-point scale (-2, -1, 0, 1, 2)
The organization also rated the essays and supplied the “official score”
Each essay was rated twice and the data captured in the file Attribute Ordinal.mtw
Open the file and evaluate the appraisers' performance
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
19Measurement System Analysis - Attribute
Minitab and Attribute Measurement Systems
Stat>Quality Tools>Attribute Agreement Analysis
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
20Measurement System Analysis - Attribute
Minitab Dialog Box
1. Double click on theappropriate variable to place it in the required dialog box:
Attribute = Rating
Samples = SampleAppraisers = Appraiser
2. Click on OK
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
21Measurement System Analysis - Attribute
Within Appraiser Percent
This output represents the percent agreement and the 95% confidence interval around that percentage
Appraiser
Pe
rce
nt
SimpsonMontgomeryHolmesHayesDuncan
100
80
60
40
20
0
95.0% C I
Percent
Date of study:
Reported by:
Name of product:
Misc:
Assessment Agreement
Within Appraisers
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
22Measurement System Analysis - Attribute
Within Appraiser Session Window Output
This output is the same information contained in the graph with the addition of a Between-Appraiser assessment
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
23Measurement System Analysis - Attribute
Let’s Do It Again
Stat>Quality Tools>Attribute Agreement Analysis
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
24Measurement System Analysis - Attribute
Introducing a Known Standard
If you have a known standard (the real answer)for the items being inspected,let Minitab know what column that information is in.
1. Double click on theappropriate variable to place it in the required dialog box
3. Click on OK
2.
(same as before)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
25Measurement System Analysis - Attribute
Appraiser vs. Standard
Appraiser
Pe
rce
nt
Simps
on
Mon
tgom
ery
Holmes
Hayes
Dunc
an
100
90
80
70
60
50
40
30
95.0% C I
Percent
Appraiser
Pe
rce
nt
Simps
on
Mon
tgom
ery
Holmes
Hayes
Dunc
an
100
90
80
70
60
50
40
30
95.0% C I
Percent
Date of study:
Reported by:
Name of product:
Misc:
Assessment Agreement
Within Appraisers Appraiser vs Standard
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
26Measurement System Analysis - Attribute
Within Appraiser
In addition to the Within-Appraiser graphic, Minitab will give percentages
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
27Measurement System Analysis - Attribute
Each Appraiser vs. Standard
Some appraisers will repeat their own ratings well but may not match the standard well (look at Duncan)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
28Measurement System Analysis - Attribute
More Session Window Output
The session window will give percentage data as to how all the appraisers did when judged against the standard
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
29Measurement System Analysis - Attribute
Kappa and Minitab
Minitab will calculate a Kappa for each (within) appraiser for each category
Note: This is only a part of the total data set for illustration
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
30Measurement System Analysis - Attribute
Kappa vs. Standard
Minitab will also calculate a Kappa statistic for eachappraiser as compared to the standard
Note: This is only a part of the total data set for illustration
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
31Measurement System Analysis - Attribute
Kappa and Minitab
Minitab will not provide a Kappa between a specific pair of appraisers, but will provide an overall Kappa between all appraisers for each possible category of response
How might this output help us improve our measurement system?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
32Measurement System Analysis - Attribute
What If My Data Is Ordinal?
Stat>Quality Tools>Attribute Agreement Analysis
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
33Measurement System Analysis - Attribute
Ordinal Data
If your data is Ordinal, you must also check this box
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
34Measurement System Analysis - Attribute
What Is Kendall’s
Kendall‟s coefficient can be thought of as an R-squared value, it is the correlation between the responses treating the data as attribute as compared to ordinal.
The lower the number gets, the more severe the misclassifications were.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
35Measurement System Analysis - Attribute
Kendall’s
Kendall‟s coefficient can be thought of as an R-squared value, it is the correlation between the responses treating the data as attribute as
compared to ordinal. The lower the number gets, the more severe the misclassifications were.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
36Measurement System Analysis - Attribute
Kendall’s (Cont.)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
37Measurement System Analysis - Attribute
Exercise: Seeing Stars
Divide into teams of two
One person will be the rater and one the recorder
Have each rater inspect each start and determine if it is Good or Bad (Kappa)
Record the results in Minitab
Mix up the stars and repeat with same rater 2 more times
Compare results to other raters and to the known standard
Take 30 minutes to complete the exercise and be prepared to review your findings with the class
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
38Measurement System Analysis - Attribute
Takeaways
How to set-up/conduct an MSA
Use attribute data only if the measurement can not be converted to continuous data
Operational definitions are extremely important
Attribute measurement systems require a great deal of maintenance
Kappa is an easy method to test how repeatable and reproducible a subjective measurement system is
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
What other comments or questions
do you have?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
40Measurement System Analysis - Attribute
References
Cohen, J., “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement, Vol. 20, pp. 37-46, 1960
Futrell, D., “When Quality Is a Matter of Taste, Use Reliability Indexes,” Quality Progress, May 1995
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
APPENDIX – A Practical Example of Kappa
Evaluating the Measurement System for Determining Civilian Awards
41Measurement System Analysis - Attribute
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
42Measurement System Analysis - Attribute
Kappa Example #1
The Chief of Staff (COS) of the 1st Infantry Division is preparing for the redeployment of 3 brigade combat teams supporting Operation Iraqi Freedom.
The Secretary of General Staff (SGS) informs the COS that awards for civilian personnel (Department of the Army Civilians and military dependents) who provided volunteer support prior to and during the deployment is always a “significant emotional issue.” There are hundreds of submissions for awards.
A board of senior Army personnel decides who receives an award. The measurement system the board uses to determine who receives an award is a major concern due to differences in board member to board member differences as well as within board member differences.
The COS directs the SGS (a certified Army Black Belt) to conduct a measurement system study using historical data to “level set” the board members. Kappa for each board member as well as Kappa between board members must be calculated.
The COS‟ guidance is to retrain and/or replace board members until the measurement system is not a concern.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
43Measurement System Analysis - Attribute
Consider the Following Data
• The Lean Six Sigma Pocket Toolbook, p.100-103 outlines the procedures for calculating Kappa. Kappa is MSA for attribute data.
• The SGS‟ study involves two categories for recommendations, “Award” and “No Award”.
• We select 40 candidate packets from historical data and ensure that 20 are definitely for “Award” and 20 are for “No Award”.
• Board Member 1 and 2 evaluate each candidate‟s packet. The results are shown in the tables on the following slides.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
44Measurement System Analysis - Attribute
Consider the Following Data
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
45Measurement System Analysis - Attribute
Consider the Following Data
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
46Measurement System Analysis - Attribute
Contingency Table for Board Member 1
Populate Each Cell with the Evaluation Data
Board Member 1 – 1st : shows the results of Board Member 1’s 1st recommendations. The 1st board member recommended an “Award” or “No Award” for each of the 40 candidates on the first review of the files.
Board Member 1 – 2nd : shows the results of Board Member 1’s 2nd recommendations. The 1st
board member recommended an “Award” or “No Award” for each of the 40 candidates on the second review of the files.
Contingency Table: Counts Board Member 1 - 1st
No Award
3
318 22
19
15 18
22
Award No Award
Award
Bo
ard
Me
mb
er
1-
2n
d
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
47Measurement System Analysis - Attribute
Contingency Table: Cell 1
The first cell represents the number oftimes Board Member 1 recommended a candidate should receive an “Award” in both the first and second evaluation.
Bo
ard
Me
mb
er
1-
2n
d
Contingency Table:Counts
Board Member 1 - 1st
No Award
3
318 22
19
15 18
22
Award No Award
Award
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
48Measurement System Analysis - Attribute
Contingency Table: Cell 2
The second cell represents the number of times Board Member 1 recommended a candidate as “No Award” the first time and “Award” the second evaluation.
Contingency Table:Counts
Board Member 1 - 1st
No Award
3
318 22
19
15 18
22
Award No Award
Award
Bo
ard
Me
mb
er
1-
2n
d
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
49Measurement System Analysis - Attribute
Contingency Table: Cell 3
The third cell represents the number of times Board Member 1 recommended “Award” on the first evaluation and “No Award” on the second evaluation.
Contingency Table:Counts
Board Member 1 - 1st
No Award
3
318 22
19
15 18
22
Award No Award
Award
Bo
ard
Me
mb
er
1-
2n
d
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
50Measurement System Analysis - Attribute
Contingency Table: Cell 4
The fourth cell represents the number of times Board Member 1 recommended “No Award” on the first evaluation and “No Award” on the second evaluation.
Contingency Table:Counts
Board Member 1 - 1st
No Award
3
318 22
19
15 18
22
Award No Award
Award
Bo
ard
Me
mb
er
1-
2n
d
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
51Measurement System Analysis - Attribute
Contingency Table: Sum of Row and Columns
The numbers on the margins are the totals of the rows and columns of data. The sum in both instances is 40, the total number of candidate packets reviewed.
Contingency Table:Counts
Board Member 1 - 1st
No Award
3
318 22
19
15 18
22
Award No Award
Award
Bo
ard
Me
mb
er
1-
2n
d
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
52Measurement System Analysis - Attribute
Contingency Table – Counts & Proportions
Board Member 1 Proportions: The lower table is the data in the upper table
represented as a percentage of the total.
Represents 18/40
0.550.45 0.55
0.45
Contingency Table:Proportions No Award
Bo
ard
Me
mb
er
1-
2n
d Award 0.375 0.075
No Award 0.075 0.475
Board Member 1 - 1stAward
Contingency Table:Counts
Board Member 1 - 1st
No Award
3
318 22
19
15 18
22
Award No Award
Award
Bo
ard
Me
mb
er
1-
2n
d
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
53Measurement System Analysis - Attribute
Contingency Table – Sum of Percentages
The sum percentages from the rows and columns. The sums must equal 1.0
0.550.45 0.55
0.45
Contingency Table:Proportions No Award
Bo
ard
Me
mb
er
1-
2n
d Award 0.375 0.075
No Award 0.075 0.475
Board Member 1 - 1stAward
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
54Measurement System Analysis - Attribute
chance
chanceobserved
P
PPK
1
The verbiage for defining Kappa will vary slightly depending on whether we are defining a Within-Rater Kappa or Between-Rater Kappa
Calculating Kappa
Pobserved
Proportion of candidates for which both Board Members agree = proportion both Board Members agree are “Award” + proportion both Board Members agree are “No Award”.
Pchance
Proportion of agreements expected by chance = (proportion Board Member 1 says “Award” * proportion Board Member 2 says “Award”)+ (proportion Board Member 1 says “No Award” * proportion Member 2 says ”No Award”)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
55Measurement System Analysis - Attribute
Pobserved is the sum of the probabilities on the diagonal:P observed =(0.375 + 0.475) = 0.850
Pchance is the probabilities for each classification multiplied and then summed:Pchance =(0.450*0.450) + (0.550*0.550) = 0.505
Then KBoard Member 1=(0.850 - 0.505)/(1 - 0.505)=0.697
Kappa for Board Member 1 is sufficiently close to 0.700 that we conclude that Board Member 1
exhibits repeatability.
Calculate Kappa for Board Member 1
0.550.45 0.55
0.45
Contingency Table:Proportions
No Award
Bo
ard
Me
mb
er
1-
2n
d Award 0.375 0.075
No Award 0.075 0.475
Board Member 1 - 1stAward
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
56Measurement System Analysis - Attribute
K Board Member 2 = ?
Calculate Kappa for Board Member 2B
oar
dM
emb
er
2-
2n
d Award
No Award
Contingency Table:Proportion
Board Member 2 - 1st
Award No Award
Bo
ard
Mem
be
r 2
-2
nd Award
No Award
Contingency Table:Counts
Board Member 2 - 1st
Award No Award
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
57Measurement System Analysis - Attribute
Kappa Between Board Members
To calculate a Kappa for between Board Members, we will use a similar procedure.
We calculate Kappa for the first recommendations of the pair of the Board Members.
NOTE: If there is a Board Member who has poor Within-Board Member repeatability (less than 85%), there is no need to calculate a Between-Board Member rating.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
58Measurement System Analysis - Attribute
Kappa – Board Member 1 to Board Member 2
Number of times both board members agreed the candidate should receive an “Award.”(using their first evaluation)
Contingency Table:Counts
Board Member 1 - 1stAward No Award
No Award 4 17 21
Bo
ard
Me
mb
er
2-
1st
Award 14 5 19
18 22
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
59Measurement System Analysis - Attribute
Kappa Between Board Members
Number of times Board Member 1 recommended “No Award” and Board Member 2 recommended “Award”. (using their first evaluation)
Contingency Table:Counts Board Member 1 - 1st
Award No Award
No Award 4 17 21
Bo
ard
Me
mb
er
2-
1st
Award 14 5 19
18 22
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
60Measurement System Analysis - Attribute
Board Member 1 to Board Member 2 Kappa
Number of times Board Member 1 recommended “Award” and Board Member 2 recommended “No Award” (using their first measurement)
Contingency Table:Counts
Board Member 1 - 1stAward No Award
No Award 4 17 21
Bo
ard
Me
mb
er
2-
1st
Award 14 5 19
18 22
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
61Measurement System Analysis - Attribute
Between Board Member Kappa
Number of times both Board Members agreed the candidate was “No Award” (using their first measurement)
Contingency Table:Counts
Board Member 1 - 1stAward No Award
No Award 4 17 21
Bo
ard
Me
mb
er
2-
1st
Award 14 5 19
18 22
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
62Measurement System Analysis - Attribute
Kappa Between Board Members
Calculate Between-Board Member Kappa:
The lower table represents the data in the top with each cell being represented as a percentage of the total.
Contingency Table:Counts
Board Member 1 - 1st
Award No Award
No Award 4 17 21
Bo
ard
Me
mb
er
2-
1st
Award 14 5 19
0.450 0.550
18 22
Board Member 1 - 1st
Award No Award
Award 0.35 0.125 0.48
No Award 0.100 0.425 0.53
Bo
ard
Me
mb
er
2-
1st
Contingency Table:Proportions
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
63Measurement System Analysis - Attribute
chance
chanceobserved
P
PPK
1
The verbiage for defining Kappa will vary slightly depending on whether we are defining a Within-Board Member Kappa or Between-Board Member Kappa
Remember How to Calculate Kappa?
Pobserved
Proportion of items on which both Board Members agree = proportion both Board Members agree “Award” + proportion both Board Members agree are “No Award”.
Pchance
Proportion of agreements expected by chance = (proportion Board Member 1 recommends “Award” * proportion Board Member 2 says “No Award”) + (proportion Board Member 1 says No Award” * proportion Board Member 2 says “No Award”)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
64Measurement System Analysis - Attribute
Pobserved is the sum of the probabilities on the diagonal:Pobserved =(0.350 + 0.425) = 0.775
Pchance is the probability for each classification multiplied and then summed:Pchance =(0.480*0.450) + (0.530*0.550) = 0.503
Then Kboard Member 1 / 2=(0.775 - 0.503)/(1 - 0.503)=0.548
The Board Members evaluate candidate packets differently too often. The SGS will retrain each Board Member before dismissing a Board Member and finding a replacement.
Calculate Kappa for Board Member 1 to Board Member 2
0.450 0.550
Contingency Table:Proportions
Board Member 1 - 1st
Award No Award
Bo
ard
Me
mb
er
2-
1st
Award 0.35 0.125 0.48
No Award 0.100 0.425 0.53
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
65Measurement System Analysis - Attribute
Improvement Ideas
How might we improve this measurement system?
Additional training
Physical standards/samples
Rater certification (and periodic re-certification) process
Better operational definitions
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
66Measurement System Analysis - Attribute
Kappa Conclusions
Is the current measurement system adequate?
Where would you focus your improvement efforts?
What rater would you want to conduct any training that needs to be done?
Class Challenge: After exposure to Minitab in the following slides, input the data from previous example into Minitab. As homework, perform the analysis and compare the computer output and simplicity with the manual calculations performed in the previous slides.Hint: You will need to stack columns.