Measurement System Analysis How-to Guide - Appendices

Measurement System Analysis

How-to Guide - Appendices Version 6.1

August 2013

2 | © 2013 Rolls-Royce plc

MSA How-to Guide

© 2013 Rolls-Royce plc

The information in this document is the property of Rolls-Royce plc and may not be

copied or communicated to a third party, or used for any purpose other than that for

which it is supplied without the express written consent of Rolls-Royce plc.

This information is given in good faith based upon the latest information available to

Rolls-Royce plc, no warranty or representation is given concerning such information,

which must not be taken as establishing any contractual or other commitment binding

upon Rolls-Royce plc or any of its subsidiary or associated companies.

© 2013 Rolls-Royce plc | 3

Step 1 Be Prepared

Step 2

Plan the Study

Step 3

Conduct the Study

Step 4 Type of Study

Continuous Data:

Gauge R&R for continuous data

Attribute Data: Attribute agreement

analysis for attribute data

Step 5

Taking action if the results are unacceptable

Step 6 Maintaining the improvement

1

2

3

4

5

6

This appendices provides supplementary information on how to carry out analysis using Minitab statistical software – together with some of the more detailed analysis of the statistical output.

CONTINUOUS DATA: Appendix 1:

Setting up and randomising the

spreadsheet in Minitab

CONTINUOUS DATA: Appendix 2:

Entering the data in Minitab

ATTRIBUTE DATA: Appendix 6: Setting up and randomising the spreadsheet in Minitab

ATTRIBUTE DATA Appendix 7: Entering the data in Minitab

Appendix 3: Carrying out Gauge R&R

in Minitab

Appendix 8: Carrying out Attribute Agreement Analysis in Minitab

Appendix 4: Supplementary Information on

Interpreting the Graphical Output from Gauge R&R in Minitab

Appendix 9: Supplementary Information on Interpreting the Output from Attribute Agreement Analysis


MSA How-to Guide

1. Setting-up and Randomising the Spreadsheet in Minitab

2. Entering the Data in Minitab

3. Carrying out Gauge R&R in Minitab

4. Supplementary Information on interpreting the output

5. FAQ for Gauge RR

Continuous Data

In This Section:


Setting up and randomising the worksheet for a Gauge R&R Study for

Variable Data 1. The starting point for setting up the worksheet for a variable data Gauge R&R

study is the same regardless of which randomisation method for the

worksheet is required. To begin go to:

Stat > Quality Tools > Gage Study > Create Gage R&R Study Worksheet

2) Enter the identity of the parts to be used

1) Enter the quantity of parts to be studied

3) Enter the number of people in the study

4) Enter the identities of the people in the study

5) Enter the number of times that each person will check each part

Appendix 1: Setting-up and Randomising the Spreadsheet in Minitab

Continuous Data

2. Complete the dialogue box for the required detail:


MSA How-to Guide

This gives you 3 options to randomise the worksheet:

3. Click on the options box

Options

At this stage you must decide which randomisation method to use taking into consideration the practicalities of running the experiment and the most economical use

of people’s time.

The different options are each described below.


Continuous Data


a. Do not randomise: As it states, this option does not randomise the data. This

option will sequence the parts then the people for each part and provide a run

order column as shown below.

1) Use the ‘Options’ to confirm selection

3) Note the sequence of parts and people

2) Make the selection and click OK will then generate a worksheet for the study


Continuous Data


MSA How-to Guide

b. Randomise all runs: This will completely randomise the order that the

measurements are taken in as shown below. This is useful to prevent the

appraisers from memorising their previous measurements and also to reduce the

impact of time related factors. It does however require all of the appraisers to be

present at once which can be impractical in many situations such as where

different shifts are worked.

As a facilitator, it can also be useful to preserve the ‘standard’ (un-randomised)

order by selecting the option ‘Store standard run order in worksheet’.

Following data collection and analysis the standard order can be used to re-sort the

recorded data so that the pattern of collection may give an insight into what happened.

This should only be done if the measurement system analysis study is not clearly

acceptable and in this case can be useful in identifying combinations which were

awkward for the appraisers.



1) Use the ‘Options’ to confirm selection 2) Check the box to

include standard order column


Continuous Data


c. Randomise runs within operator: This will prevent memory of measurements by

the people undertaking the study but preserves the appraiser sequence. This

enables a study appraiser’s time to be managed as only one appraiser needs to

be present at specified times. As a facilitator, it can also be useful to preserve

the ‘standard’ (non-randomised) order by selecting the option.

This is the most commonly used option;

On completion of Note 4, Minitab returns you to previous dialogue box (Create Gage

R&R Worksheet) then press OK on this screen.

Minitab will now generate the worksheet, you will need to add in your data column and

collect the data before running the study.




2) Check the box to include standard order column


Continuous Data


MSA How-to Guide

Maintaining Data Integrity

It is often overlooked that data integrity starts when the data is entered. In statistical

software such as Minitab it is common to see data formatting and entry errors causing

issues.

The two most common issues to be aware of are as follows:

1) Areas of the worksheet have been previously used OR the wrong sort of data

has been entered resulting in the column being in the wrong data format

The wrong type of data format for columns then has the effect of hiding columns that

are expected to be numeric (or vice versa) when conducting an MSA study.

The most common occurrences of this is when a space is added somewhere in the

column or when the letter O is used instead of 0 (zero). In both cases, even if the

typing error is rectified this will change the format of the column from numeric to text

format.

Text format columns can be identified by the addition of a ‘T’ to the column number as

in the example above.

A ‘space’ was typed in and turns the column type to text

Appendix 2: Entering the Data in Minitab

Continuous Data


2) The second issue is that of human mistakes when entering the data. To guard against this, the facilitator of the measurement study must control the

study to maintain the concentration, time, speed and discipline required to

type/record each data point. In addition to this, it is possible to assist the person entering the data to select the

correct cell by highlighting the line (descriptions and details) of the active entry.

The example shown has used the option ‘randomise runs within operators’ with the

next entry being from appraiser 2 for part identity 9. It can also be very beneficial to record comments when entries are typed. This

additional information can be useful for analysis where the Measurement System is

not acceptable and further investigation is needed.

Data Type Considerations Types of data are very specific to each MSA study. For Gauge R&R studies the data

is variable and has to be the same units of measure as the operating process. For Attribute Agreement Analysis this is attribute data BUT this can be in the format of

whole (count or scale) numbers or text values.

Left click on the row number will highlight the complete row


Continuous Data


MSA How-to Guide

Appendix 3: Running the Analysis

Continuous Data

To run the analysis then use the menu commands: Stat> Quality Tools> Gage Study> Gage R&R Study (crossed)


A dialogue box will appear. Enter the data into the fields as shown below:

1. Minitab will display in

this window, appropriate

elements of the

worksheet for selection,

transfer the columns for

parts appraisers and

measurement data as

shown.


Continuous Data

2. Click against ANOVA in “Method of

Analysis” This should be the default as Minitab

opens the dialogue box, however should

always be checked

3. Then click OK


MSA How-to Guide

Click on “Options” to enter the tolerance of the characteristic being

measured. For our example this is 0.5mm. Enter 0.5 into the “Upper spec – Lower spec”

Click on “Do not display percentage contribution” & “Do not display percentage study

variation”.

Click “OK” only once. [This simplifies the graphical output to remove graphs not necessary for our

analysis]


Continuous Data


Click on “Gage Info” to enter the relevant equipment, team and

information for the study. It is also good practice to record the date of the study for future reference

Fill out the details requested Click “OK” then “OK” again.


Continuous Data


MSA How-to Guide

The Graphical output will appear as below.

Click on Show sessions folder icon to review the numerical

output”

Part-to-PartReprodRepeatGage R&R

200

100

0

Perc

ent

% Tolerance

10 9 8 7 6 5 4 3 2 110 9 8 7 6 5 4 3 2 110 9 8 7 6 5 4 3 2 1

0.10

0.05

0.00

Parts

Sam

ple

Range

_R=0.0307

UCL=0.0789

LCL=0

1 2 3

10 9 8 7 6 5 4 3 2 110 9 8 7 6 5 4 3 2 110 9 8 7 6 5 4 3 2 1

2.00

1.75

1.50

Parts

Sam

ple

Mean

__X=1.819UCL=1.8504LCL=1.7876

1 2 3

10987654321

2.00

1.75

1.50

Parts

321

2.00

1.75

1.50

Appraiser

10987654321

2.00

1.75

1.50

Parts

Avera

ge

1

2

3

Appraiser

Gage name: v ernier caliper

Date of study : 16th A ug 2006

Reported by : HA SF A D

Tolerance:

M isc: Measurement Sy stem A naly sis

Components of Variation

R Chart by Appraiser

Xbar Chart by Appraiser

Measurement by Parts

Measurement by Appraiser

Parts * Appraiser Interaction

Gage R&R (ANOVA) for Measurement

Appendix 3: The Numerical Output

Continuous Data


Co

mp

on

en

ts o

f Va

riatio

n

In the following appendix the results and interpretation is explained for each of the six

graphs in the Minitab Graphical Output. Please note that all of the statistical analysis and

graphs should be considered before making conclusions for the study and for the potential

actions required. Note also that ‘no action required’ is a possible and valid conclusion.

Appendix 4: Supplementary Information on interpreting the Graphical Analysis

Output from Gauge R&R Studies

Continuous Data

The first graph to look at is the “Components of Variation” on the top right

of the graphical output.

This graph shows where most of the variation in the study came from. The Gauge R&R column shows the % Tolerance taken up by the

measurement system variation. Remember this was 76.99%

If the Part-to-Part columns are high (or very high compared to the others)

this tells us that most of the variation in the study was due to the fact that

the parts being measured were not identical (which we would expect).

If the Repeat columns are high compared to the others, this indicates that

there is a problem with Repeatability (i.e. one or more of the appraisers is

inconsistent with themselves). The remaining graphs will help us

investigate this further.

If the Reprod columns are high compared to the others, this indicates that

there is a problem with Reproducibility (i.e. some of the appraisers are

inconsistent with each other). The remaining graphs will help us

investigate this further.

The Gauge R&R column is the variation component total for Repeat and

Reprod.

In cases such as this example where a problem is identified with the

repeatability and/or the reproducibility of the measurement system then

the remaining graphs should be examined to investigate further. Where no problem is identified from the analysis of the components of

variation then there is no need to examine the remaining graphs


MSA How-to Guide

Summary – Components of Variation Graph



Continuous Data

* 30% is a generally used acceptance criteria, however manufacturing standards may have tighter requirements. Be sure to consult the relevant measurement standards for your area. Details are contained within the SABRe Supplier Management System Requirements document.


Gra

ph

ica

l Data

Next we will look at the R Chart by Appraiser graph. This chart shows the Range of the results for each appraiser for each of the 10 parts.

Refer to SPC “How to” for more information on R charts.



Continuous Data

Interpreting the Graphical Data

For a perfectly consistent measurement system, all of the ranges on

the graph would be zero i.e. each part would be measured the same

giving no (zero) range.

However, it is unlikely that they will all be zero, therefore we use this

chart to help us identify any measurements of concern.

We interpret this graph by saying that any point which is above the

upper red line is worth investigating, as this indicates that the range of

results for that appraiser and part was higher than expected.

So in the case study example we can see that appraiser 2 has a

bigger range than the other appraisers. Julie and the team make note

to ask appraiser 2 if they did anything different from the instructions

given and move on the next graph


MSA How-to Guide



Continuous Data

Summary – R Chart by Appraiser


Inte

rpre

tatio

n o

f gra

ph

ica

l ou

tpu

t

If you are not sure on interpreting control limits on an Xbar chart then ask a

local Black Belt to help you choose the most appropriate type of MSA.



Continuous Data

Next Julie and the team look at the Xbar Chart by Appraiser graph –

this is the graph from the case study.

This graph shows the average measurement for each part and

appraiser.

Ideally we want the patterns of the data to be identical for all 3

appraisers. If they are not, we should investigate.

The 2 red lines on the chart are control limits. We would expect all at

least 50% of the points to lie outside the control limits (red lines) on

this chart. This is different and the opposite to the conventional use of

SPC charts.

As we can, even though we do have the majority of parts outside the

control limits we can also see the patterns for each appraiser look

different. This is again an indication of poor Reproducibility.

Julie and the team note the results of this graph and ask each

appraiser what they did to identify differences then move to the next

one.


MSA How-to Guide



Continuous Data

Summary – Xbar Chart by Appraiser


Inte

rpre

tatio

n o

f gra

ph

ica

l ou

tpu

t

Next Julie and the team look at the Measure by Part graph – this is the graph from the

case study.



Continuous Data


So in the case study example we can see that:

This graph shows circles for all measured values of each part,

together with the average values for each part (shown by ‘crossed

circles’.)

The average values (crossed circles) are connected by the straight

lines

The graph allows us to compare how consistent the measurements

for each of the parts were in the study.

If the measurement system is consistent, there should be very little

scatter between the measurements for each individual part (in other

words, the circles for each part should almost be on top of each other

or overlapping).

We interpret this graph by saying that any part for which there is a

noticeably larger spread in the results, might be worth investigating. In this case parts 10, 8, 4 & 5 appear to have greater variation than

the rest of the parts. The team need to consider why these parts were

more difficult to measure? Julie and the team note this finding and

move on to the next graph.


MSA How-to Guide



Continuous Data

Summary – Measurement by Parts


Inte

rpre

tatio

n o

f gra

ph

ica

l ou

tpu

t



Continuous Data


Next we will look at the Measurement by Appraiser graph:

This graph shows a box plot for all measured values of each item,

together with the average measurement for each appraiser (shown by

‘crossed circles’).

The average values (crossed circles) are connected by the straight

lines. If the measurement system is perfectly consistent, we would

expect the average values for the 3 appraisers to be the same – in

which case the connecting lines would be horizontal.

We would also expect the spread of results (boxes and whiskers) for

all 3 appraisers to be the same (however, unlike the previous graph,

we wouldn’t necessarily expect the spread to be small, as the results

for all of the parts are shown against each appraiser).

We interpret this graph by saying that if, for any appraiser, there is a

noticeably larger spread in results, or the average value is noticeably

different from the others, this might be worth investigating.

So in the case study example we can see that appraiser 2 has a

larger spread of result than the other appraisers. We can also see the

average values line is not straight indicating a Reproducibility

problem.

Again Julie and the team note the findings and move onto the final

graph.


MSA How-to Guide



Continuous Data

Summary – Measure by Appraiser Graph


Inte

rpre

tatio

n o

f gra

ph

ica

l ou

tpu

t



Continuous Data


Finally, we will look at the Part*Appraiser Interaction graph:

This graph overlays the average measurements for each item as

measured by each person.

If the measurement system is perfectly consistent, we would expect

all of the lines to be on top of each other so only one line is seen.

Overlaying lines is the ideal situation here, however there can be

occasions when parallel lines occur. This would indicate that the parts

and appraiser interaction is consistent BUT bias between appraisers

exists.

We interpret this graph by saying that if any of the lines is noticeably

separate from the other 2 lines (for one or more of the parts), this is

worth investigating.

Julie and the team agree that this graph confirms some of their

thoughts from the previous graphs. They are now ready to summarise

their findings.


MSA How-to Guide



Continuous Data

Summary – Part*Appraiser Interaction




Continuous Data

Use of Gauge Run Charts

One additional graphical tool which can be used to assess differences in measurements between different operators and different parts is a Gauge Run Chart. You can use a gauge run chart in combination with a Gage R&R Study to help determine what is causing the variability in the measuring system. Create a gauge run chart as follows: 1. To begin go to Stat > Quality Tools > Gage Study > Gage Run Chart

2. Complete the dialogue box for the required details

Then click ‘OK’

Enter the columns for

parts appraisers and

measurement data as

shown.

Click on ‘Gage Info’ to

enter the relevant

equipment and study

references

If known the historical process mean

can be entered and will be plotted as

a reference line on the graph


MSA How-to Guide



Continuous Data

Use of Gauge Run Charts

The graphical output will appear as shown: This is a plot of all of the observations by operator (denoted by different colours) and by part number (each box numbered 1 – 10 represents one of the 10 parts). The horizontal reference line is the overall mean of the measurements. The plot allows you to see if any patterns are evident in the data. For instance, you might see that one operator consistently measures higher than the others or that the measurements on certain parts vary more when compared to other parts. Here for example you can see that for part 4, appraiser 1 (in black) has measured higher than the other two appraisers. You can also see for part 10 noticeable difference between the measurements of the three appraisers. Looking at the repeatability within appraisers, for parts 2 and 8 it can be seen that appraiser 2 (in red) noticeably differs in their three measurements indicating a repeatability problem.


Appendix 5: FAQ for Gauge R&R (Typical Manufacturing Questions)

Continuous Data

Q: “Where can I find the acceptance criteria for MSA in manufacturing?” A: “The criteria and other information are contained within SABRe Supplier

Management System Requirements document. Further guidance information on

measurement and inspection is available on the supplier portal, specifically in the Guide

to Dimensional Measurement Equipment document.”

Q: “We only have 2 operators using this gauge, shall I take part in the study to

make the numbers up?” A: “Not unless the operator is trained to use the gauge and familiar with the component

being measured. Having an untrained operator may result in the failure of the study due

to the stability and/or reproducibility of the measurements due to the untrained operator.

It would be far better to compromise on the amount of measurement readings than to

perform a study that is not representative of the way the process works” Q: “We cannot get access to 10 parts, will 5 do?” A: “Lowering the sample size will affect the uncertainty of the test. However there are

times when this may be required. In difficult situations a compromise may be required

but the analyst should be mindful of this when interpreting the data. For example if a

gauge R&R returns 19% of tolerance with only 5 parts there is a reasonable case for

either acquiring further parts for study or asking the operators to repeat the

measurements 4 or 5 times rather than the usual 3” Q: “We can’t get parts that represent the full process variation as the only ones

we have are from the same batch. What should we do?” A: “If the sample does not represent the true process variation then this will affect the

results of gauge R&R against study variance, the number of distinct categories and the

limits on the X bar chart on the R&R output. If this is the case 2 options exist. Either proceed and study the % Gauge R&R against

tolerance.” Q: “Our gauge measures hundreds of features, do I have to run gauge R&R study

on each of them or can I use read-across methodology?” A: “The Quality Management System requires that all product features/characteristics

are measured using capable measurement systems. That said there are situations

where similar features measured with the same gauge presents an opportunity to

demonstrate capability without direct study of every single feature as long as this is

done robustly. This must however be done in a robust and traceable way. Read across

is not permitted on CMM equipment, not because the CMM’s tend to be incapable but

because of the risk of program errors due to the manual nature of the program

creation.”


MSA How-to Guide

Appendix 5: FAQ for Gauge R&R

Continuous Data

Q: “The gauge is automated (there is no operator influence)! What do I do?” A: “First be sure that there really is not any operator influence – for instance if there is

a setup process which is manual then this may lead to reproducibility problems. If the

gauge is completely automated then a study can be performed with only 1 operator.

The gauge R&R study will report only on the repeatability element of the gauge R&R. If

there is the opportunity to study two gauges simultaneously (e.g. 2 CMM’s) these can

be identified within the R&R study to allow the reproducibility due to different

equipment rather than operator.” Q: “I have a surface finish gauge, and it keeps failing R&R. What do I do?” A:”Some gauges are notoriously difficult to perform gauge R&R on. Some gauges are

the best available for a given measurement. In this situation contact a measurement

practitioner or Metrologist.” Q: “Under what circumstances should I repeat the gauge R&R?” A: “A gauge R&R should be repeated whenever the process (either measurement

process or manufacturing process) changes significantly or the part tolerance is

changed. Also when turnover of labour is high” Q: “My gauge R&R against tolerance is very good but I only get 1 distinct

category. Is my measurement process good or not?” A: “It is likely that the parts selected for the study are not representative of the total

process variation. This will affect the %R&R against study variance, the number of

distinct categories and the X bar chart. If the parts are representative then the

measurement process is not adequate for the application of SPC analysis as the

majority of the variation seen will be from the measurement system and not the

underlying process.” Q: “How many decimal places on my gauge shall I use when conducting the

gauge study?” A: “As a minimum you should ensure that the study represents the requirements on the

part drawing but the more the better. For instance if the drawing requires measurement

to 2 decimal places, run the gauge R&R to 3.” Q: “My gauge passes the gauge R&R study. Is this all I need to consider?” A: “No – gauge R&R will not highlight gauge bias. For instance it is possible to be

repeatibly wrong. Comparison with a known standard will enable you to study the

amount of gauge bias. Calibration is not done on production parts, real parts can

introduce large differences.”


6. Setting-up and Randomising the Spreadsheet in Minitab

7. Entering the Data in Minitab

8. Carrying out Attribute Agreement Analysis in Minitab

9. Supplementary Information on how to interpret the output

Attribute Data

In This Section:


MSA How-to Guide

Attribute Agreement Analysis Worksheet Configurations

1. Setting up the worksheet for an attribute data, Attribute Agreement Analysis is

very similar to a Gauge R&R study where, the worksheet construction all start

from an identical point independent of which randomisation for the worksheet is

used. This is found at:

Stat > Quality Tools > Create Attribute Agreement Analysis Worksheet


Attribute Data


2. Complete the dialogue box for the required detail:

3) Enter the identity of the parts to be used and the text standard

2) Enter the quantity of parts to be studied

4) Enter the number of people in the study

5) Enter the identities of the people in the study

6) Enter the number of times each person is to inspect each part

Dropdown Selection Sample Standard/attribute unknown: This provides a worksheet very similar to that of a

Gauge R&R study but does need to have an additional column manually added so

that a ‘standard’ agreement can be compared to it. This is the most flexible option. Sample Standard/attribute in text: This constructs the worksheet with the additional

column of text ‘standards’ for comparison, i.e. When comparing judgements such as

good or bad against the standard which is also stated as good or bad. This is the

option used in the following pages. Sample Standard/attribute in numbers: This constructs the worksheet with the

additional column of numerical ‘standards’ for comparison, i.e. When judgements are

made using a scale (often 1 to 5 or 1 to 10), and the standard should be an exact

match. Sample Standard/attribute in worksheet: This favours manual lists already in the

worksheet and provides selection of those column references for the study.

1) There are 4 drop down selections here (see below)


Attribute Data


MSA How-to Guide

This gives you 3 options to randomise the worksheet:

3. Click on the options box

Options


Attribute Data



3) Note the sequence of parts, people and the ‘standard’

2) Make the selection and click OK to generate a worksheet for the study

a. Do not randomise: As it states, this option does not randomise the data. This

option will sequence the parts then the people for each part and provide a run

order column as shown below.


Attribute Data


MSA How-to Guide





b. Randomise all runs: This will completely randomise the order that the

measurements are taken in as shown below. This is useful to prevent the

appraisers from memorising their previous measurements and also to reduce the

impact of time related factors. It does however require all of the appraisers to be

present at once which can be impractical in many situations such as where

different shifts are worked.

As a facilitator, it can also be useful to preserve the ‘standard’ (un-randomised)

order by selecting the option ‘Store standard run order in worksheet’.

Following data collection and analysis the standard order can be used to re-sort the

recorded data so that the pattern of collection may give an insight into what happened.

This should only be done if the measurement system analysis study is not clearly

acceptable and in this case can be useful in identifying combinations which were

awkward for the appraisers.


Attribute Data






c. Randomise runs within operator: This will prevent memory of measurements by

the people undertaking the study but preserves the appraiser sequence. This

enables a study appraiser’s time to be managed as only one appraiser needs to

be present at specified times. As a facilitator, it can also be useful to preserve

the ‘standard’ (non-randomised) order by selecting the option.

This is the most commonly used option.


Attribute Data


MSA How-to Guide

Maintaining Data Integrity It is often overlooked that data integrity starts when the data is entered. In statistical

software such as Minitab it is common to see data formatting and entry errors causing

issues.

The two most common issues to be aware of are as follows:

1) Areas of the worksheet have been previously used OR the wrong sort of data

has been entered resulting in the column being in the wrong data format

The wrong type of data format for columns then has the effect of hiding columns that

are expected to be numeric (or vice versa) when conducting an MSA study.

The most common occurrences of this is when a space is added somewhere in the

column or when the letter O is used instead of 0 (zero). In both cases, even if the

typing error is rectified this will change the format of the column from numeric to text

format.

Text format columns can be identified by the addition of a ‘T’ to the column number as

in the example above.

A ‘space’ was typed in and turns the column type to text


Attribute Data


2) The second issue is that of human mistakes when entering the data. To guard against this, the facilitator of the measurement study must control the

study to maintain the concentration, time, speed and discipline required to

type/record each data point. In addition to this, it is possible to assist the person entering the data to select the

correct cell by highlighting the line (descriptions and details) of the active entry.

The example shown has used the option ‘randomise runs within operators’ with the

next entry being from appraiser 2 for part identity 9. It can also be very beneficial to record comments when entries are typed. This

additional information can be useful for analysis where the Measurement System is

not acceptable and further investigation is needed.

Data Type Considerations Types of data are very specific to each MSA study. For Gauge R&R studies the data

is variable and has to be the same units of measure as the operating process. For Attribute Agreement Analysis this is attribute data BUT this can be in the format of

whole (count or scale) numbers or text values.

Left click on the row number will highlight the complete row


Attribute Data


MSA How-to Guide

To run an Attribute Measurement System Analysis (MSA) then use the menu commands:

Stat>

Quality Tools> Attribute Agreement Analysis


Attribute Data


A dialogue box will appear. Enter the data into the fields as shown below:

Click to confirm data is listed downwards (stacked) This is the format Minitab generates for the worksheet

Enter the column containing the ‘standard’ to be compared to

Only check this box when a scale or multiple class of judgement is used


Attribute Data


MSA How-to Guide

It is good practice to use the ‘Information’ button to record details about the MSA study directly into the displayed results – click on Information button

The information dialogue will appear, so that you can complete the details as appropriate.

Once completed, click OK button to close the information screen, then click OK again to close on main window.


Attribute Data


Appendix 8: The Graphical Output

Attribute Data

The Graphical output will appear as below.

Click on Show sessions folder icon to review the numerical output”


MSA How-to Guide

Appendix 8: The Numerical Output

Attribute Data

Click on Show graphs folder icon to return to the graphical output.


Click on Show worksheet folder icon to return to the worksheet

Appendix 8: Running the Worksheet

Attribute Data


MSA How-to Guide

Attribute Agreement Analysis Worksheet Configurations

1. Setting up the worksheet for an attribute data, Attribute Agreement Analysis is

very similar to a Gauge R&R study where, the worksheet construction all start

from an identical point independent of which randomisation for the worksheet is

used. This is found at:

Stat > Quality Tools > Create Attribute Agreement Analysis Worksheet

Attribute Data

Appendix 9: Supplementary Information on interpreting the Output of Attribute

Agreement Analysis


Interpretation

of graphical

output

The ‘blue’ dots have been explained as the actual proportion of agreement within the

sample of parts appraised in the study. Also shown are the ‘red’ lines which end in a

cross. These indicate the confidence intervals for each Appraiser and for each

Appraiser vs. Standard.

The actual numbers for each confidence interval are recorded in the session window

and used in plotting this graph.

These confidence intervals take into account the fact that the actual agreement %

calculated (as represented by the blue dot) is based only on a relatively small sample

of data. If it was possible to know the ‘true’ agreement % of the appraiser (based on

every part they ever inspected) then this % would be likely to be different from the %

seen in the study sample. The confidence interval indicates the possible range of

values that the ‘true’ % agreement could be. Minitab defaults to a confidence level of

95%. This means that we can interpret the confidence interval for Appraiser 1 for

example as saying “I am 95% confident that the true % of within appraiser agreement

for appraiser 1 is between 62.1% and 96.8% (which is the range indicated by the blue

crosses and red line).

In addition to the ‘How to guide: Measurement System Analysis’ the following pages

give supplementary information on some of the statistical concepts to deepen your

understanding of how to fully interpret the output

Appendix

6

Attribute Data


Agreement Analysis


MSA How-to Guide Attribute Data


Agreement Analysis

Each ‘red’ line and two crosses should be as short as possible indicating less potential

error in the ‘blue’ dot point indication, when using 95% confidence (the default level for

most statistical analysis).

These confidence intervals are calculated using the F distribution which also requires

a thing called degrees of freedom. The larger the degrees of freedom, which can be

influenced by using a larger selection of items for inspection (sample size) and the

quantity of matched agreements (such as Pass / Pass and the standard also as Pass)

also contribute to reducing the confidence intervals.

The F distribution is not symmetrical and therefore the confidence intervals can look a

little odd, one (usually the lower) will be longer than the other.

Interpreting the Lower Confidence Interval It may be desirable in some instances where the quality of the measurement system is

highly critical to use the lower confidence interval (lower red cross) rather than the

observed agreement (blue dot) when assessing the measurement system against the

rules of thumb described on page 46 . This assesses the measurement system based

on the worst case scenario.

You must however take into consideration that the sample size will significantly

influence this. Where the sample size is small (less than 20 parts for example) the

lower confidence intervals will nearly always fall below 70% even when the average

agreement within the sample is fairly good). In these circumstances you should

discuss with a Black Belt an appropriate sample size to use for the assessment of

critical measurement systems. If the measurement needed is highly critical then

consideration should also be given to whether it is possible to redesign the

measurement system to use variable data rather than attribute and assess the system

using Gauge R&R instead.


The Kappa (correctly quoted as Fleiss Kappa) works on a scale of -1 to 1, where -1

indicates total disagreement between each measurement run and the standard. A

value of 0 (zero) indicates a 50:50 chance of correctly assessing the part which implies

appraisers are ‘guessing’ whether to pass or fail the part. The best possible outcome is

a Kappa value or 1 which indicates total agreement to each round of measures and to

the standard.

Within Appraisers table is shown for a simple Fail or Pass judgement which gives a

Kappa value similar BUT not the same as the percent listing.

Fleiss’ Kappa becomes very useful when a scale or multiple class of judgement is

used which then has a Kappa value indicating agreement for each scale value. An example of this could be shade judgements, where the ends of the scale, shades A

or B (light), I or J (dark) have very high Kappa values near to 1 and middle shades

such as E or F have lower (approximately 0.778291 say) showing for that person these

were harder to judge as correct.

Interpretation

of the

numerical

output

Each of the percentage results shown in the session window tables have been

explained in the ‘How to guide: Measurement System Analysis’. This section will

explain the other statistics such as the ‘Kappa’ listed and the P-value. Each of the

tables is constructed with the same format and therefore the ‘Within Appraisers’ table

will be used for these explanations.

Attribute Data


Agreement Analysis


MSA How-to Guide

The P-value (which is short for the probability value) is a back-up statistic describing

the chance of having a Fleiss Kappa near 0 (zero). As the tabular notation shows, the judgement is made about the ‘chance’ (probability)

where the lower the P (vs. > 0) column p-value shows a low number, the less

statistical chance the Kappa values are 0 (zero) or less. In other words, the p-values of 0.0000 are very good and what should be seen.

Interpretation

of the

numerical

output

P-value Judgements

Attribute Data


Agreement Analysis


Change History

Revision Date Description of Change Author Owner Approval

V6.1 20/08/2013 Guide reformatted for

SABRe D Prodger D Prodger D Prodger

Document update policy

This document may be updated periodically. Major amendments will be shown as an update from one

revision number to a higher revision number (e.g. revision 1 to revision 2) and therefore the content of

the higher revision will be regarded as the latest requirements. A minor amendment will be shown as a

number change after a decimal point (e.g. revision 1.1 to revision 1.2) and therefore any of these

revisions may be regarded as the latest requirements until a major amendment is introduced

Measurement System Analysis

How-to Guide - Appendices

© Rolls-Royce plc 2013

The information in this document is the property of Rolls-Royce

plc and may not be copied, communicated to a third party or

used for any purpose other than that for which it is supplied,

without the express written consent of Rolls-Royce plc.

While the information is given in good faith based upon the

latest information available to Rolls-Royce plc, no warranty or

representation is given concerning such information, which must

not be taken as establishing any contractual or other

commitment binding upon Rolls-Royce plc or any of its

subsidiary or associated companies.

Measurement System Analysis How-to Guide - Appendices

Documents