Summary of POD Study Results - CNDE iastate · Floyd W. Spencer Distinguished Member of Technical Staff Sandia National Laboratories ASNT 16th Annual Research Symposium Orlando, Florida

1

Summary of POD Study Results

Floyd W. Spencer

Distinguished Member of Technical StaffSandia National Laboratories

ASNT 16th Annual Research SymposiumOrlando, FloridaMarch 26-30, 2007

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration

under contract DE-AC04-94AL85000.

2

AIRLINE MAINTENANCE FACILITY EXPERIMENT

Realism of Inspection (specimens & location)

Accessibility

Arrangeable layouts

3

2nd set of Specimens

4

POD summary by Facility examples

5

Facility E - Probability of De tection Curves(184 flawed sites in 782 de tection opportunities)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200

crack size (mils)

E1 (132,2)E2 (126,3)E2R (125,6)E3 (135,2)E4 (133,53)

6

Facility G - Probability of Detection Curves(184 flawed sites in 782 de tection opportunities)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.9

1

0 20 40 60 80 100 120 140 160 180 200

crack size (mils)

0.8

G1 (91,5)G1R (94,4)G2 (126,6)G3 (89,10)G4 (124,59)

7

Facility Differences•Facility differences significant

– gauged by inspection-to-inspection variances

•Inspection-to-inspection variation significant– gauged by repeatability within

inspector

8

Proportion of Detects - All Inspections

flaw size (mils)

prop

ortio

n of

det

ectio

ns

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400

Individual flaws

no thresh fit

C=.023 fit

2 points at (812, .98)

9

( ))ln()1()( acaPOD ⋅+Φ⋅−= βα

∑ ∑∑ ⋅⋅+⋅+i

jiijii III ααα

∑ ∑∑ ⋅⋅+⋅+i

jiijii III βββ

Model extension Probit with fitted asymptote

Generalized parameters to include factor effects under study

10

crack angle(Iang)

surface condition(Isurf)

lap splice position(Ipos)

number of cracks at rivet (Inum)

procedure followed(Iproc)

density of cracked rivet sites (Idens)

level 0

11 to 22 degs.

painted low 2 template low (~10%)

level 1

horizontal bare high 1 sliding high (~40%)

Factors modeled in the POD

11

Values are calculated as averages across other factors. All values are in thousandths of an inch.

levelangle surface

/facility

pos flaw # procedure/facility

density

50th percentile 0 72 73 69 63 71 66

50th percentile 1 63 62 66 72 64 69

90th percentile 0 109 115 105 95 109 104

90th percentile 1 96 90 100 110 96 101

Factor Effects in mils

12

Conclusions from Experiment•Individual inspector differences are a major factor affecting inspections

•Environmental factors can influence an inspection

•Inspection misses result that are independent of crack length

13

Automated Ultrasonic Inspection Result Summary

General Goals were to establish•Capability•Assurance of adequate inspection transfer to customer

14

Example of image to be interpreted

15

Lab phase 12 356 16 18 7 9 4 4

Field phase 8 238 15 16 6 7 4 3

Test

secti

ons

Insp

ectio

n Si

tes

< 0.

050

inch

0.05

to 0

.10

inch

0.10

to 0

.15

inch

0.15

to 0

.20

inch

> 0

.20

inch

not m

easu

red

Flaw distributions

16

Experimental Variables and levels for Laboratory experiment

VariableLow level (-1) High level (1) Nominal level (0)

1. time base delay nominal - 0.005 nominal + 0.005 determined in calibration

used 0.35-ch 1 0.355-ch 2

0.36-ch 1 0.365-ch 2 0.355-ch 1 0.36-ch 2

2. depth velocity table value for probe angle - 1 °

table value for probe angle + 1 °

tabled value determined from probe angle

used 85400 in/sec 88500 in/sec 87000 in/sec

3. receiver gain nominal - 0.6 dB nominal + 0.6 dB as determined at time of calibration

used 35.6 dB 36.8 dB 36.2 dB

4. scanner skew 0.25 inch left 0.25 inch right centered

5. probe pressure pressure off nominal arbitrary

used 0 - 1 lbs. indicated 16 lbs indicated 16 lbs indicated on dial

17

Laboratory Results

•Variations in 2 different signal aspects (area and amplitude) quantified with respect to expected variation allowed by procedures

18

Table 3. Effects of Experimental factors on selected signals – as % of nominal

Experimental Factors

Response time delay depth velocity

gain skew probe pressure

time delay * depth vel.

Inner Transducer

Average signal strength

in interaction 7.8 1.2 2.9

Area of flaw signal 11.2 17.0

Average signal strength of flaw

areain interaction 7.1 15.4

Outer Transducer

Average signal strength 4.4 9.1 2.9 6.3

Area of flaw signal in interaction 19.8

Average signal strength of flaw

area4.7 9.9 4.6

19

Field Inspection Results•Gain and time base delays observed in the field data exceeded the range studied in the laboratory phase

• Misidentification of hole from the image not uncommon–Able to be determined from saving

inspectors images and comparing to calls made

20

Average PoDs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.05 0.1 0.15 0.2crack length (inch)

prob

abili

ty

4par

4par align

2par

2par align

21

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2

LJLJLJLJLJUIUIUIUIUIUITITITI

PoD Curves fit to individual inspections

22

Common Image Set

•Nominal run of Laboratory phase•10 inspectors made calls

– on 356 images

23

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.90

1.00

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14crack length (inch)

prob

abili

tyRange of POD fits to calls on common images

0.80

24

Conclusions•Target flaw size (0.050 inch) estimated to have average detection rate of 0.70

•Set-up variation in the field higher than expected

•Decision process a large contributor to observed variation–Lead to development of automated

decision process (neural net)

25

References for full reports• Spencer, F. and Schurman D., “Reliability

Assessment at Airline Inspection Facilities, Volume III: Results of an Eddy Current Inspection Reliability Experiment,” DOT/FAA/CT-92/12,III, May 1995.

• Mullis, R. and MacInnis, T., “C-141 Spanwise Splice Advanced NDI Method,” The First Joint DoD/FAA/NASA Conference on Aging Aircraft" in Ogden, Utah, July 10, 1997. (NDTnet – November 1997, Vol.2 No. 11)

• Andrew, G., MacInnis, T., and Mullis, T. “Second-layer ultrasonic inspection of C-141 splice joints,”Nondestructive Evaluation of Aging Aircraft, Airports, and Aerospace Hardware, eds Rempt, R and Broz, A, SPIE Proceedings Vol. 2954, 1996.