CT image testing

CT image testingCT image testing

What is a CT image?What is a CT image?

CT= computed CT= computed tomographytomography

Examines a person Examines a person in “slices”in “slices”

Creates density-Creates density-based imagesbased images

Three different window densitiesThree different window densities

Bone- densest only bone shows Bone- densest only bone shows detail, the rest is too dark to make detail, the rest is too dark to make outout

Soft tissue- moderate density, bone Soft tissue- moderate density, bone is white, soft tissue is detailed, and is white, soft tissue is detailed, and “air space” is too dark“air space” is too dark

Lung- low density, bone and soft Lung- low density, bone and soft tissue white, and “air space” is tissue white, and “air space” is detaileddetailed

Bone WindowBone Window

Soft Tissue WindowSoft Tissue Window

Lung WindowLung Window

What a radiologist has to look atWhat a radiologist has to look at

How diagnoses are decidedHow diagnoses are decided

Look for “abnormalities” in densityLook for “abnormalities” in density

Why is this cumbersome?Why is this cumbersome?

Our project- dare to compareOur project- dare to compare

Test the new Test the new “unified LSU CT “unified LSU CT image”image”

HypothesisHypothesis

Unified LSU CT image:Unified LSU CT image: Equally accurateEqually accurate FasterFaster

MethodsMethods

Our testOur test ROC AnalysisROC Analysis ROCKit SoftwareROCKit Software

ROC ROC Receiver Operating CharacteristicsReceiver Operating CharacteristicsDeveloped in 1950’sDeveloped in 1950’sStatistical decision theory Statistical decision theory Used in business, economics, etcUsed in business, economics, etc

Goal: Is to be a meaningful tool for judging Goal: Is to be a meaningful tool for judging the performance of a diagnostic imaging the performance of a diagnostic imaging systemsystem

Example Example

X X Population Population

yy healthy healthy

z z diseased diseased Gaussian distribution Gaussian distribution Decision boundaryDecision boundary

Interested in optimum decision Interested in optimum decision boundary boundary

EXAMPLEEXAMPLENon-diseased

casesDiseased

cases

Threshold

EXAMPLEEXAMPLENon-diseased

casesDiseased

cases

more typically:

DECISION BOUNDARYDECISION BOUNDARY

TRUE NEGATIVETRUE NEGATIVE TRUE POSITIVETRUE POSITIVE FALSE NEGATIVEFALSE NEGATIVE FALSE POSITIVEFALSE POSITIVE What happens when you place the What happens when you place the

decision boundary?decision boundary?

DECISION BOUNDARYDECISION BOUNDARY

Threshold

Non-diseased

cases

Diseasedcases

FALSE NEGATIVE IN RED

TRUE POSITVE IN GREEN

DECISION DECISION BOUNDARYBOUNDARY

Threshold

Non-diseasedcases

Diseasedcases

FALSE POSITIVEIN WHITE

TRUENEGATIVE IN WHITE

SENSITIVITY TPFSENSITIVITY TPF

Threshold

Non-diseasedcases

Diseasedcases

SENSITIVITY = TP / TP + FNGREEN /GREEN+RED

FNF = 1-TPF

SPECIFICITY SPECIFICITY TNFTNF

Threshold

Non-diseasedcases

Diseasedcases

SPECIFICITY = TN / TN + FPLARGE WHITE / LARGEWHITE + SMALL WHITE

FPF = 1-TNF

PREVALENCEPREVALENCE

One of the important properties of One of the important properties of ROC is that it is prevalence ROC is that it is prevalence independentindependent

Low prevalence is rareLow prevalence is rare ExampleExample

ROC IS NOT PREVALENCE ROC IS NOT PREVALENCE DEPENDENT!DEPENDENT!

ROC ROC CURVECURVE

TP

F,

sens

itivi

tyFPF, 1-specificity

Entire ROC curve

COMPARING CURVESCOMPARING CURVES

ROC EXPERIMENTROC EXPERIMENT

DatasetsDatasets Two optionsTwo options

BINARYBINARY MULT-SCALE RATINGMULT-SCALE RATING

Fitting curvesFitting curves

RockitRockit Who developed Rockit?Who developed Rockit?

-Charles E. Metz @University of Chicago-Charles E. Metz @University of Chicago What is Rockit?What is Rockit?

-curve fitting software calculates-curve fitting software calculates points for a Roc curvepoints for a Roc curve

-calculates max probability of two -calculates max probability of two Gaussian Gaussian

distributionsdistributions What is a Roc Curve?What is a Roc Curve?

-describes how good the imaging systems is -describes how good the imaging systems is and accuracy of the given data and viewer’s and accuracy of the given data and viewer’s resultsresults

Example 1Example 1 100 patients, 40 negative, 60 positive100 patients, 40 negative, 60 positive

GivenGivenPP PP NN PP NN NN P…P…

Confidence RatingConfidence Rating1. Definitely Negative1. Definitely Negative2. Possibly Negative2. Possibly Negative3. Not sure3. Not sure

4. Possibly Positive4. Possibly Positive5. Definitely Positive5. Definitely Positive

This is the viewer’s results This is the viewer’s results 55 44 22 44 11 22 3….3….

Now, from the confidence rating we consider 1=N and 2, 3, 4, 5=P to interpret view’s results.Now, from the confidence rating we consider 1=N and 2, 3, 4, 5=P to interpret view’s results.Interpretation of viewer’s resultInterpretation of viewer’s resultPP PP PP PP NN PP P…P…

Comparing Comparing givengiven data and data and interpretationinterpretationTPTP TNTN FPFP FNFN 4545 25 25 0 0 13 13

(a) TP- disease is present, diagnosed correctly(a) TP- disease is present, diagnosed correctly(b) TN-disease is not present, diagnosed correctly(b) TN-disease is not present, diagnosed correctly(c) FP-person been diagnosed not having the disease, but the disease is present(c) FP-person been diagnosed not having the disease, but the disease is present(d) FN-person been diagnosed having the disease, but the disease is not present.(d) FN-person been diagnosed having the disease, but the disease is not present.

Now, we compute the FPF and TPF. In order to compute the FPF, you need this formula FPF=1-TP/TP+FN and TPF=1-TN/TN+FP.Now, we compute the FPF and TPF. In order to compute the FPF, you need this formula FPF=1-TP/TP+FN and TPF=1-TN/TN+FP.

Therefore, in our case FPF=1-45/45+13= .224 and TPF=1-25/25+0=0Therefore, in our case FPF=1-45/45+13= .224 and TPF=1-25/25+0=0

So, our first point is (.224, 0).So, our first point is (.224, 0).

Example 2Example 2 Now we can compute a second point on the Roc curve.Now we can compute a second point on the Roc curve.

We consider a different confidence rating. We consider a different confidence rating.

1, 2=N and 3, 4, 5=P1, 2=N and 3, 4, 5=PResults based on viewer’s answer from above Results based on viewer’s answer from above PP PP NN PP NN NN NN N….N….

The new Truth table is:The new Truth table is:TPTP TNTN FPFP FNFN 4646 28 28 0 0 15 15

Now the FPF=1-46 / 46+15 =.245Now the FPF=1-46 / 46+15 =.245 and TPF= 1-28/ 28+0 = 0 and TPF= 1-28/ 28+0 = 0

Now, we have our second point (.245, 0)Now, we have our second point (.245, 0)

Based upon the viewer’s results, a third and forth point can be created by changing the confidence Based upon the viewer’s results, a third and forth point can be created by changing the confidence ratings.ratings.1, 2, 3=N 1, 2, 3=N and and 4, 5=P4, 5=P ->third point->third point1, 2, 3, 4=N 1, 2, 3, 4=N and and 5=P5=P ->forth point->forth point

Therefore, five confidence ratings give you four points on the Roc curve.Therefore, five confidence ratings give you four points on the Roc curve.

Rockit inputRockit input For Condition 1: d For Condition 1: d

Enter the Total Number of Actually-Negative Cases (an integer):Enter the Total Number of Actually-Negative Cases (an integer):4040

Beginning with category 1 and separated by blanks,Beginning with category 1 and separated by blanks,Enter (on one line, integers only) the number of responses to Actually-Enter (on one line, integers only) the number of responses to Actually-Negative cases in each category:Negative cases in each category: 1 2 3 4 51 2 3 4 5 23 10 2 3 223 10 2 3 2

For Condition 1: d For Condition 1: d Enter the Total Number of Actually-Positive Cases (an integer):Enter the Total Number of Actually-Positive Cases (an integer):

6060 Beginning with category 1 and separated by blanks,Beginning with category 1 and separated by blanks, Enter (on one line, integers only) the number of responses to Actually-Enter (on one line, integers only) the number of responses to Actually- Positive cases in each category:Positive cases in each category: 1 2 3 4 51 2 3 4 5 2 2 3 10 432 2 3 10 43

RockitRockit

a, b parametersa, b parameters

- a represents - a represents separation of the separation of the Gaussian distributionsGaussian distributions

-b represents width of -b represents width of two Gaussian two Gaussian distributionsdistributions

Open excel template Open excel template to enter a, b to enter a, b parametersparameters

Roc curve is producedRoc curve is produced

Conventional Binormal ROC Curves

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

FPF

TP

F

a=2.10 b=0.90 Az=0.9445 Example