Michigan Assessment Consortium Common Assessment Development Series Module 14 – Presenting the Results of an Assessment.

Michigan Assessment Michigan Assessment ConsortiumConsortium

Common Assessment Common Assessment Development SeriesDevelopment Series

Module 14 –Module 14 –Presenting the ResultsPresenting the Results

of an Assessmentof an Assessment

Developed byDeveloped by

Bruce R. Fay, PhD &Bruce R. Fay, PhD &

Ellen Vorenkamp, EdDEllen Vorenkamp, EdDAssessment ConsultantsAssessment Consultants

Wayne RESAWayne RESA

SupportSupport

The Michigan Assessment Consortium The Michigan Assessment Consortium professional development series in professional development series in common assessment development is common assessment development is funded in part by the funded in part by the Michigan Michigan Association of Intermediate School Association of Intermediate School AdministratorsAdministrators in cooperation with … in cooperation with …

In Module 14 you will learn aboutIn Module 14 you will learn about

Score types…Score types… Standards-based reports…Standards-based reports… Graphical Representations…Graphical Representations…

So, you’ve…So, you’ve…

Developed a test (for use as a ‘common’ Developed a test (for use as a ‘common’ assessment)assessment)

Pilot / field-tested it (right?)Pilot / field-tested it (right?) Looked at the field test results (of course)Looked at the field test results (of course)

Now what?Now what?

Presenting Your ResultsPresenting Your Results

Before you present the results of your test, you Before you present the results of your test, you need to be clear about:need to be clear about:

Who the audience isWho the audience is

Why they are seeing this data? (What?)Why they are seeing this data? (What?)

Why they should care about it? (So what?)Why they should care about it? (So what?)

What you want them to do as a result of seeing it? What you want them to do as a result of seeing it? (Now what?)(Now what?)

SCORE TYPESSCORE TYPES

A score by any other nameA score by any other name

Many score types that you may have Many score types that you may have heard of are really only appropriate for heard of are really only appropriate for Norm-Referenced Tests (NRT), such as Norm-Referenced Tests (NRT), such as percentile rank, stanine, and grade level percentile rank, stanine, and grade level equivalent.equivalent.

Your common assessment is a Criterion-Your common assessment is a Criterion-Referenced Test (CRT), so lets focus on Referenced Test (CRT), so lets focus on score types that are appropriate for that.score types that are appropriate for that.

Raw ScoresRaw Scores

Number of items correct orNumber of items correct or Number of points earnedNumber of points earned

Q? What’s the difference?Q? What’s the difference?

A! None, if each item has the same point A! None, if each item has the same point value, otherwise…value, otherwise…

Scaled ScoreScaled Score(equal weight)(equal weight)

If each test item has the same “weight”, If each test item has the same “weight”, say 1 point (1 if correct, 0 if wrong) then say 1 point (1 if correct, 0 if wrong) then % correct is:% correct is:

The simplest scaled score you can createThe simplest scaled score you can create The same as %points earnedThe same as %points earned Puts the raw score on a scale of 0 – 100Puts the raw score on a scale of 0 – 100

Scaled ScoreScaled Score(unequal weight)(unequal weight)

If each test item does not have the same If each test item does not have the same number of points number of points (there are weighted and/or (there are weighted and/or

partial credit items on the test) partial credit items on the test) thenthen % correct becomes % of total possible % correct becomes % of total possible

points earnedpoints earned You still end up with a 0 – 100 scaleYou still end up with a 0 – 100 scale

% Correct Features (Issues)% Correct Features (Issues)FeaturesFeatures

A “common” scale, as A “common” scale, as in “widely used”in “widely used”

A “common” scale, as A “common” scale, as in “the same in “the same regardless of raw regardless of raw score points”score points”

Intuitively interpretable Intuitively interpretable (maybe)(maybe)

Permits comparisons Permits comparisons between different testsbetween different tests

IssuesIssues

Can/will be Can/will be misinterpretedmisinterpreted

Can make a 10 point Can make a 10 point test and a 100 point test and a 100 point test appear equally test appear equally importantimportant

Widely held belief that Widely held belief that scores in certain scores in certain ranges (60-70, 70-80, ranges (60-70, 70-80, etc.) have some etc.) have some inherent meaninginherent meaning

Interpretation of % CorrectInterpretation of % Correct

Q? Is 50% correct good or bad?Q? Is 50% correct good or bad?

A!: We don’t know yet. We don’t discuss A!: We don’t know yet. We don’t discuss standard–setting until the next module standard–setting until the next module (15).(15).

But most people think it is intuitively But most people think it is intuitively obvious that this is a “bad” score.obvious that this is a “bad” score.

Other ways to scale?Other ways to scale?

Yes, but we don’t really need them…Yes, but we don’t really need them…

STANDARDS-BASED STANDARDS-BASED REPORTSREPORTS

Two kinds of “standards”Two kinds of “standards”

Content Content StandardsStandards

The definition of The definition of the content to be the content to be learned; what learned; what students are to students are to know and be know and be able to doable to do

Performance Performance StandardsStandards

The definition of The definition of how good is good how good is good enough on a test enough on a test to determine if, or to determine if, or the extent to the extent to which, students which, students know and can doknow and can do

Reporting byReporting byContent StandardsContent Standards

This is our concern in this moduleThis is our concern in this module The next module (15) deals with The next module (15) deals with

performance standardsperformance standards

Let’s consider…Let’s consider…

A test covering 5 GLCEs with 5 selected-A test covering 5 GLCEs with 5 selected-response items per GLCE, with each response items per GLCE, with each item worth 1 point (25 points total).item worth 1 point (25 points total).

Q? What does a raw score of 20 (a % Q? What does a raw score of 20 (a % correct scaled score of 80%) mean?correct scaled score of 80%) mean?

A! It dependsA! It depends

Depends on What?Depends on What?

Student AStudent A

GLCE 1: 4/5GLCE 1: 4/5 GLCE 2: 4/5GLCE 2: 4/5 GLCE 3: 4/5GLCE 3: 4/5 GLCE 4: 4/4GLCE 4: 4/4 GLCE 5: 4/5GLCE 5: 4/5

Student BStudent B

GLCE 1: 5/5GLCE 1: 5/5 GLCE 2: 5/5GLCE 2: 5/5 GLCE 3: 5/5GLCE 3: 5/5 GLCE 4: 3/5GLCE 4: 3/5 GLCE 4: 2/5GLCE 4: 2/5

Same or different?

How about these two?How about these two?

Student CStudent C

GLCE 1: 5/5GLCE 1: 5/5 GLCE 2: 5/5GLCE 2: 5/5 GLCE 3: 4/5GLCE 3: 4/5 GLCE 4: 3/5GLCE 4: 3/5 GLCE 5: 3/5GLCE 5: 3/5

Student DStudent D

GLCE 1: 5/5GLCE 1: 5/5 GLCE 2: 5/5GLCE 2: 5/5 GLCE 3: 5/5GLCE 3: 5/5 GLCE 4: 5/5GLCE 4: 5/5 GLCE 5: 0/5GLCE 5: 0/5

These 4 examples all have a raw score of 20 (80% correct) but represent 4 different performances by the students.

Another way to see itAnother way to see it

GLCE A # A % B # B % C # C % D # D %

1 4 80 5 100 5 100 5 100

2 4 80 5 100 5 100 5 100

3 4 80 5 100 4 80 5 100

4 4 80 3 60 3 60 5 100

5 4 80 2 40 3 60 0 0

total 20 80 20 80 20 80 20 80

Scores by “Standard”Scores by “Standard”

Remember, we haven’t set performance Remember, we haven’t set performance standards yet, so we really can’t say what standards yet, so we really can’t say what these scores meanthese scores mean

Even so, 5 out 5 may suggest that a student Even so, 5 out 5 may suggest that a student knows the material and 0 out 5 may suggest knows the material and 0 out 5 may suggest that they don’t that they don’t (depends on item-GLCE match)(depends on item-GLCE match)

However…even though this is a CRT, you can’t However…even though this is a CRT, you can’t make instructional decisions without the make instructional decisions without the context of the overall pattern of scorescontext of the overall pattern of scores

Say what?Say what?

There will often be extreme scores There will often be extreme scores (outliers) that are not representative of (outliers) that are not representative of most of the scores in a set.most of the scores in a set.

Q? What if most of the students scored a Q? What if most of the students scored a 0 or a 1 on GLCE 5 in the example?0 or a 1 on GLCE 5 in the example?

A! Maybe a picture would helpA! Maybe a picture would help

GRAPHICAL GRAPHICAL REPRESENTATIONSREPRESENTATIONS

Or, I can see clearly nowOr, I can see clearly now

Guidelines for Good GraphsGuidelines for Good Graphs

Title & SubtitlesTitle & Subtitles Data Source and Time FrameData Source and Time Frame Axis LabelsAxis Labels LegendLegend Viewable ColorsViewable Colors Readability (3-D doesn’t make it better)Readability (3-D doesn’t make it better)

Appropriate TypeAppropriate Type

Bar GraphsBar Graphs Line GraphsLine Graphs ScatterplotsScatterplots Stem & LeafStem & Leaf Pie Charts (evil)Pie Charts (evil)

Results for 25 studentsResults for 25 students(# scoring at each score point for each GLCE)(# scoring at each score point for each GLCE)

The DataThe Data

Here’s how the spreadsheet is set upHere’s how the spreadsheet is set up

0 1 2 3 4 5GLCE 1 1 2 4 9 6 3GLCE 2 0 1 2 10 7 5GLCE 3 1 4 8 9 3 0GLCE 4 7 4 2 2 4 6GLCE 5 8 7 5 3 2 0

Note: This will be replaced with a table so it looks better

Let’s Assume…Let’s Assume…

We have established that 3 out of 5 on each We have established that 3 out of 5 on each standard is an acceptable standard of evidence standard is an acceptable standard of evidence that a student understands the GLCE in that a student understands the GLCE in question (hey, these were hard items)question (hey, these were hard items)

Then students who score a 3, 4, or 5 on the Then students who score a 3, 4, or 5 on the cluster of items for a GLCE can be considered cluster of items for a GLCE can be considered “proficient” while students with a 2, 1, or 0 are “proficient” while students with a 2, 1, or 0 are not.not.

Proficiency by StandardProficiency by Standard(for 25 Students)(for 25 Students)

GLCE # Not Prof % Not Prof # Prof % Prof

1 7 28 18 72

2 3 12 22 88

3 13 52 12 48

4 13 52 12 48

5 20 80 5 20

This is what the previous data looks like in table form.

Would a picture help?

Proficiency by StandardProficiency by Standard(for 25 Students)(for 25 Students)

Here’s the dataHere’s the data

GLCE NP Prof1 7 182 3 223 13 124 13 125 20 5

Note: this will be replaced with a table so it looks better

Repeated MeasuresRepeated Measures

If you test the same content on more than one If you test the same content on more than one occasion, you can look at your test results over occasion, you can look at your test results over time.time.

As an example, lets look at test results for our As an example, lets look at test results for our class of 25 students on a pre-test, two class of 25 students on a pre-test, two intermediate tests, and a post-test covering the intermediate tests, and a post-test covering the same five GLCEs. We will look only at GLCE same five GLCEs. We will look only at GLCE 1, with 5 points possible each time1, with 5 points possible each time..

The Data – Results for 25 students on The Data – Results for 25 students on GLCE 1 on 4 test administrations by GLCE 1 on 4 test administrations by score pointscore point

Score Points

Pre-Test Test 1 Test 2 Post-Test

0 9 6 3 1

1 6 4 2 1

2 4 3 2 2

3 3 5 6 5

4 2 4 7 9

5 1 3 5 7

(This is a somewhat idealized example), but interpret it with caution!

And here’s the And here’s the picture – Results picture – Results for 25 students on 4 tests by score pointfor 25 students on 4 tests by score point

The Excel spreadsheetThe Excel spreadsheet

Score 5 4 3 2 1 0

Pre-test 9 6 4 3 2 1

Test 1 6 4 3 5 4 3

Test 2 3 2 2 6 7 5

Post-test 1 1 2 5 9 7

Note: This will be replaced with a table for better viewing

ConclusionsConclusions

AudienceAudience PurposePurpose Technical ConsiderationsTechnical Considerations

What? So what? Now what?What? So what? Now what?

Next ModuleNext Module