Standardization the properties of objective tests.

StandardizationStandardizationthe properties of the properties of

objective testsobjective tests

Properties of Objective Properties of Objective TestsTestsThere are three standards by There are three standards by

which you can judge an which you can judge an objective testobjective test

StandardizationStandardizationReliabilityReliabilityValidityValidity

Properties of Objective Properties of Objective TestsTestsStandardizationStandardization – scoring & use of – scoring & use of

scores does not vary across scores does not vary across situationssituations

Reliability Reliability – scores are consistent – scores are consistent and remain stable over timeand remain stable over time

ValidityValidity – the test measures what it – the test measures what it intends to measureintends to measure

Standardization PrinciplesStandardization Principles

Objective ScoringObjective Scoring

DirectionsDirections

ConsistencyConsistency

Accuracy and timelinessAccuracy and timeliness


AdministrationAdministration

Appropriate conditions specifiedAppropriate conditions specified

MaterialsMaterials

Probing / CoachingProbing / Coaching


Guidelines for interpretation and Guidelines for interpretation and useuse

With whom?With whom?

For what purpose?For what purpose?

What do high and low scores mean?What do high and low scores mean?


Norm tablesNorm tables

Based on largeBased on large

Representative samplesRepresentative samples

From a defined populationFrom a defined population


Specialized norm tablesSpecialized norm tables

Subgroup differencesSubgroup differences

For example: age, gender, For example: age, gender, race, primary language, etc.race, primary language, etc.


Raw scores and standard Raw scores and standard scores provided where scores provided where appropriateappropriate

Standard scoresStandard scoresPercentile ranksPercentile ranksAge standardized scoresAge standardized scores


Technical manualTechnical manual

Test development processTest development processGuidelines for administration, Guidelines for administration,

scoring, and interpretationscoring, and interpretationNorm tablesNorm tablesMeets standards for Ed. & Psych. Meets standards for Ed. & Psych.

teststests

Norm TablesNorm Tables

Meaningful for interpretation Meaningful for interpretation when:when:

Norm referenced interpretation Norm referenced interpretation meets the goal of the testmeets the goal of the test

Not a criterion referenced testNot a criterion referenced test



Relative position in a group has Relative position in a group has interpretative meaninginterpretative meaning

Examinee is a member of the Examinee is a member of the populationpopulation



The norm sample is large and The norm sample is large and representative of the populationrepresentative of the population

The right norm table is usedThe right norm table is used


All those taking the test for a All those taking the test for a given administration may given administration may work as a norm sample for an work as a norm sample for an admissions or personnel admissions or personnel selection purposeselection purpose


However, the correct reference However, the correct reference group varies by the purposegroup varies by the purpose

Career counselingCareer counselingPlacement in the appropriate Placement in the appropriate

coursescoursesSelection for a remedial programSelection for a remedial program

Interpreting Standard Interpreting Standard ScoresScoresRaw score is transformed into Raw score is transformed into

a standard scorea standard scorez = (score – mean)/SDz = (score – mean)/SDz score = SDs units away from z score = SDs units away from

meanmeanIncludes measure of middle Includes measure of middle

and spreadand spread

Interpreting Standard Interpreting Standard ScoresScoresz = 0, average scorez = 0, average scorez <=-1, low scorez <=-1, low scorez >=1, high scorez >=1, high scorez is converted to some other z is converted to some other

scaling:scaling:MeanMean 5050 100100 500500SDSD 1010 1515 100100

Interpreting Standard Interpreting Standard ScoresScorespp. 42,43,48 in book give pp. 42,43,48 in book give

guidelinesguidelinesEasiest to use when converted to Easiest to use when converted to

percentilespercentiles% of population that scores at or % of population that scores at or

below a given scorebelow a given scoreCan be thought of as a rank out Can be thought of as a rank out

of 100 members of the populationof 100 members of the population

Interpreting Standard Interpreting Standard ScoresScoresCommon interpretation strategies:Common interpretation strategies:

Normal range is middle 68% of the Normal range is middle 68% of the population (T=40-60, z=-1 to 1, population (T=40-60, z=-1 to 1, etc.)etc.)

Low and high scores fall outside Low and high scores fall outside this range (lower and upper 16%)this range (lower and upper 16%)

Interpreting Standard Interpreting Standard ScoresScoresCommon interpretation Common interpretation

strategies:strategies:

Normal range is middle 50% of Normal range is middle 50% of the population (Quartiles 2 & 3)the population (Quartiles 2 & 3)

Low and high scores fall outside Low and high scores fall outside this range (Quartiles 1 and 4)this range (Quartiles 1 and 4)

Interpreting Standard Interpreting Standard ScoresScoresSafer to make broad classification Safer to make broad classification

like “Low”, “Within the normal, or like “Low”, “Within the normal, or expected, range”, or “High” than expected, range”, or “High” than fine distinctions.fine distinctions.

All scores have some All scores have some measurement error in them.measurement error in them.

Look for patterns across the Look for patterns across the battery, across multiple sources.battery, across multiple sources.

An Example from WCCSAn Example from WCCS

Christina, a 1Christina, a 1stst grade student grade student at our school, took the Stanford at our school, took the Stanford Achievement Test last year. Achievement Test last year. Here are her Word Study Skills Here are her Word Study Skills subtest scores.subtest scores.

Number Number Percent Scaled Nat'l ACSI Gradeof Items Correct Correct Score PR PR Equivalent

30 25 83.33% 621 81-7 57-5 3.7

Percent CorrectPercent Correct

The number of correct The number of correct responses, or the raw score, is responses, or the raw score, is divided by the total number of divided by the total number of questions, then multiplied by questions, then multiplied by 100 and expressed as a 100 and expressed as a percentage.percentage.


30 25 83.33% 621 81-7 57-5 3.7

Percent CorrectPercent Correct

Christina gave the correct Christina gave the correct answer to 83.33% of the answer to 83.33% of the questions on the Word Study questions on the Word Study Skills section of the test. Skills section of the test.


30 25 83.33% 621 81-7 57-5 3.7

Scaled ScoreScaled Score

The raw score is standardized The raw score is standardized and normalized, then rescaled and normalized, then rescaled to the desired scaling.to the desired scaling.

z = (Raw Score – Mean) / SDz = (Raw Score – Mean) / SD Scaled Score ≈ 500 + (100*z)Scaled Score ≈ 500 + (100*z)


30 25 83.33% 621 81-7 57-5 3.7


Scaled Scores have many Scaled Scores have many convenient properties from a convenient properties from a statistical standpoint.statistical standpoint.

However, for most people, However, for most people, percentile ranks are easier to percentile ranks are easier to understand.understand.


30 25 83.33% 621 81-7 57-5 3.7


Christina scored more than Christina scored more than one Standard Deviation above one Standard Deviation above average. Her scores are in the average. Her scores are in the above average range.above average range.


30 25 83.33% 621 81-7 57-5 3.7

Percentile RankPercentile Rank

A percentile rank is a statement of A percentile rank is a statement of the percentage of persons in a given the percentage of persons in a given group who fall at or below a given group who fall at or below a given score.score.

The most common way of reporting The most common way of reporting test scores and the easiest to use.test scores and the easiest to use.


30 25 83.33% 621 81-7 57-5 3.7


Christina scored as well or Christina scored as well or better than 81% of all students better than 81% of all students in the nation who took this in the nation who took this section of the test.section of the test.


30 25 83.33% 621 81-7 57-5 3.7


Christina scored as well or Christina scored as well or better than 57% of all students better than 57% of all students in ACSI schools who took this in ACSI schools who took this section of the test.section of the test.


30 25 83.33% 621 81-7 57-5 3.7


This pattern is typical for our This pattern is typical for our students on average.students on average.– ≈ ≈ 8080thth percentile nationally percentile nationally– ≈ ≈ 6060thth percentile for ACSI students percentile for ACSI students– What does this mean?What does this mean?


30 25 83.33% 621 81-7 57-5 3.7

StanineStanine

StaStandard score of ndard score of ninenine units units Developed by the military to Developed by the military to

contain test score information in contain test score information in one column on an IBM punch cardone column on an IBM punch card

Nine groups (1-9), ½ SD, range of Nine groups (1-9), ½ SD, range of PRsPRs


30 25 83.33% 621 81-7 57-5 3.7

StanineStanine

Christina’s scores fall in the 7Christina’s scores fall in the 7thth stanine, or above average stanine, or above average compared to all students nationally.compared to all students nationally.

Christina’s scores fall in the 5Christina’s scores fall in the 5thth stanine, or average for ACSI stanine, or average for ACSI students.students.


30 25 83.33% 621 81-7 57-5 3.7

Grade Equivalent ScoresGrade Equivalent Scores

Attempt to translate test scores Attempt to translate test scores into the grade (grade and month) into the grade (grade and month) when the score is typical.when the score is typical.

Have an intrinsic appeal.Have an intrinsic appeal. Are problematic statistically.Are problematic statistically. Based on extrapolations.Based on extrapolations.


30 25 83.33% 621 81-7 57-5 3.7

Grade Equivalent ScoresGrade Equivalent Scores

Christina, a 1Christina, a 1stst grade student at grade student at our school, in the area of Word our school, in the area of Word Study Skills, is performing at the Study Skills, is performing at the level of a typical 3level of a typical 3rdrd grade grade student in the seventh month of student in the seventh month of the school year (on the 1the school year (on the 1stst grade grade test).test).


30 25 83.33% 621 81-7 57-5 3.7

An SAT ExampleAn SAT Example

Mark, a 12Mark, a 12thth grade student at grade student at our school, took the SAT test our school, took the SAT test last year. Here are his scores.last year. Here are his scores.

Verbal PR Quant. PR Total PR

620 83 570 66 1190 61


Section mean ≈ 500, SD ≈ Section mean ≈ 500, SD ≈ 100100

Range = 200-800 (-3z to +3z)Range = 200-800 (-3z to +3z) Total mean ≈ 1000, SD ≈ 200 Total mean ≈ 1000, SD ≈ 200 Range = 400-1600Range = 400-1600


620 83 570 66 1190 61


Mark scored a 620 on the Mark scored a 620 on the verbal section of the test. His verbal section of the test. His score was more than one score was more than one Standard Deviation above the Standard Deviation above the mean and is considered above mean and is considered above average.average.


620 83 570 66 1190 61


Mark’s score on the verbal Mark’s score on the verbal section of the test was as good section of the test was as good or better than 83% of the or better than 83% of the students who took the test.students who took the test.


620 83 570 66 1190 61


Mark scored a 570 on the Mark scored a 570 on the quantitative section of the test. quantitative section of the test. His score was within the His score was within the normal range and is considered normal range and is considered average.average.


620 83 570 66 1190 61


Mark’s score on the Mark’s score on the quantitative section of the test quantitative section of the test was as good or better than was as good or better than 66% of the students who took 66% of the students who took the test.the test.


620 83 570 66 1190 61


Mark scored a 1190 total score Mark scored a 1190 total score and his score was within the and his score was within the normal range and is considered normal range and is considered average.average.


620 83 570 66 1190 61


Mark’s total score was as good Mark’s total score was as good or better than 61% of the or better than 61% of the students who took the test.students who took the test.


620 83 570 66 1190 61

General PrinciplesGeneral Principles

Tests do not measure innate Tests do not measure innate abilityability

Test scores result from a Test scores result from a combination of:combination of:– Innate abilityInnate ability– Environmental influencesEnvironmental influences– Test taker motivationTest taker motivation– Properties of the test itselfProperties of the test itself

Cautions about Cautions about InterpretationInterpretationA low score in one norm A low score in one norm

group may be high in another, group may be high in another, and vice versa.and vice versa.

A low score on one test will A low score on one test will not necessarily lead to a high not necessarily lead to a high score on another test.score on another test.

Cautions about Cautions about InterpretationInterpretationInterpretation is part art or Interpretation is part art or

clinical intuition and clinical intuition and experience.experience.

Become familiar with case Become familiar with case studies in manuals.studies in manuals.

Standardization the properties of objective tests.

Documents

test norm

test slide

norm tables meaningful

norm sample

use of scores

low scores

timeliness slide

interpretation guidelines