What Should You Consider When Talking About Technical Assessments? John M. Townsend Tennessee Board of Regents NACTEI 27 th Annual Conference.

What Should You What Should You Consider When Talking Consider When Talking

About Technical About Technical Assessments?Assessments?

John M. TownsendJohn M. TownsendTennessee Board of RegentsTennessee Board of Regents

NACTEI 27NACTEI 27thth Annual Conference Annual Conference

Why are you assessing?Why are you assessing?

Most important questionMost important question

– For Perkins IV?For Perkins IV?

– For Program Improvement?For Program Improvement?

– For student attainment?For student attainment?

If For Student AttainmentIf For Student Attainment

At what level is appropriate?At what level is appropriate?

– Entry levelEntry level

– Mastery levelMastery level

– Industry certification levelIndustry certification level

When do you assess?When do you assess?

End-of-programEnd-of-program

End-of-courseEnd-of-course

End of PathwayEnd of Pathway

Cluster relatedCluster related

Norm- or Competency-Norm- or Competency-referencedreferenced

Norm-referencedNorm-referenced

– Comparison Comparison among among individualsindividuals

– Demonstrates Demonstrates changes of changes of students as a students as a groupgroup

Competency-Competency-referencedreferenced– Attainment of an Attainment of an

individualindividual– Measurable Measurable

learning learning outcomes based outcomes based upon industry upon industry standardsstandards

Buy or Develop?Buy or Develop?

1.1. Standardized instrumentsStandardized instruments

2.2. Centralized state-developed Centralized state-developed

instrumentsinstruments

3.3. Local assessments recognized by stateLocal assessments recognized by state

Performance or CognatePerformance or Cognate

PerformancePerformance– What the student What the student

should be able to should be able to dodo

– ChecklistChecklist– Limited number Limited number

of areas can be of areas can be assessed in assessed in specified timespecified time

CognateCognate– What the student What the student

should knowshould know

– Test-itemsTest-items– Many areas can Many areas can

be assessed in a be assessed in a specified timespecified time

Building an AssessmentBuilding an Assessment

AdoptAdopt – seek items or assessments that – seek items or assessments that

you can adopt as isyou can adopt as is

AdaptAdapt – items or assessments that can be – items or assessments that can be

modified to fit your standardsmodified to fit your standards

BuildBuild – items or assessments when others – items or assessments when others

cannot be foundcannot be found

Table of Specifications Table of Specifications (blueprint)(blueprint)

Determine competencies to be Determine competencies to be assessedassessed

Determine competency groups (3 – 10)Determine competency groups (3 – 10)

Determine percentage of each Determine percentage of each cognitive typology per groupcognitive typology per group

Development QuestionsDevelopment Questions

Is cost and time worth benefit of Is cost and time worth benefit of

test?test?

Who should develop the assessment?Who should develop the assessment?

Test security issues?Test security issues?

Multiple-choice elements:Multiple-choice elements:

StemStem

Correct responseCorrect response

Foils or distractorsFoils or distractors

Fixed-form Fixed-form vv Variable-form Variable-form

Fixed-formFixed-form

High reliabilityHigh reliability Needs high Needs high

securitysecurity

Limited number of Limited number of test items neededtest items needed

Variable-formVariable-form

Good reliabilityGood reliability Lesser security Lesser security

neededneeded Requires a large Requires a large

number of test number of test itemsitems

Assessment AdministrationAssessment Administration

Decide the Decide the mediamedia for administration for administration

Identify Identify whowho should conduct the should conduct the assessmentassessment

Determine Determine whenwhen the assessment the assessment activities should occuractivities should occur

Reporting issues:Reporting issues:

1.1. Grading of the assessmentGrading of the assessment

2.2. Assessment improvement Assessment improvement needsneeds

3.3. Reports neededReports needed

Assessment ReviewAssessment Review

ValidityValidity – the degree to – the degree to which certain inferences can which certain inferences can be made from test scores.be made from test scores.

ReliabilityReliability – the degree of – the degree of consistency between two or consistency between two or more measures of the same more measures of the same thing.thing.

Item & Assessment AnalysisItem & Assessment Analysis

Find a good statistician, Find a good statistician, psychometrician or psychometrician or educational psychologist educational psychologist !!

Ethical Issues in TestingEthical Issues in Testing

Code of Fair Testing Code of Fair Testing Practices in EducationPractices in Education

by theby theJoint Committee on Testing Practices,Joint Committee on Testing Practices,

American Educational Research Association, American American Educational Research Association, American Psychological Association & Psychological Association &

National Council on Measurement in EducationNational Council on Measurement in Education

Legal IssuesLegal Issues

Accommodations for members of Accommodations for members of special populationsspecial populations

Access to the test-items by Access to the test-items by parents and othersparents and others

Utilization of assessments in Utilization of assessments in relation to passing or graduationrelation to passing or graduation

Development CycleDevelopment Cycle

Competency learning Competency learning outcome developmentoutcome development

Blueprint developmentBlueprint developmentTest-item developmentTest-item developmentField TestingField Testing Item-analysisItem-analysis

Development CycleDevelopment Cycle

Set cut scoresSet cut scoresTest form developmentTest form developmentAssessment administrationAssessment administrationReporting of resultsReporting of results

Contact Information:Contact Information:

Dr. John M. Townsend

Executive Director, Workforce Development

Office of Academic AffairsTennessee Board of Regents

1415 Murfreesboro Pike, Suite 350Nashville, TN 37217

615-366-4428

[email protected]