3/12/20031 RUSH TO ASSESSMENT: Designing Assessment to Produce Failure Roberta Madison, Gloria Melara and Robert Lingard.

3/12/2003 1

RUSH TO ASSESSMENT:

Designing Assessment to Produce Failure

Roberta Madison, Gloria Melaraand Robert Lingard

3/12/2003 2

Session Overview

An example of a recent assessment project at CSUN

A group activity designed to gain insight into effective approaches to assessment

Summary and review of group results

3/12/2003 3

Session Objectives

At the end of the session participants should be able to: Identify potential problems in

designing outcomes assessment Select/create appropriate assessment

measures and tools Identify the steps in designing a

productive assessment project

3/12/2003 4

Common Problems in Assessment

Assessment results are often needed quickly

Insufficient resources (people, tools, etc.) are usually applied

Assessment is sometimes viewed as an end in itself (e.g., an accreditation team must be convinced that assessment is being done)

3/12/2003 5

An Assessment Example

Examine an approach taken toward assessing Computer Literacy at CSUN

Look at the difficulties and problems encountered

Suggest some recommendations for improving the assessment process

3/12/2003 6

Why Was Computer Literacy Chosen for

Assessment?

Ensuring the Computer Literacy of students was viewed as an important University goal

The effectiveness of CSUN’s general education course in Computer Literacy was unknown

3/12/2003 7

The Approach Taken To get results quickly, an existing

instrument, TekXam, was selected to measure student’s computer skills

Students taking the General Education course in Computer Literacy and a control group not taking this course were given pre tests and post tests

The results were analyzed

3/12/2003 8

Study Results

Module

Subject

Number of students with pre- and post-tests

p value

1 General Concepts

13 p<0.01

2 Web Page Creation

3 p>0.05

3 Presentation Creation

4 p<0.001

4 Spreadsheets 16 p<0.001

5 Word Processing

10 Data unavailable

3/12/2003 9

TekXam Passing Rates

26.09%

61.90%

25.93%

10.00%

84.00%

4.17%

0%

20%

40%

60%

80%

100%

GeneralComputerConcepts

Presentations Word Processing Web Authoring Spread Sheets InternetConcepts

3/12/2003 10

Completion Rates

Pre Test

Post Test

Percent completing both tests

Students taking CL Course

108 46 43%

Students not taking CL Course

31 5 16%

Total 139 51 37%

3/12/2003 11

Problems Encountered

The TekXam instrument did not test what our Computer Literacy course covered

TekXam was difficult for students to use

Administration of TekXam was tedious and rigid

Student participation rate was low

3/12/2003 12

Conclusions and Recommendations

A clear definition of Computer Literacy is needed before it can be assessed

Clear (measurable) objectives need to be established for the assessment

Incentives are needed to increase student participation

Better assessment tools are needed

3/12/2003 13

A Sample Problem

Assume you are given the task of assessing the effectiveness of online instruction for a particular program

What needs to be done?

3/12/2003 14

First Group Task(Risk Analysis)

Identify potential problems or obstacles that might be encountered in trying to do this assessment

List them in order of importance (high probability & high impact)

3/12/2003 15

Problems/Obstacles Identified

3/12/2003 16

Second Group Task

For one of the problems identified suggest possible solutions

List the advantages and disadvantages of each solution

3/12/2003 17

How Can We Ensure a Successful

Assessment? Make sure the goal of the

assessment is clearly understood Establish measurable objectives

toward achieving the goal Determine how to measure each

objective (in multiple ways)

3/12/2003 18

GOAL

OBJECTIVE

OBJECTIVE

OBJECTIVE

MEASUREMEASURE

MEASURE

MEASUREMEASURE

MEASURE

MEASURE

MEASURE

MEASURE

3/12/2003 19

GOAL

OBJECTIVE

DATA DATA DATADATADATA DATA DATADATADATA

ANALYZE

CHANGES?

MEASUREMEASUREMEASURE MEASUREMEASURE

MEASUREMEASURE

MEASUREMEASURE

Obstacles

3/12/2003 20

Final Group Task

Suppose your task was to assess the effectiveness of online instruction in teaching written composition

Specify one measurable objective and identify two or more ways to measure it.

3/12/2003 21

Measurable Objective Example

Students will be able to write a grammatically correct English sentence with a subject and a verb.

3/12/2003 22

Conclusions A successful assessment project must be

carefully planned with sufficient resources allocated (we must avoid rushing to assessment)

The goal or goals of the assessment must be clearly understood

Specific measurable objectives must be established

Appropriate tools must be found/created to measure each objective in multiple ways

3/12/2003 23

References A short class feedback/assessment. It assesses students’

perception of the classroom environment. http://www.cod.edu/outcomes/SoaClsDs/StopStartCont_Assessment.PDFhttp://www.cod.edu/outcomes/SoaClsDs/feedback.PDF

This assessment activity clarifies expectations and mutual responsibilities for student learning and success.http://home.att.net/~p.klassen/SuccessAttribution.htm

An extensive list of many college assessment sites from North Carolina State University http://www2.acs.ncsu.edu/UPA/assmt/resource.htm

Nine Principles of Good Practice for Assessing Student Learning from AAHE, Alexander W. Astin, et.al http://www.aahe.org/assessment/principl.htm

A good resource with information from classroom to institutional assessmenthttp://www.cod.edu/Dept/Outcomes/AssessmentBook.pdf http://www.cod.edu/Outcomes/

3/12/2003 24

Contact Information

Roberta Madison - [email protected]

Gloria Melara - [email protected]

Robert Lingard - [email protected]