Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951.

Sizing up Your Students, Part 3:

Designing Effective Evaluations

Marti Atkinson and Bill DohertyApril 10, 2008

For audio call Toll Free 1-888-886-3951

and use PIN/code 240228

Parts of this presentation were adapted from materials created by:Noelle C. Griffin PhD, Assistant Director, UCLA Center for the Study of Evaluation

Original material can be found at: http://www.lmu.edu/Page18044.aspx

Housekeeping

• Maximize your CCC Confer window.• Mute your phone (*6) if you have visitors or noise in your office.• Please note phone audio may be in presenter-only mode.• Ask questions over the phone when the presenter prompts. • Ask questions throughout presentation via the chat window.• Turn on or off Closed Captioning by clicking on the icon.• Save the presentation or chat by clicking on the icon.

AgendaBackground and Terms

Planning your Evaluation Strategy - Defining your Goals, Objectives & Outcomes

- Choosing your Measurements

- Online Quizzes and Exams - Performance based assessments

Analyzing data – rubrics

Background and TermsEnormous body of literature on this subject

Download list of resources at

http://nlight.com/ONE/evalsSurveys_resources.doc

Background and Terms

Some Basics

Assessments versus evaluations

Reliability - repeated measurements are same or very similar

Validity – you are measuring what you say you are

Background and Terms

Kinds of Measures

Quantitative: generates numeric data

Qualitative: generates descriptive data

Background and Terms

Collection Methods

Survey, Quizzes, and Exams

Interviews

Observations

Work Products

Background and Terms

Authentic – Performance & Behavioral

Assignments

Projects – individual and group

Behavioral observations– General Participation– Discussion contributions

Portfolio

Background and Terms

Triangulation of Data

Planning Your Evaluation Strategy

Define your course goals, learning objectives and outcomes

Determine what kind data and measurement tools will best indicate how well your Learning Objectives & Outcomes are being met?

Planning your Evaluation Strategy Goals, Objectives, Outcomes

Goals - Represent the overall mission

Objectives - Broader perspective on what generally students should know, value or do

Outcomes - What students should demonstrate/represent relative to objective

Why bother with Goals, Objectives, and Outcomes?

They provide the basis for: – Course content– Evaluating students– Communications

• Students• Other faculty and administration• Outside world

– Curriculum planning

ExampleGoal: The course will prepare students to become a

rocket scientist

Objective: Students will be able to understand the working components of a rocket and methods for a successful launch

Outcome: Students will be able to build and launch a rocket.

Learning Objectives and Outcomes Bloom’s Taxonomy of Learning

Knowledge of terminology (define, identify, describe)

Comprehension (discuss, explain, summarize)

Application (apply, utilize, interpret)

Analysis (analyze, compare, contrast)

Synthesis (integrate, organize, construct)

Evaluation (appraise, defend, judge)

Measurement Tool – Surveys, Quizzes & Exams

Types of Items

Quantitative (numbers) often closed ended (limited responses)

• True/False• Multiple choice• Ratings• Single answer

Qualitative (narrative & content) often open ended items

• Short answer

• Essay

Measurement Tool - Quizzes & Exams

Examples of Closed Ended Items

What is your gender?• Male – Female

What is your age ? (10-25) (26-40) (41-55) (55+)

Which college do you attend? EVC Butte Foothill De Anza

Who is your favorite artist of the 20th Century?• Picasso• Joan Miro• Marc Chagall• Salvador Dali

Measurement Tool - Quizzes & Exams

Examples of Open Ended Items

Describe the stylistic elements of Art Deco.

Why did Van Gogh cut off his ear?

What is art?

What is your life story until this point?

Measurement Tool - Quizzes & Exams

Watch for Bias in Constructing Items

How long ago did you stop using street drugs?• Just this week

• A month ago

• A year ago

• Over a year ago

Measurement Tool Performance Assessments

Portfolios

Capstone assignments

Performances/projects

Internships/field activities

Students evaluating students

Measurement Tool – Performance Assessments Potential Strengths

Direct evidence of student learning

Students motivated

Realistic context

Can be integrated into the grading process

Data collection can be relatively unobtrusive to students

Measurement Tool – Performance Assessments Potential Weaknesses

Additional development/coordination time

Faculty trust/buy in critical

Faculty time for scoring

Reliability/validity uncertain

Lack of norms/comparative data

Measurement Tool – Performance Assessments Portfolios

What is it? A collection of multiple targeted pieces of student work that provide evidence of a learner’s achievements along a continuum of learning.

Can be formative or summative (i.e., developmental vs. showcase)

Can be “online” or “old school”

Measurement Tool – Performance Assessments Portfolio Pitfalls

“Kitchen sink” approach

Lack of pre-planning

Retrospective re-creation

Request vs. requirement (i.e., built into program)

Under-estimating faculty time involved

Measurement Tool – Performance Assessments Capstone Assignment/Project

What is it? A culminating project or assignment that provides evidence of achievement of learning goals; generally integrative in nature.

Does not only have to be last year; can be spaced in terms of developmental milestones in the program providing longitudinal/comparative data

Measurement Tool – Performance Assessments Capstone Considerations

Does assignment capture the breadth of program learning expectations?

Does assignment link back to what is actually being taught in the courses?

Measurement Tool – Performance Assessments Data Collection Process

How will assignments be collected/saved for analysis?

Integrate into regular classroom process, otherwise compliance difficult

Will collection be online or hard copies?

Where will data be stored?

How will anonymity be maintained?

Measurement Tool – Performance Assessments Scoring Criteria

Rubrics: A set of criteria that identify expected dimensions of achievement regarding specific assignments, as well as levels of achievement along those dimensions

Checklists

Other approaches

Measurement Tool – Performance Assessments Strategies for Developing Rubrics

Professional org/ discipline standards

Review representative student work (failing though exemplary) and analyze what criteria led to those grades

Student feedback

Analysis of grading criteria already used by faculty to grade similar assignments

Ask faculty to grade sample work and provide criteria they used, looking for commonalities

Pilot/practice use

Measurement Tool – Performance Assessments Students

Students evaluating themselves

Students evaluating other students

– Need to train students in critiquing guidelines http://nlight.com/ONE/critiquingGuidelines.doc

– Use of rating scales useful

Evaluating Team Projects

References“Assessing for Learning” (2004). Peggy L. Maki, Stylus

Press

“Assessment Clear & Simple: A Practical Guide for Institutions, Departments, and General Education” (2004). Barbara E. Walvoord, Jossey-Bass.

“A Practical Guide to Alternative Assessment” (1992). Joan L. Herman, Pamela Aschbacher & Lynn Winters, ASCD.

“Scoring Rubrics in the Classroom” (2001). Judith Arter & Jay McTighe, Corwin Press.

Q & A

Ask now, or via email if you prefer:

Marti Atkinson, [email protected]

Bill Doherty, [email protected]

Evaluation Survey Link

http://tinyurl.com/24n7lq

Upcoming/Archived Seminars

For upcoming desktop seminars and links to recently archived seminars, check the @ONE Web site at:

http://www.cccone.org/seminars/index.php

Thanks for Attending

For information on @ONE’s trainings go to:

http://www.cccone.org/index.php