Program Level Assessment

Program Level Assessment

Workshop forCollege of the Redwoods

Fred TrappAugust 18, 2008

Assessment of Program Effectiveness• What the program will do or achieve

• Curriculum• Retention• Graduation• Placement • Satisfaction (graduate and employer)

Assessment of Student Learning Outcomes• What students will do or achieve• Direct methods of assessment

Direct Measures of Learning• Capstone experience• Standardized tests• Performance on national licensure certification or

professional exams • Locally developed tests • Essay questions blind scored by faculty• Juried review of projects• Externally reviewed exhibitions performances• Evaluation of internships based upon program

learning outcomes

Indirect Measures of Learning

• Alumni, employer, and student surveys (including satisfaction surveys)

• Exit interviews of graduates and focus groups graduate follow up studies

• Retention and transfer studies• Length of time to degree• Graduation and transfer rates• Job placement rates

Steps of Implementation• Step One: Developing Assessment

Plans• Step Two: Developing Implementation

Strategies• Step Three: Collecting Data, Analyzing

& Reflecting on its meaning• Step Four: Implementing Change as

the result of assessment

Assessment Language• Program Outcomes/ Topic Areas• Student Learning Outcomes• Means of Assessment/ Evidence• Criteria for Assessment• Standards of Success/ Quality of

Performance

Communication (program outcome topic)

SpeakRelate Listen ParticipateWrite

Program Outcome Topic & Student Learning Outcomes

Student learning outcomes

Means of Assessment/Evidence- Circumstances (learning events)

• Assignments (in class and out of class)• Feedback on practice• Self evaluation• Peer evaluation• Role Play• Pre Tests• Simulation

Goal

OutcomeOutcome OutcomeOutcomeOutcome

Object (deliverable or performance)

Learning events

Means of Assessment/Evidence- Deliverable or Performance

Communication

SpeakRelate Listen ParticipateWrite

Speech

Means of Assessment/Evidence- Deliverable or Performance

InterviewEssay

Communication

SpeakRelate Listen ParticipateWrite

Speech

Criteria for Assessment

Verbal deliveryNonverbal delivery

OrganizationContent

criteria

Communication

SpeakRelate Listen ParticipateWrite

Speech

Standards of Success/ Quality of Performance

Verbal deliveryNonverbal delivery

organizationcontent

Quality of Performance

Student Learning Outcomes

• Students should be able to<<action verb>> <<something>>

• Learner Centered• Specific• Action oriented• Cognitively appropriate for the

program level

KNOWLEDGECOMPREHENSION

APPLICATIONANALYSISSYNTHESISEVALUATION

CiteCountDefineDraw

IdentifyList

NamePointQuoteRead

ReciteRecordRepeatSelectState

TabulateTell

TraceUnderline

AssociateClassifyCompareComputeContrast

DifferentiateDiscuss

DistinguishEstimateExplainExpress

ExtrapolateInterpolate

LocatePredictReportRestateReview

TellTranslate

ApplyCalculateClassify

DemonstrateDetermineDramatize

EmployExamineIllustrateInterpretLocate

OperateOrder

PracticeReport

RestructureScheduleSketchSolve

TranslateUse

Write

AnalyzeAppraiseCalculate

CategorizeClassifyCompareDebate

DiagramDifferentiateDistinguish

ExamineExperiment

InspectInventoryQuestionSeparate

SummarizeTest

ArrangeAssemble

CollectComposeConstruct

CreateDesign

FormulateIntegrateManageOrganize

PlanPrepare

PrescribeProduceProposeSpecify

SynthesizeWrite

AppraiseAssessChoose

CompareCriticize

DetermineEstimateEvaluate

GradeJudge

MeasureRankRate

RecommendReviseScoreSelect

StandardizeTest

Validate

KNOWLEDGECOMPREHENSION

APPLICATIONANALYSISSYNTHESISEVALUATION

CiteCountDefineDraw

IdentifyList

NamePointQuoteRead

ReciteRecordRepeatSelectState

TabulateTell

TraceUnderline

AssociateClassifyCompareComputeContrast

DifferentiateDiscuss

DistinguishEstimateExplainExpress

ExtrapolateInterpolate

LocatePredictReportRestateReview

TellTranslate

ApplyCalculateClassify

DemonstrateDetermineDramatize

EmployExamineIllustrateInterpretLocate

OperateOrder

PracticeReport

RestructureScheduleSketchSolve

TranslateUse

Write

AnalyzeAppraiseCalculate

CategorizeClassifyCompareDebate

DiagramDifferentiateDistinguish

ExamineExperiment

InspectInventoryQuestionSeparate

SummarizeTest

ArrangeAssemble

CollectComposeConstruct

CreateDesign

FormulateIntegrateManageOrganize

PlanPrepare

PrescribeProduceProposeSpecify

SynthesizeWrite

AppraiseAssessChoose

CompareCriticize

DetermineEstimateEvaluate

GradeJudge

MeasureRankRate

RecommendReviseScoreSelect

StandardizeTest

Validate

Course outcomes

KNOWLEDGECOMPREHENSION

APPLICATIONANALYSISSYNTHESISEVALUATION

CiteCountDefineDraw

IdentifyList

NamePointQuoteRead

ReciteRecordRepeatSelectState

TabulateTell

TraceUnderline

AssociateClassifyCompareComputeContrast

DifferentiateDiscuss

DistinguishEstimateExplainExpress

ExtrapolateInterpolate

LocatePredictReportRestateReview

TellTranslate

ApplyCalculateClassify

DemonstrateDetermineDramatize

EmployExamineIllustrateInterpretLocate

OperateOrder

PracticeReport

RestructureScheduleSketchSolve

TranslateUse

Write

AnalyzeAppraiseCalculate

CategorizeClassifyCompareDebate

DiagramDifferentiateDistinguish

ExamineExperiment

InspectInventoryQuestionSeparate

SummarizeTest

ArrangeAssemble

CollectComposeConstruct

CreateDesign

FormulateIntegrateManageOrganize

PlanPrepare

PrescribeProduceProposeSpecify

SynthesizeWrite

AppraiseAssessChoose

CompareCriticize

DetermineEstimateEvaluate

GradeJudge

MeasureRankRate

RecommendReviseScoreSelect

StandardizeTest

Validate

Program outcomes

Possible Outcomes?

• Students will appreciate….• Students will be exposed to….• Students will demonstrate knowledge of

the history, theories and applications of…

• Students will attend….• Faculty will provide students with

opportunities to….

Possible Outcomes

• Students will appreciate…• Students will be exposed to….• Students will demonstrate knowledge of

the history, theories and applications of…• Students will attend….• Faculty will provide students with

opportunities to….

Writing Program Level Student Learning Outcomes• Student Learning Outcomes

• Students will recognize, analyze, and interpret human experience in terms of personal, intellectual, and social contexts.

• Students should be able to critically comprehend, interpret, and evaluate written, visual, and aural material.

Compound outcome statements will be problematic to assess.

Program Level Student Learning Outcomes• Students will be able to:

• Design a solution to a technical problem• Interpret performance data• Demonstrate the functions of key operating

systems• Articulate the ethical standards of the

profession• Produce diagnostic quality radiographic images• Evaluate the advantages of alterative solutions

Course LevelStudent Learning Outcomes• Students will be able to:

• Students will have a basic knowledge of human anatomy

• Students will be able to define 200 medical terms

• Students should know the history of the United States

• Students will be able to translate dollars into pounds and Euros

Course vs. Program Assessment• Single course• Course concepts

• Fewer confounds from outside the course detractors

• Faculty (indiv. & small groups)

• “Micrometer” precision• Less fear generated in faculty• Faculty professional

obligation, less overhead cost• No external certifications

• Multiple courses• Program performance

competencies (post grad)• More confounds from outside

the course detractors• Faculty (small & large groups)

• “Two sticks & a chain” precision

• Potential fear generated in faculty

• Faculty professional obligation, but some overhead cost

• External certifications possible

Identify Criteria for Assessing Program Level Outcomes• Must be shared across faculty• A way to further define the student

learning outcome• Provides a common language for

describing student learning• Must be outcome specific• Number of criteria will vary by outcome

Means of Assessment/Evidence- Deliverable or Performance + Criteria

Communication

SpeakRelate Listen ParticipateWrite

Salespresentation

deliverycontent

organizationOral

Report

Eulogy

volumepoise

conclusion

Speaking

Course Course Course Course Course

eye contactstyle

appearance

gesturesrate

evidence

sourcesexamples

organization

transitionsverbal variety

attention getter

The Reality of Assessing Program LevelStudent Learning Outcomes

volumepoise

conclusion

eye contactstyle

appearance

gesturesrate

evidence

sourcesexamples

organization

transitionsverbal variety

attention getter

Can our students deliver an effective public speech?

The Reality of Assessing Program LevelStudent Learning Outcomes

Program Learning Outcome Example #1

Gather factual information and apply it to a given problem in a manner that is relevant, clear, comprehensive, and conscious of possible bias in the information selected

BETTER: Students will be able to apply factual information to a problem

Criteria: RelevanceClarity

Comprehensiveness Aware of Bias

Program Learning Outcome Example #2

Imagine and seek out a variety of possible goals, assumptions, interpretations, or perspectives which can give alternative meanings or solutions to given situations or problems

BETTER: Students will be able to provide alternative solutions to situations or problems

Criteria: Variety of assumptions, perspectives, interpretationsAnalysis of comparative advantage

Program Learning Outcome Example #3

Formulate and test hypotheses by performing laboratory, simulation, or field experiments in at least two of the natural science disciplines (one of these experimental components should develop, in greater depth, students’ laboratory experience in the collection of data, its statistical and graphical analysis, and an appreciation of its sources of error and uncertainty)

BETTER: Students will be able to test hypotheses.Criteria:Data collectionStatistical AnalysisGraphical AnalysisIdentification of sources of error

Standards for Success/Quality of Performance

• Rubric• Scale or description for assessing each of

the criteria• Two to Five-point scales for each criteria• Anchored with descriptions and supported

by examples

• Portion of the Whole Group Which is Successful

ExceedsExpectations

MeetsExpectations

Does not meetExpectationsCriteria

Performance Anchors:

Descriptionsof

Performance

OUTCOME:

Standards for Success/Quality of Performance Rubric

YES!!!!AlmostNot YetComponents

Performance Anchors:OUTCOME:

Standards for Success/Quality of Performance Rubric

ExemplaryEssentialBasicComponents

Performance Anchors:

Standards for Success/Quality of Performance RubricOUTCOME:

Student LearningOutcome

3

2

1

4

Criteria Test Questions

1 2 3 4

5 6 7 8

10 11 12

1713 14 15 16

9

18

Student LearningOutcome

3

2

1

4

Criteria Test Questions

1 2 3 4

5 6 7 8

10 11 12

1713 14 15 16

9

18

78%

56%

82%

42%

%correct

Studentlearning outcomes Course

1

X

X

X

Course2

X

X

X

X

X

Course3

X

X

X

X

Course4

X

X

X

X

X

Course5

X

X

X

X

7

1

234

56

Identify Assessment Points in the Program

Studentlearning outcomes Course

1

X

X

X

Course2

X

X

X

X

X

Course3

X

X

X

X

Course4

X

X

X

X

X

Course5

X

X

X

X

7

1

234

56

Phase 1

Communicate EffectivelyDemonstrate Oral Communication Skills

VerbalDelivery

NonverbalDelivery

Organization

Does notmeet Meets Exceeds

20%

13%

24%

65%

57%

58%

15%

30%

18%

Program-wide Assessment Report

Is there a problem with performance in any of the criteria?