What Works - Course and Program Assessment in STEM PKAL Network – 7 October 2013 Dr. Dianne Raubenheimer, Director of Research, Planning and Assessment,

What Works - Course and Program Assessment in STEM

PKAL Network – 7 October 2013Dr. Dianne Raubenheimer,

Director of Research, Planning and Assessment, Meredith College

[email protected]

By the end of this session, participants will be able to: Identify some purposes of assessmentExplain selected assessment terminologyArticulate ways to align outcomes,

teaching, learning and assessment Consider strategies for linking course and

program assessment

Ice Breaker: Write down some thoughts

A show of hands:Gathering information Measuring goals and outcomesUsing diverse measures and sourcesAnalyzing and interpreting informationReporting resultsEngaging stakeholdersMaking improvements

Assessment is the systematic collection, review, and use

of information about educational programs undertaken for the purpose

of improving student learning and development

(Palomba & Banta,1999). 

Task: Individually - Write down different types of

assessment tasks you use in your courses. Use one small post-it for each type of assessment.

As a group - Sort the post-its into two columns – direct and indirect assessment methods Direct assessments provide for direct examination or

observation of student knowledge of skills against measureable learning outcomes.

Indirect assessments of student learning ascertain the perceived extent of value of learning experiences. They assess opinions or thoughts about student knowledge or skills.

Task: Looking at the different assessment

strategies. In your groups, discuss some thoughts about the purposes of assessment

You may be able to cluster some of the methods with particular purposes

Assessment of student learning (knowledge, skills and attitudes)

Improvement of student learning Improvement of teaching Program development, monitoring and

improvement SoTL, SoA, SoP

Accreditation Public accountability / Performance

scorecard

Internal / Intrinsic

External / Extrinsic

Outcomes

Reciprocal relationship

An example from NC State : Using an observation rubric, observed 15

instructors in different disciplines teaching All indicated they were using innovative

practices with technology Instructor noted the learning outcomes for the

lesson being taught Observer used the SOLO taxonomy to score the

SOLO level of instruction Observer used the SOLO taxonomy to score the

SOLO level at students were engaged in the class

Then gathered student work from the faculty that tested the outcome

Examined the assessment task and student work using the SOLO taxonomy (Raubenheimer, et al, 2007)

SOLO category Representation Type of outcome (Biggs, 2004)

Unanticipated Extension(Extended Abstract)

  Create SynthesiseHypothesise ValidatePredict DebateTheorise

Logically Related (Relational)

Apply OutlineDistinguish AnalyseClassify ContrastSummarise Categorise

Multiple Points(Multistructural)

  Explain DefineList SolveDescribe Interpret

Single Point (Unistructural)

  State RecogniseRecall QuoteNote Name

Instructor SOLO

2

5

10

1

2

3

8

12

6

2

6

21

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Single point (e.g. tell,state, recall)

Multiple points (e.g.rework, clarify, explain)

Logically related (e.g.inquire, apply, distinguish)

Unanticipated extension(e.g. imagine, elaborate)

Always

Often

Sometimes

Rarely

Never

Student Solo

5

7

9

123

4

3

1

4

3

1

2

1

22

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Single point (e.g. tell, state, recall) Multiple points (e.g. clarify,

explain, define, describe)

Logically related (e.g. inquire,

apply, analyze)

Unanticipated extension (e.g.

imagine, elaborate, synthesize)

Always

Often

Sometimes

Rarely

Never

Scored Using SOLO Taxonomy

Level of outcome taught

Level of related student assessment task

Single 3 7

Multiple point 2 3

Logically related 8 2

Unanticipated extension

0 1

In your groups: Discuss ways in which you align teaching

and assessment of student learning outcomes?

Select 3 individuals to each act as chair, scribe and reporter for the group

http://www.arapahoe.edu/about-acc/presidents-office/assessment

On your own:Use the “Program Level-Assessment of Student

Learning: Self-assessment worksheet” to assess your program at your institution

In your groups: Discuss ways in which you align teaching

and assessment of student learning outcomes in your courses (columns 1-3)?

Select 3 individuals to each act as chair, scribe and reporter for the group

Ensure the program outcomes reflect what you want students to have learned/attained/ achieved by the time they graduateGenerating appropriate program outcomesDevelop program outcomes - course

assessment matricesSelecting appropriate assessment

tasks/methodsDevelop a system for keeping track of

program dataGather, review data and use data for ongoing,

systematic program review and improvement

Develop an implementation matrix associated with program outcomesH= high, M=medium; Low = L

Course1

Course2

Course3

Course4

Course5

Etc

Outcome 1

H M H

Outcome 2

M H M L

Outcome 3

H L M

Outcome 4

H L L H

Etc

Develop an implementation and assessment matrix associated with program outcomes I = implement; A = assessCourse

1Course2

Course3

Course4

Course5

Etc

Outcome 1

I & A I I & A

Outcome 2

I I & A I I

Outcome 3

I & A I I & A

Outcome 4

I & A I I I & A

Etc

Expected Outcomes

BIO183

BME 201

BME 203

BME 204

BME210

BME 252

BME 301

BME 302

Co-curricula/ other sources

Outcome (a) Graduating senior survey

Sub-Outcome a1 A A A

Sub-Outcome a2 A ASub-Outcome a3 A A ASub-Outcome a4 A A FE examOutcome (b) Graduating

senior surveySub-Outcome b1 x UG Research

Sub-Outcome b2 A A A UG Research

Sub-Outcome b3 A A

Sub-Outcome b4 A A

Outcome (e) Graduating senior survey

Sub-Outcome e1 A A A A Design day

Sub-Outcome e2 A A

Sub-Outcome e3 A

Sub-Outcome e4 ASub-Outcome e5 A

Articulate sub-outcomes on the matrix

GraduatingSenior Survey

FacultySurvey

WPS 332Test & homework

scores

WPS 360Math-

intensive test questions

WPS 475Math skills

test

Goal Actual Goal Actual Goal Actual Goal Actual Goal ActualE:

P: 30%+C:

NI:<10%

E: 16%P: 78%C: 7%NI: 0%

E: P: 30%+

C: NI:<10

%

E: 11%P: 67%C: 9%NI: 3%

Avg. scoresB or

higher

Avg. Score

B+

Avg. scoresB or

higher

Avg. Score

B+

Avg. scoresB or

higher

Avg. Score

B+

Determine assessment methods and expected performance criteria(level of performance )

Develop a system for recording and tracking results

Best practices indicate that assessment

is systematic and systemic. It occurs at multiple levels in an

institution, vertically and horizontally,

involving processes that are connected and integrated

Assessment: Arapahoe Community College http://www.arapahoe.edu/about-acc/presidents-office/assessment

Biggs, J. (2004). Teaching for quality learning at university. Berkshire, UK: Open University Press. Heer, R. (2009). A Model of Learning Objectives. Center for Excellence in Learning and Teaching,

Iowa State University. Maki, P.L. (2004). Assessing for learning: Building a sustainable commitment across the institution.

Sterling, VA: Stylus. Mertler, C.A. (2003). Classroom assessment: A practical guide for educators. Los Angeles: Pyrczak. Ozturk, H. & Raubenheimer, C.D. (2011). PAT: An online program A\assessment tool. Paper

presented at 2011 American Society for Engineering Education International Conference, Vancouver, Canada.

Palomba, C.A & Banta, T.W. (1999). Assessment essentials: Planning, implementing and improving assessment in higher education. San Francisco, CA: Jossey-Bass.

Raubenheimer, C. D., Spurlin, J., Martin, S., & Mehlenbacher, B. Faculty in Technology-Rich Contexts: Connecting Teaching, Learning, and Assessment in the Classroom. 2007 UNC TLT Conference Proceedings.

Rogers, G. (2004). Self-Assessment: Quality Assurance of Program Level-Assessment of Student Learning. Rose Hulman Institute of Technology.

Spurlin, J.E., Rajala, S.A., & Lavelle, J.P. (2008). Designing better engineering education through assessment: A practical resource for faculty and department chairs on using assessment and ABET criteria to improve student learning. Sterling, VA, Stylus Publishing.

[email protected]