Assessment = Learning What evidence will you and your peers accept? Diane Ebert-May, Ph.D. Lyman Briggs School Botany and Plant Pathology Michigan State.

Assessment = LearningAssessment = LearningWhat evidence will you and your peers

accept?

Diane Ebert-May, Ph.D.Lyman Briggs School

Botany and Plant Pathology

Michigan State University

Innovations

Change

InquiryActive Learning

FacultyStudents Institutions

Outcomes Assessment

Content/Process

Self-EfficacyCooperative

Learning

toward toward

necessitates

by byby

collaboratewith

supportedby

achieve

developuse

in-depthvalue

in-depth

e.g.e.g. e.g. e.g.

drives

necessitates

Reasoning

Cooperative GroupsCooperative Groups

4 students per group Person A, B, C, D in each group First - read question/think about answer

individually Discuss: A with B C with D Form group consensus

Q1 - EngagementQ1 - Engagement

Imagine I invited you to visit and review my course Introductory Biology to help me decide if I was practicing “active learning.” What criteria and dimensions of teaching would you look for in my class?

Oral report from groups

Biological Literacy for AllBiological Literacy for All Utilize process of scientific inquiry to think

creatively and formulate questions about real-world problems

Effectively communicate an understanding of and links among biological principles and concepts to peers and others

Gain confidence in ability to write about, criticize and analyze concepts in biology

Biological Literacy for All (2)Biological Literacy for All (2) Reason logically and critically to evaluate

information Develop positive attitudes about the

relevance of biology to their lives Demonstrate positive interdependence and

individual accountability within cooperative groups

NSF- Undergraduate Course and NSF- Undergraduate Course and Curriculum Development GrantCurriculum Development Grant

Is it possible to implement strategies for active, inquiry-based learning and cooperative group interaction in large courses?

Do these strategies promote more effective learning by more students?– 600 -700 students in class meetings– 450 students in laboratory

Compared Two Approaches to Teaching

Compared Two Approaches to Teaching

Traditional Experimental

Who Asks QuestionsWho Asks Questions

Traditional: Lab manual/faculty Experimental: Students

Inquiry Lab/Class MeetingInquiry Lab/Class Meeting Traditional: Confirmatory Experimental: Inquiry

Functional Cooperative GroupsFunctional Cooperative Groups

Traditional: Not Often Experimental: Often

Writing/Reading to Learn Writing/Reading to Learn

Traditional: Not Often Experimental: Often

TA Professional DevelopmentTA Professional Development

Traditional: Superficial Experimental: In-depth, on-going

AssessmentAssessment Traditional: Tests Experimental: Multiple Forms

– Aligned with student outcomes

Cooperative Groups: 4 Students

Engage 5 - 10 minutes

Explore 10 - 15 minutes

Explain 10 - 20 minutes

Elaborate 10 - 15 minutes

Evaluate 10 minutes

Learning Cycle Model in Class Meetings

Learning Cycle Model in Class Meetings

Experimental DesignExperimental DesignCovariates

Pre-Test / Factor Scores

Control Lectures Experimental / Control Lectures Experimental Lecture / LabControl / Experimental Labs Experimental Labs

•Fall 1994 •Spring 1995 •Fall 1995

• Self-Efficacy • Self-Efficacy •Self-Efficacy

• NABT • NABT •NABT

• Process Skills I, II • Process Skills I, II • Process Skills I, II

• TOSRA •TOSRA •TOSRA

Independent VariablesIntervention --- TA --- Gender --- Ethnicity --- Age

(Lab, Lecture)

Q2 - ExplorationQ2 - Exploration

What evidence will you (and your peers) accept that indicates your students have learned and achieved the outcomes of your course?

Written group report

Self-Efficacy SubscalesSelf-Efficacy Subscales

Fac 1: Writing and critiquing; analytical skills (.92)

Fac 2: Generalizability to other science courses (.87)

Fac 3: Biological literacy (.88)

* p < .01 (n =338)

Self-Efficacy, Fall 94

Experimental Condition

EXPCON

4.4

4.2

4.0

3.8

3.6

3.4

Factor Scores*

PosFac 1

PosFac 2

PosFac 3

NABT Process, Fall 1994

p < .01 (n =356)

Lecture

Experimental

Control

14.2

14.0

13.8

13.6

13.413.213.0

NABT ContentNABT Content No difference in scores between

traditional and control groups

Model with 700 studentsModel with 700 students

Two Sections @ 350 students each Taught back to back on schedule T, Th - 75 minute periods Team taught - 2 faculty as team

partners in both sections

Experimental Totaln = 266

029%

126%

>145%

Control Totaln = 287

018%

113%

>169%

Number of Absences

0

50

100

150

200

250

300

350

section 1 section 2

B 41%

C 30%

D 11%

F 8%

A 10%

B 39%

C 29%

D 12%

F 13%

A 8%

Nu

mbe

r of

Stu

dent

sGrade Distribution

Assessment and Curriculum Design

Assessment and Curriculum Design

Two sides of the Same Coin

Design Course GoalsDesign Course Goals Course - goals Modules - objectives Activities - objectives

Develop Performance Expectations

Develop Performance Expectations

Descriptions of Student Performance Providing evidence convincing to you,

your colleagues, and significant others that students have achieved a course goal or module objective

Descriptions of the conditions under which the evidence is collected

AssessmentAssessment Data collection with a purpose

What data are collected?What data are collected? Measures of students’ knowledge and

abilities– Answers to items on short answer tests– Essays– Position Papers– Oral Presentations– Poster Presentations

Assessments Linked to GoalsAssessments Linked to Goals

Goals articulated with outcomes that are measurable or observable (actions)

Functions of Assessments Functions of Assessments (Hodson 1992)

summative - description of students’ level of attainment

formative - diagnostic feedback to students and instructor

evaluative - provide instructors feedback about the effectiveness of the curriculum experiences

educative - engage students in interesting, challenging, significant experiences to develop further insight and understanding

Development of RubricDevelopment of Rubric Assessment tasks linked to outcomes Define performance standard for

biologically literate student.

Categories Categories Name of category Criteria within categories -- discriminate

among qualities you determine important

Reach consensus - students involved

Scoring Rubric for Essay Questions

BIO 226Ecology

Level of Achievement General Presentation Reasoning,Argumentation

Exemplary (10 pts) • Addresses the question• States a relevant argument• Presents arguments in alogical order• Uses acceptable style andgrammar (no errors)

• Demonstrates a clear andcomplete understanding ofthe question• Uses several argumentsand backs arguments withexamples, data that supportthe conclusion

Quality (8 pts) •Combination of abovetraits, but less consistentlyrepresented (1-2 errors)

•Same as above but lessthorough•Uses only one argumentand example that supportsconclusion

Adequate (6 pts) • Does not address thequestion explicitly, thoughdoes so tangentially• States a somewhatrelevant argument• Presents some argumentsin a logical order• Uses adequate style andgrammar (more than 2errors)

• Demonstrates minimalunderstanding of question,uses a small subset ofpossible ideas for supportof the argument

Needs improvement(4 pts)

• Does not address thequestion• States no relevantarguments• Is not clearly or logicallyorganized• Fails to use acceptablestyle and grammar

• Does not demonstrateunderstanding of thequestion• Does not provide evidenceto support response to thequestion

No Answer (0 pts)

Ebert-May, D. 1998.

Engagement problemEngagement problem E.O. Wilson states that insects and other land-

dwelling arthropods are so important that if all were to disappear, humanity probably could not last more than a few months. Based upon what you know today, do you believe that this statement is true? Write an essay and explain your reasoning as logically and thoroughly as you can. Do not use any outside resources for your essay.

What is the evidence we accept What is the evidence we accept that students have learned?that students have learned?

What do we accept as a reasoned argument? What are the criteria we use for evaluation of

the argument?

WARRANT (W)

EVIDENCE (E) CONCLUSION (C)Specific informationrelied on to supporta given conclusion (E1, E2,...,En).

Assertions put forthfor acceptance.

Statement justifying the move from Evidence to Conclusion

“Evidence indicates that... E1, E2,...En.”

“So... C”

“Because E1, E2,...En, then C.”

Argument StructureArgument Structure

Why use argument structure in Why use argument structure in science?science?

Consistent with nature of science Justifies conclusions to the learning

community Develops problem solving and critical

thinking skills Develops art of questioning

What about content?What about content? Argument and inquiry drive

understanding of content Content drives argument and inquiry

What did we find?What did we find? Students seldom disagreed with the

statement Some misconceptions about content

appeared Students rarely used warrants to

support evidence

What did we do?What did we do?

Revised QuestionRevised Question The keynote speaker at a scientific meeting made

the statement that “insects and other land-dwelling arthropods are so important that if all were to disappear, humanity probably could not last more that a few months.” This generated considerable discussion, both pro and con at the meeting. What do you think about this statement? Construct an argument to explain your views and to illustrate your reasoning.

Question: Question: When we introduce innovation into our

classes, how are student evaluations affected?

Self-EvaluationSelf-Evaluation Students who internalize valued

achievement targets so thoroughly as to be able to confidently and completely evaluate their own and each other’s work, almost automatically become better performers in their own right.

McMillan and Forsyth (1991)

Hig

hD

eg

ree

Mo

de

rate

De

gre

e

So

me

wh

at

Min

ima

lD

eg

ree

No

t A

t A

ll

0

50

100

150

200

250

Nu

mb

er

of

Stu

den

ts

Hig

hD

eg

ree

Mo

de

rate

De

gre

e

So

me

wh

at

Min

ima

lD

eg

ree

No

t A

t A

ll

Student Evaluation

Q2 Effectively describe biological concepts to peers

Goal 2 - Effectively describe Goal 2 - Effectively describe biological concepts to peersbiological concepts to peers

“So much group work made me realize that understanding a concept and being able to communicate a concept are different things.”

“...I find myself explaining things I was unable to explain before.”

Hig

h

Degre

e

Modera

te

Degre

e

Som

ew

hat

Min

imal

Degre

e

Not

At

All

0

50

100

150

200

250

Num

ber

of

Stu

dents

Hig

h

Degre

e

Modera

te

Degre

e

Som

ew

hat

Min

imal

Degre

e

Not

At

All

Student Evaluation

Q3 Confidence in ability to write, criticize, analyze biological concepts

Goal 3 - Confidence in ability to write about, Goal 3 - Confidence in ability to write about, criticize, analyze concepts in biologycriticize, analyze concepts in biology

“I always was good at biology (or at least got good grades in biology), but never really understood any CONNECTIONS. This class made it clear to me that everything is connected to each other. We were forced to write down how we UNDERSTOOD concepts, not simply to memorize parts and functions.”

Hig

hD

egre

e

Modera

te

Degre

e

Som

ew

hat

Min

imal

Degre

e

Not

At

All

0

50

100

150

200

250N

um

ber

of

Stu

dents

Hig

hD

egre

e

Modera

te

Degre

e

Som

ew

hat

Min

imal

Degre

e

Not

At

All

Student Evaluation

Q4 Ability to use process of scientific inquiry to think creatively and formulate questions

about real-world problems

4 - Use process of scientific inquiry to 4 - Use process of scientific inquiry to think creatively and formulate questions...think creatively and formulate questions...

“...we did our position papers on two real world issues that we deal with every day. One paper we did was on the environment and the other was on breast cancer...This class really makes you think about how important scientists are in the world today. They try to formulated and answer questions that will help us survive in the future...”

Hig

hD

egre

e

Modera

te

Degre

e

Som

ew

hat

Min

imal

Degre

e

Not

At

All

0

50

100

150

200

250N

um

ber

of

Stu

dents

Hig

hD

egre

e

Modera

te

Degre

e

Som

ew

hat

Min

imal

Degre

e

Not

At

All

Student Evaluation

Q7 Positive interdependence and individual accountability within groups; reason logically

and critically

7 - Positive interdependence; individual 7 - Positive interdependence; individual accountability within cooperative groupaccountability within cooperative group

“Working in groups requires so much responsibility. This means coming to class every day, having your input for all of the questions and quizzes and most of all showing up on your own time to complete homework assignments, etc.... Working as a team requires dedication and cooperation This is how everyone will succeed.”

0

20

40

60

80

100

Factor 1 Factor 2 Factor 3

Change in Self-EfficacyChange in Self-Efficacy

+ change

no change

- change

Per

cent

So what?So what? Pre-service teacher education Scientifically literate population Research in the disciplines about learning Recognizing, Evaluating, and Rewarding

Excellent Undergraduate Teaching - NRC Study Committee on Higher Education