HoelzerHoward_RubricBasesAssessment

Rubric-based Rubric-based Assessment of Critical Assessment of Critical

Thinking SkillsThinking SkillsGuy Hoelzer and Christie HowardGuy Hoelzer and Christie Howard

Department of BiologyDepartment of BiologyUNRUNR

or

More than “Just the Facts, More than “Just the Facts, Ma’am”:Ma’am”:

Assessment of Critical Thinking Assessment of Critical Thinking for Biology Majors and Programsfor Biology Majors and Programs

Guy Hoelzer and Christie HowardGuy Hoelzer and Christie HowardDepartment of BiologyDepartment of Biology

UNRUNR

Our Motivating FactorsOur Motivating Factors

1.1. Critical thinking (CT) has been identified by Critical thinking (CT) has been identified by the Biology faculty as one of the key things we the Biology faculty as one of the key things we want our majors to be able to do well upon want our majors to be able to do well upon graduationgraduation

2.2. CT is notoriously difficult to teach and assessCT is notoriously difficult to teach and assess• Not part of our normal course contentNot part of our normal course content

3.3. Emphasis on CT in the capstone course for Emphasis on CT in the capstone course for Biology majors leaves the impression that Biology majors leaves the impression that they are not very good at CT and don’t really they are not very good at CT and don’t really understand what we mean by itunderstand what we mean by it

4. Indeed, they don’t seem to 4. Indeed, they don’t seem to trust the instructor’s trust the instructor’s

explanation of CTexplanation of CT It seems too subjective, so the target is the It seems too subjective, so the target is the

mind of the instructor rather than anything of mind of the instructor rather than anything of value to the studentvalue to the student

It seems that every instructor means It seems that every instructor means something different by CT, so it doesn’t seem something different by CT, so it doesn’t seem like a real or particular kind of skilllike a real or particular kind of skill

Instructors too often pay lip-service to CT Instructors too often pay lip-service to CT without giving it weight in gradingwithout giving it weight in grading

5. Generic CT Skills vs. 5. Generic CT Skills vs. Discipline-Specific Skills?Discipline-Specific Skills?

We would like to generate majors who We would like to generate majors who “think like a biologist”“think like a biologist”

On the other hand, CT skills ought to be On the other hand, CT skills ought to be generally useful in any intellectual generally useful in any intellectual endeavorendeavor

6. Student Assessment Of CT 6. Student Assessment Of CT Skills Is Hard Enough - Can Skills Is Hard Enough - Can

We Do Programmatic We Do Programmatic Assessment Of CT?Assessment Of CT?

We would like to engineer a curriculum that We would like to engineer a curriculum that builds on student CT skills in an integrated builds on student CT skills in an integrated

and coherent fashionand coherent fashion

We Propose To …We Propose To …

emulate the rubric-based institutional model emulate the rubric-based institutional model developed at Washington State Universitydeveloped at Washington State University

[see http://wsuctproject.wsu.edu/][see http://wsuctproject.wsu.edu/]

The CT Rubric for Biology The CT Rubric for Biology StudentsStudents

Will include the key components of CT Will include the key components of CT identified by the Biology facultyidentified by the Biology faculty

Will serve as a template that can be modified Will serve as a template that can be modified to meet the needs of specific coursesto meet the needs of specific courses

Will be a primary tool for assessing student Will be a primary tool for assessing student work at two or more points within the major’s work at two or more points within the major’s curriculumcurriculum Track development of the CT skills of individual Track development of the CT skills of individual

studentsstudents Focus on an Intro course (Howard) and the Focus on an Intro course (Howard) and the

capstone course (Hoelzer)capstone course (Hoelzer)

Developing the RubricDeveloping the Rubric H&H will design a 1H&H will design a 1stst draft of the rubric based draft of the rubric based

on the relevant assignments in their respective on the relevant assignments in their respective coursescourses Students are asked to criticize scientific papers in Students are asked to criticize scientific papers in

each courseeach course H&H will do a pilot study by using the1H&H will do a pilot study by using the1stst draft of

the rubric to assess 20-30 sample student rubric to assess 20-30 sample student papers from each coursepapers from each course Does the rubric effectively discriminate student Does the rubric effectively discriminate student

performance over the full range encountered in both performance over the full range encountered in both courses?courses?

A Faculty RetreatA Faculty Retreat Introduce the rubric to the Biology faculty (from Introduce the rubric to the Biology faculty (from

both UNR and TMCC) and solicit feedback at a both UNR and TMCC) and solicit feedback at a faculty retreatfaculty retreat

TMCC will be included to facilitate articulation TMCC will be included to facilitate articulation for transfer studentsfor transfer students Ideally, the rubric will be useful and ultimately used Ideally, the rubric will be useful and ultimately used

in TMCC courses as well as UNR coursesin TMCC courses as well as UNR courses Outside evaluators (experts) will be present to Outside evaluators (experts) will be present to

contribute to the discussion and report to H&H contribute to the discussion and report to H&H on the efficacy of our processon the efficacy of our process Evaluators, like John Mahaffy, will be consulted Evaluators, like John Mahaffy, will be consulted

throughout the duration of the projectthroughout the duration of the project

Consistency in Rubric ScoringConsistency in Rubric Scoring H&H will design a second draft of the rubric H&H will design a second draft of the rubric

based on feedback from the retreatbased on feedback from the retreat Clarify and calibrate the rubric with an Clarify and calibrate the rubric with an Inter-Inter-

rater reliability (IRR) studyrater reliability (IRR) study Each member of the IRR team will score the Each member of the IRR team will score the

same set of 20-30 pieces of student worksame set of 20-30 pieces of student work Meet to discuss differences among raters and Meet to discuss differences among raters and

possible causes of those differencespossible causes of those differences H&H will again revise the rubric based on the H&H will again revise the rubric based on the

IRR results to maximize repeatability and IRR results to maximize repeatability and tune the scoring scale to student variancestune the scoring scale to student variances

Another Faculty RetreatAnother Faculty Retreat

Explain the revised rubric to the facultyExplain the revised rubric to the faculty Workshop in which all faculty will use the Workshop in which all faculty will use the

rubric to score student workrubric to score student work Faculty trainingFaculty training Another round of faculty input into rubric Another round of faculty input into rubric

developmentdevelopment Effectively another (informal) IRR studyEffectively another (informal) IRR study Fostering faculty “buy-in” on use of the rubricFostering faculty “buy-in” on use of the rubric

A Novel Student IRR StudyA Novel Student IRR Study

It is just as important that the meaning of the It is just as important that the meaning of the rubric be clear to students (consistently rubric be clear to students (consistently interpreted among students) as to the facultyinterpreted among students) as to the faculty Student interpretations should be the same as the Student interpretations should be the same as the

intended meaningintended meaning Students should recognize a consistent thread Students should recognize a consistent thread

regarding CT across courses and instructorsregarding CT across courses and instructors A team of students will use the rubric to score A team of students will use the rubric to score

anonymous student work in an IRR studyanonymous student work in an IRR study

Applying The RubricApplying The Rubric Sample work from the Intro and Capstone Sample work from the Intro and Capstone

courses will be collected and assessedcourses will be collected and assessed As the project ages, we will collect repeated As the project ages, we will collect repeated

samples from individual students where samples from individual students where possiblepossible

We will aim to codify a comparable use of the We will aim to codify a comparable use of the rubric in one or more 300-level courses from rubric in one or more 300-level courses from the Biology major’s curriculum and encourage the Biology major’s curriculum and encourage use of our CT rubric wherever appropriate use of our CT rubric wherever appropriate throughout the Biology curriculumthroughout the Biology curriculum

Work by the Assessment TeamWork by the Assessment Team Team will meet after each semester to assess at least Team will meet after each semester to assess at least

30 papers from each course30 papers from each course Each meeting will start with a small IRR study to align Each meeting will start with a small IRR study to align

the scoring practices of team membersthe scoring practices of team members Each piece of student work will be scored by two team Each piece of student work will be scored by two team

members, and discrepant scores will be resolved by a members, and discrepant scores will be resolved by a third team memberthird team member

We have proposed to pay all participants in this We have proposed to pay all participants in this project for 3 yearsproject for 3 years We hope that use of the rubric by the faculty for both student We hope that use of the rubric by the faculty for both student

and programmatic assessment will become valued and and programmatic assessment will become valued and routine, and easily incorporated into ordinary departmental routine, and easily incorporated into ordinary departmental activities when the grant has endedactivities when the grant has ended

Some Planned Data AnalysesSome Planned Data Analyses Track patterns of growth in student CT skillsTrack patterns of growth in student CT skills Compare performance in CT skills with other Compare performance in CT skills with other

variablesvariables E.g., GPA, number of CT-type Biology courses E.g., GPA, number of CT-type Biology courses

taken, number of CLA courses takentaken, number of CLA courses taken Compare 1Compare 1stst generation college student generation college student

performance with othersperformance with others Compare transfer with non-transfer studentsCompare transfer with non-transfer students Compare assessed CT skills with the Compare assessed CT skills with the

perceptions of our alumni and their perceptions of our alumni and their employersemployers

Refine Our Curricular MapRefine Our Curricular Map Obtain a more detailed understanding of Obtain a more detailed understanding of

the extent to which CT skills are taught the extent to which CT skills are taught and/or assessed in particular coursesand/or assessed in particular courses

Are there gaps in the curriculum allowing Are there gaps in the curriculum allowing some students to miss regular attention to some students to miss regular attention to CT skills?CT skills?

Use the results of assessments and Use the results of assessments and analyses to suggest curricular analyses to suggest curricular modifications addressing problems modifications addressing problems revealed by the processrevealed by the process

Our Guiding Principles:Our Guiding Principles:Faculty Input and TrainingFaculty Input and Training

Workshops and retreats focused on Workshops and retreats focused on defining and assessing CTdefining and assessing CT

Discussion involvement and presentations Discussion involvement and presentations by outside expertsby outside experts

Well paid faculty will participate in this Well paid faculty will participate in this project from start to finishproject from start to finish Improved quality controlImproved quality control Faculty buy-in regarding use of the rubric and Faculty buy-in regarding use of the rubric and

the value of the value of programmaticprogrammatic assessment assessment

Our Guiding Principles:Our Guiding Principles:Including TMCCIncluding TMCC

Engaging TMCC Biology faculty in this sort Engaging TMCC Biology faculty in this sort of project should:of project should: Improve our understanding of pedagogy at Improve our understanding of pedagogy at

TMCCTMCC Improve understanding by TMCC faculty of Improve understanding by TMCC faculty of

our curricular expectationsour curricular expectations Facilitate the transition of transfer students Facilitate the transition of transfer students

into the UNR Biology majorinto the UNR Biology major

Our Guiding Principles:Our Guiding Principles:Communicate Our Process and Communicate Our Process and

Outcomes to Others Outcomes to Others Align efforts with the UNR Core Curriculum plans to Align efforts with the UNR Core Curriculum plans to

assess general (not discipline-specific) CT skillsassess general (not discipline-specific) CT skills Inform Biologists at all other UCCSN institutions of Inform Biologists at all other UCCSN institutions of

our project, our rubric, and our resultsour project, our rubric, and our results Once our rubric is adopted by the department for Once our rubric is adopted by the department for

use throughout the curriculum, it will be published use throughout the curriculum, it will be published on the departmental websiteon the departmental website

We will publish our process and analytical results in We will publish our process and analytical results in a journal specializing in educational researcha journal specializing in educational research E.g., E.g., The Journal of Research in Science TeachingThe Journal of Research in Science Teaching