Assessing Critical Thinking Skills Dr. Barry Stein - Professor of Psychology, Director of Planning, Coordinator of TTU Critical Thinking Initiative Dr.

Assessing Critical Thinking Skills

Dr. Barry SteinDr. Barry Stein - Professor of Psychology, - Professor of Psychology,

Director of Planning, Coordinator of TTU Director of Planning, Coordinator of TTU Critical Thinking InitiativeCritical Thinking Initiative

Dr. Ada HaynesDr. Ada Haynes - Professor of Sociology - Professor of Sociology

Jenny UntersteinJenny Unterstein - Doctoral Student - Doctoral Student

Tennessee Technological UniversityTennessee Technological University

http://iweb.tntech.edu/cti/

Assessing Critical Thinking for Continuous Improvement Why?Why? How?How?

Process & ProductProcess & Product

What is Critical Thinking

Classic EmphasisClassic Emphasis

R easoning

Evaluate A rgum ents and C onclusions

What is Critical Thinking

Expanded Contemporary EmphasisExpanded Contemporary Emphasis

R easoning

Evaluate A rgum ents and C onclusions

C om m unication

C reativ ity

Problem Solv ing

Evaluate Ideas and P lans

Life-Long Learning Skills

Evaluate One's U nderstanding

Why Critical Thinking is Important for Good Citizens? Informed and Rational Informed and Rational

Participation Participation Free from Ignorance, Free from Ignorance,

Confusion, and Confusion, and Unjustified ClaimsUnjustified Claims

Ability to Compete Ability to Compete GloballyGlobally

Goals 2000: Educate Goals 2000: Educate America ActAmerica Act

Critical Thinking in an Information Age Information becomes Information becomes

obsolete quicklyobsolete quickly Gauge AccuracyGauge Accuracy Sort Relevant from Sort Relevant from

IrrelevantIrrelevant Sort Quality from Sort Quality from

“Junk Science”“Junk Science”

Researchers Find Bread Causes Crime 99.5% of people who committed crimes 99.5% of people who committed crimes

consumed bread 24 hours before the crimeconsumed bread 24 hours before the crime In areas where bread consumption is low In areas where bread consumption is low

there are few crimes reportedthere are few crimes reported

Relative Importance of Skills for Employers

5

6

Problem

Sol

ving

Comm

unica

tion

Team W

ork

Lear

ning

Skil

ls

Critica

l Thi

nking

Ethics

Technica

l Skil

ls

Work

ing

with D

ivers

ity

Knowledge/Skill Type

Very Important

Recent TTU Employer Survey

Why Assess Critical Thinking?

Need to measure success for accountabilityNeed to measure success for accountability Assessment can focus faculty attention on Assessment can focus faculty attention on

the need to improve critical thinkingthe need to improve critical thinking

The Test Development Process

Maximize face validity for facultyMaximize face validity for faculty Maximize faculty involvement in scoringMaximize faculty involvement in scoring

Develop an assessment tool that Develop an assessment tool that encourages continuous improvementencourages continuous improvement

Recommended Steps

1.1. Identify faculty members interested in Identify faculty members interested in critical thinkingcritical thinking

2.2. Find a suitable existing assessment toolFind a suitable existing assessment tool

3.3. Evaluate the Evaluate the strengths and weaknesses strengths and weaknesses by by testing on a representative sampletesting on a representative sample

4.4. Determine if a new assessment is needed Determine if a new assessment is needed for your institutionfor your institution

Faculty Identify CT skills for Assessment

Focus on what graduates should be able to Focus on what graduates should be able to do in the real world that is not discipline do in the real world that is not discipline specificspecific

Skills that are relevant to being a Skills that are relevant to being a good citizengood citizen good consumer of good consumer of informationinformation productive member of societyproductive member of society

Major Areas of Assessment Identified by TTU Faculty

Evaluate claims, hypotheses, conclusionsEvaluate claims, hypotheses, conclusions Apply new information to solve real-world Apply new information to solve real-world

problemsproblems Communicate ideas effectivelyCommunicate ideas effectively

Detailed list

Develop Assessment Questions

Small groups or individuals to design Small groups or individuals to design questions/problemsquestions/problems Minimize familiarity biasMinimize familiarity bias Maxmize correspondence to real-world Maxmize correspondence to real-world

taskstasks Larger faculty groups evaluate face validityLarger faculty groups evaluate face validity

Develop Scoring Criteria

Clear & ReliableClear & Reliable More points reflect better skillsMore points reflect better skills Anticipate possible answersAnticipate possible answers Provide guidelines for scoring creative Provide guidelines for scoring creative

answersanswers Refine, refine, refineRefine, refine, refine

Refine Test

Improve Face Validity – expand faculty Improve Face Validity – expand faculty input input

Establish Criterion Validity – correlate with Establish Criterion Validity – correlate with other tests and performance measuresother tests and performance measures

Establish Construct Validity – relate Establish Construct Validity – relate underlying skills to contemporary theoryunderlying skills to contemporary theory

Maximize Inter-rater ReliabilityMaximize Inter-rater Reliability

Maximize Reliability of Scoring

Scoring workshopsScoring workshops Multiple Graders for each questionMultiple Graders for each question Modify scoring criteria using faculty inputModify scoring criteria using faculty input Modify or eliminate questions that cannot Modify or eliminate questions that cannot

be reliably scoredbe reliably scored

TTU’s CAT© (Critical thinking Assessment Test)

Tested on over 600 students in 3 yearsTested on over 600 students in 3 years Inter-rater reliability = Inter-rater reliability = 0.78 to 0.98 0.78 to 0.98 Test-retest reliability > .60Test-retest reliability > .60 Internal consistency Internal consistency = .64 = .64 Correlation with ACT (.Correlation with ACT (.659)659) and CCTST ( and CCTST (.645).645) Sensitive to class standing (Sensitive to class standing (freshmanfreshman vs vs senior senior)) Sensitive to effects of a Sensitive to effects of a class in critical thinkingclass in critical thinking Seven underlying factors (71.8% variance)Seven underlying factors (71.8% variance)

CAT materials

CAT features

One hour examOne hour exam Mostly short answer essayMostly short answer essay Faculty scored in workshopsFaculty scored in workshops

10 faculty score 120-150 exams in 1 day10 faculty score 120-150 exams in 1 day Detailed scoring guideDetailed scoring guide

Status and Future

Involve other institutions to improve face Involve other institutions to improve face validityvalidity

Evaluate and correct cultural biasEvaluate and correct cultural bias Develop national normsDevelop national norms

Thank You

TTU Critical Thinking Initiativehttp://iweb.tntech.edu/cti/