Developing Your Own Course- Specific Assessment Tool: Strategies for Teaching Faculty CS-5.10, Track 2 David J. Silva Vice Provost for Academic Affairs.

Developing Your Own Course-Developing Your Own Course-Specific Assessment Tool:Specific Assessment Tool:

Strategies for Teaching FacultyStrategies for Teaching FacultyCS-5.10, Track 2CS-5.10, Track 2

David J. SilvaDavid J. SilvaVice Provost for Academic AffairsVice Provost for Academic Affairs

Professor of Linguistics & Distinguished Teaching ProfessorProfessor of Linguistics & Distinguished Teaching Professor

9th Annual Texas A&M University Assessment Conference, February 24, 2009

22

The IssueThe Issue

Pressure to assessPressure to assess– … … increasingly at the course levelincreasingly at the course level

Faculty concernsFaculty concerns– administrative intrusionadministrative intrusion– perceived lack of utility (“just another hoop”)perceived lack of utility (“just another hoop”)– lack of knowledge about assessmentlack of knowledge about assessment– lack of experience developing assessment toolslack of experience developing assessment tools– additional burden (“doing more with the same”)additional burden (“doing more with the same”)

33

Addressing Potential TensionAddressing Potential Tension

Faculty concernsFaculty concerns– administrative administrative

intrusionintrusion– perceived lack of perceived lack of

utilityutility(“just another hoop”)(“just another hoop”)

– lack of knowledge lack of knowledge about assessmentabout assessment

– lack of experience lack of experience developing developing assessment toolsassessment tools

– additional burden additional burden (“doing more with the (“doing more with the same”)same”)

Admin ResponsesAdmin Responses– administrative administrative

scaffoldingscaffolding– focus on benefits to focus on benefits to

instructors / programs instructors / programs (“knowledge= power”)(“knowledge= power”)

– provide professional provide professional developmentdevelopment

– provide focused provide focused seminars and associ-seminars and associ-ated incentivesated incentives

– focus on integration focus on integration (“maximizing existing (“maximizing existing resources”)resources”)

44

Contextualized Approach: QEPContextualized Approach: QEP

““Active Learning:Active Learning: Pathways to Higher Order Thinking at UT Arlington” Pathways to Higher Order Thinking at UT Arlington”– Ten Pilot ProjectsTen Pilot Projects

Education: Literacy StudiesEducation: Literacy Studies

Engineering: First-Year, Electrical Engineering, CapstoneEngineering: First-Year, Electrical Engineering, Capstone

Honors College: SeminarHonors College: Seminar

Liberal Arts: English/Libraries, History, Political ScienceLiberal Arts: English/Libraries, History, Political Science

Nursing: CapstoneNursing: Capstone

Science: MathematicsScience: Mathematics

– In search of a second path for assessment (“Triangulation”)In search of a second path for assessment (“Triangulation”)In place: Criterion-based course rubricsIn place: Criterion-based course rubrics

Problematic: Program-level assessment, multiple choice CT testProblematic: Program-level assessment, multiple choice CT test

Developing appreciation for the role of Developing appreciation for the role of contextcontext in in assessing HOTS – Content Based Assessment (CBA)assessing HOTS – Content Based Assessment (CBA)

55

Multifaceted ApproachMultifaceted Approach

Ongoing professional developmentOngoing professional development– Workshop on Grading and Assessment; Generation NeXtWorkshop on Grading and Assessment; Generation NeXt– Teaching Circles for assessment as well as for active learningTeaching Circles for assessment as well as for active learning– Afternoon seminar / discussion group on active learningAfternoon seminar / discussion group on active learning– Special One-Day QEP PI workshop on CBASpecial One-Day QEP PI workshop on CBA

Continued appeal to faculty’s innate curiosityContinued appeal to faculty’s innate curiosity– Assessment = ResearchAssessment = Research– Research earns rewardResearch earns reward

Resources for InnovationResources for Innovation– Faculty awardFaculty award– Small seed grantsSmall seed grants– Faculty Development LeavesFaculty Development Leaves

Focus on integration: “maximizing existing resources”Focus on integration: “maximizing existing resources”

66

QEP’s Existing ResourcesQEP’s Existing Resources

Basic understanding of assessmentBasic understanding of assessment

Appreciation for the cyclic nature of Appreciation for the cyclic nature of assessmentassessment

Clearly articulated student learning Clearly articulated student learning outcomes (SLOs)outcomes (SLOs)

Desire to optimize research resultsDesire to optimize research results

Community of PracticeCommunity of Practice

Culture of good citizenshipCulture of good citizenship

77

Goals of the CBA WorkshopGoals of the CBA Workshop

Reconnect PIs with Bloom’s TaxonomyReconnect PIs with Bloom’s Taxonomy

(Re-)identify the primary SLO(Re-)identify the primary SLO

Look toward authenticity: the disciplineLook toward authenticity: the discipline– ““What sorts of … What sorts of …

knowledgeknowledge

comprehensioncomprehension

application … application …

… … does a successful ________ draw upon?”does a successful ________ draw upon?”

Begin to draft assessment questionsBegin to draft assessment questions

88

What Are We Creating?What Are We Creating?

Direct AssessmentDirect Assessment– Complements indirect assessmentComplements indirect assessment

Interim Questionnaires (engagement, effort)Interim Questionnaires (engagement, effort)Focus Group DataFocus Group Data

Value Added OutcomesValue Added Outcomes– Complements absolute outcomesComplements absolute outcomes

Rubric DataRubric Data

Embedded Assessment (potentially)Embedded Assessment (potentially)– Used as course-initial orientation toolUsed as course-initial orientation tool– Used as part of final exam (optional)Used as part of final exam (optional)

Summative AssessmentSummative AssessmentTriangulationTriangulation Thanks to Mary J. Allen

for this framework

99

Let’s “Learn Active!”Let’s “Learn Active!”

1.1. Course InformationCourse Information2.2. Key SLOKey SLO

– What’s the What’s the chiefchief objective objective (Broadly Conceptualized)(Broadly Conceptualized)??

3.3. Disciplinary skillsDisciplinary skills– What do people who do X do?What do people who do X do?– What do people who do X need to know?What do people who do X need to know?

4.4. Course-related skillsCourse-related skills– How does this course contribute to (3)?How does this course contribute to (3)?– Beginnings of possible questionsBeginnings of possible questions

1010

Bloom ↔ Discipline ↔ CourseBloom ↔ Discipline ↔ Course

Let’s see a bit of what we’ve accomplished thus far…Let’s see a bit of what we’ve accomplished thus far…

YY--NN

Bloom’s LevelsBloom’s Levels Discipline / What Discipline / What Practitioners DoPractitioners Do

This Particular This Particular CourseCourse

KnowledgeKnowledge

ComprehensionComprehension

ApplicationApplication

AnalysisAnalysis

SynthesisSynthesis

EvaluationEvaluation

1111

HIST 1311HIST 1311

““History of the United States”History of the United States”

Professor Stephanie ColeProfessor Stephanie Cole

130 – 145 students130 – 145 students

Innovation: Personal Response SystemsInnovation: Personal Response Systems

1212

HIST 1311HIST 1311

In recent years, there has been considerable In recent years, there has been considerable debate in the United States regarding the status debate in the United States regarding the status of same sex marriage. In some states, such of same sex marriage. In some states, such unions are legal. In others, they are prohibited unions are legal. In others, they are prohibited by the state Constitution. From an historian’s by the state Constitution. From an historian’s perspective, which of the following events would perspective, which of the following events would provide the most insight when it comes to provide the most insight when it comes to understanding the sort of conflict regarding understanding the sort of conflict regarding federal vs. states rights?federal vs. states rights?a)a) The Teapot Dome scandal of the 1920’sThe Teapot Dome scandal of the 1920’sb)b) The Missouri Compromise of 1820The Missouri Compromise of 1820c)c) The ratification of the 14th Amendment theThe ratification of the 14th Amendment the

US Constitution in 1868US Constitution in 1868d)d) The Nullification Crisis of the 1830’sThe Nullification Crisis of the 1830’s

1313

POLS 2311POLS 2311

““Government of the United States”Government of the United States”

Professor Rebecca DeenProfessor Rebecca Deen

60 – 110 students60 – 110 students

Innovation: Campaign Service ModuleInnovation: Campaign Service Module

1414

POLS 2311POLS 2311

Which of the following is Which of the following is notnot a a component of the Michigan Model of component of the Michigan Model of voting behavior?voting behavior?

a) partisan identificationa) partisan identificationb) social groupsb) social groups

c) candidate evaluationsc) candidate evaluations

d) issue preferenced) issue preference

1515

LING 2301LING 2301

““Intro to the Study of Human Language”Intro to the Study of Human Language”

Professor David SilvaProfessor David Silva

25 – 30 students25 – 30 students

Innovations: Blog; Public Service ProductInnovations: Blog; Public Service Product

1616

LING 2301LING 2301

When a person suffers from the neurological When a person suffers from the neurological condition known as aphasia, not only are his condition known as aphasia, not only are his language capacities diminished in some language capacities diminished in some regard, but in addition, he presents difficulties regard, but in addition, he presents difficulties with:with:

a)a) moving the left side of his body (arm, leg, mouth, moving the left side of his body (arm, leg, mouth, etc.)etc.)

b)b) moving the right side of his body (arm, leg, mouth, moving the right side of his body (arm, leg, mouth, etc.)etc.)

c)c) his nighttime visionhis nighttime vision

d)d) remembering the names of close friends and family remembering the names of close friends and family membersmembers

e)e) breathing normally, particularly on high-ozone days breathing normally, particularly on high-ozone days

1717

LING 2301LING 2301

As a parent, you are hoping to invest wisely in the As a parent, you are hoping to invest wisely in the linguistic development of your 3-year old. More linguistic development of your 3-year old. More specifically, you want your toddler to enter school specifically, you want your toddler to enter school as an English-Chinese bilingual. Hence, you:as an English-Chinese bilingual. Hence, you:

a)a) take Chinese classes and share what you’ve learned with your take Chinese classes and share what you’ve learned with your child each daychild each day

b)b) subscribe to a Chinese-language satellite TV channel and have subscribe to a Chinese-language satellite TV channel and have your child watch children’s programming each morning from your child watch children’s programming each morning from 8:00 to 11:308:00 to 11:30

c)c) hire a Chinese housekeeper, who takes your child when hire a Chinese housekeeper, who takes your child when running errands each day, and who speaks Chinese with your running errands each day, and who speaks Chinese with your child regularlychild regularly

d)d) leave your child with a Chinese babysitter, who works from her leave your child with a Chinese babysitter, who works from her home because she is raising a 1-year old and a 3-year old of home because she is raising a 1-year old and a 3-year old of her ownher own

e)e) enroll your child in an ethnically diverse preschool, making enroll your child in an ethnically diverse preschool, making sure that there are Chinese-speaking children in the programsure that there are Chinese-speaking children in the program

1818

LING 2301: ResultsLING 2301: Results

Pre-TestPre-Test Post-TestPost-Test ΔΔ

Mean Proportion Mean Proportion Correct Correct (r(rss=0.504, p=0.039)=0.504, p=0.039)

39%39%(sd=27%)(sd=27%)

65%65%(sd=24%)(sd=24%)

+26%+26%

Mean Adjusted ScoreMean Adjusted Score(zero-baseline)(zero-baseline)

-2.18-2.18(sd=4.0)(sd=4.0)

+3.65+3.65(sd=4.3)(sd=4.3)

+5.82+5.82

Proportion below zero Proportion below zero baselinebaseline 82%82% 12%12%

Proportion withProportion withValue Added > 0Value Added > 0 82%82% (2 zero, 1 negative)(2 zero, 1 negative)

LING 2301: SurpriseLING 2301: Surprise

Which of the following Which of the following conditions is conditions is notnot considered considered a prerequisite to successful a prerequisite to successful language acquisition?language acquisition?

a)a) cognitive capacity (i.e., cognitive capacity (i.e., developmentally normal brain developmentally normal brain function)function)

b)b) access to data in the form of access to data in the form of appropriate textbooks and/or appropriate textbooks and/or electronic media (e.g., CDs)electronic media (e.g., CDs)

c)c) opportunities for authentic opportunities for authentic interactioninteraction

d)d) integrative motivationintegrative motivatione)e) instrumental motivationinstrumental motivation

92% correct (Pre-test)

53% chosen on Post-test!

2020

Still to be DoneStill to be Done

Moving pedagogy into the higher domains as Moving pedagogy into the higher domains as appropriateappropriate– Rethinking Rethinking knowledgeknowledge as something else as something else

Working on how to write the questionsWorking on how to write the questions– Learning from others, e.g. TTU’s CATLearning from others, e.g. TTU’s CAT

Validating each CBA instrumentValidating each CBA instrument

Continuing to assess the QEP as we move Continuing to assess the QEP as we move toward Phase IItoward Phase II

Fostering a culture of assessment campus-wideFostering a culture of assessment campus-wide

2121

What Do We Have for Our Efforts?What Do We Have for Our Efforts?

Some answers to “WIIFM?”Some answers to “WIIFM?”

Some more dedicated amateur assessorsSome more dedicated amateur assessors– note: the root of note: the root of amateuramateur is “love” is “love”

Some additional movement in our ongoing Some additional movement in our ongoing shift into a “culture of assessment”shift into a “culture of assessment”

“If we don't change direction soon,we'll end up where we're going.”

-Irwin Corey

Thank YouThank You

ありがとう Asante DankeDiolch Gracias Gràcies

Grazie 감사합니다 KöszönömMahalo Merci Obrigado

Спасибо 　　 Tack Terima Kasih谢谢

David J. SilvaDavid J. [email protected]@uta.edu

Be A Maverick. ™