Developing Your Own Course-Developing Your Own Course-Specific Assessment Tool:Specific Assessment Tool:
Strategies for Teaching FacultyStrategies for Teaching FacultyCS-5.10, Track 2CS-5.10, Track 2
David J. SilvaDavid J. SilvaVice Provost for Academic AffairsVice Provost for Academic Affairs
Professor of Linguistics & Distinguished Teaching ProfessorProfessor of Linguistics & Distinguished Teaching Professor
9th Annual Texas A&M University Assessment Conference, February 24, 2009
22
The IssueThe Issue
Pressure to assessPressure to assess– … … increasingly at the course levelincreasingly at the course level
Faculty concernsFaculty concerns– administrative intrusionadministrative intrusion– perceived lack of utility (“just another hoop”)perceived lack of utility (“just another hoop”)– lack of knowledge about assessmentlack of knowledge about assessment– lack of experience developing assessment toolslack of experience developing assessment tools– additional burden (“doing more with the same”)additional burden (“doing more with the same”)
33
Addressing Potential TensionAddressing Potential Tension
Faculty concernsFaculty concerns– administrative administrative
intrusionintrusion– perceived lack of perceived lack of
utilityutility(“just another hoop”)(“just another hoop”)
– lack of knowledge lack of knowledge about assessmentabout assessment
– lack of experience lack of experience developing developing assessment toolsassessment tools
– additional burden additional burden (“doing more with the (“doing more with the same”)same”)
Admin ResponsesAdmin Responses– administrative administrative
scaffoldingscaffolding– focus on benefits to focus on benefits to
instructors / programs instructors / programs (“knowledge= power”)(“knowledge= power”)
– provide professional provide professional developmentdevelopment
– provide focused provide focused seminars and associ-seminars and associ-ated incentivesated incentives
– focus on integration focus on integration (“maximizing existing (“maximizing existing resources”)resources”)
44
Contextualized Approach: QEPContextualized Approach: QEP
““Active Learning:Active Learning: Pathways to Higher Order Thinking at UT Arlington” Pathways to Higher Order Thinking at UT Arlington”– Ten Pilot ProjectsTen Pilot Projects
Education: Literacy StudiesEducation: Literacy Studies
Engineering: First-Year, Electrical Engineering, CapstoneEngineering: First-Year, Electrical Engineering, Capstone
Honors College: SeminarHonors College: Seminar
Liberal Arts: English/Libraries, History, Political ScienceLiberal Arts: English/Libraries, History, Political Science
Nursing: CapstoneNursing: Capstone
Science: MathematicsScience: Mathematics
– In search of a second path for assessment (“Triangulation”)In search of a second path for assessment (“Triangulation”)In place: Criterion-based course rubricsIn place: Criterion-based course rubrics
Problematic: Program-level assessment, multiple choice CT testProblematic: Program-level assessment, multiple choice CT test
Developing appreciation for the role of Developing appreciation for the role of contextcontext in in assessing HOTS – Content Based Assessment (CBA)assessing HOTS – Content Based Assessment (CBA)
55
Multifaceted ApproachMultifaceted Approach
Ongoing professional developmentOngoing professional development– Workshop on Grading and Assessment; Generation NeXtWorkshop on Grading and Assessment; Generation NeXt– Teaching Circles for assessment as well as for active learningTeaching Circles for assessment as well as for active learning– Afternoon seminar / discussion group on active learningAfternoon seminar / discussion group on active learning– Special One-Day QEP PI workshop on CBASpecial One-Day QEP PI workshop on CBA
Continued appeal to faculty’s innate curiosityContinued appeal to faculty’s innate curiosity– Assessment = ResearchAssessment = Research– Research earns rewardResearch earns reward
Resources for InnovationResources for Innovation– Faculty awardFaculty award– Small seed grantsSmall seed grants– Faculty Development LeavesFaculty Development Leaves
Focus on integration: “maximizing existing resources”Focus on integration: “maximizing existing resources”
66
QEP’s Existing ResourcesQEP’s Existing Resources
Basic understanding of assessmentBasic understanding of assessment
Appreciation for the cyclic nature of Appreciation for the cyclic nature of assessmentassessment
Clearly articulated student learning Clearly articulated student learning outcomes (SLOs)outcomes (SLOs)
Desire to optimize research resultsDesire to optimize research results
Community of PracticeCommunity of Practice
Culture of good citizenshipCulture of good citizenship
77
Goals of the CBA WorkshopGoals of the CBA Workshop
Reconnect PIs with Bloom’s TaxonomyReconnect PIs with Bloom’s Taxonomy
(Re-)identify the primary SLO(Re-)identify the primary SLO
Look toward authenticity: the disciplineLook toward authenticity: the discipline– ““What sorts of … What sorts of …
knowledgeknowledge
comprehensioncomprehension
application … application …
… … does a successful ________ draw upon?”does a successful ________ draw upon?”
Begin to draft assessment questionsBegin to draft assessment questions
88
What Are We Creating?What Are We Creating?
Direct AssessmentDirect Assessment– Complements indirect assessmentComplements indirect assessment
Interim Questionnaires (engagement, effort)Interim Questionnaires (engagement, effort)Focus Group DataFocus Group Data
Value Added OutcomesValue Added Outcomes– Complements absolute outcomesComplements absolute outcomes
Rubric DataRubric Data
Embedded Assessment (potentially)Embedded Assessment (potentially)– Used as course-initial orientation toolUsed as course-initial orientation tool– Used as part of final exam (optional)Used as part of final exam (optional)
Summative AssessmentSummative AssessmentTriangulationTriangulation Thanks to Mary J. Allen
for this framework
99
Let’s “Learn Active!”Let’s “Learn Active!”
1.1. Course InformationCourse Information2.2. Key SLOKey SLO
– What’s the What’s the chiefchief objective objective (Broadly Conceptualized)(Broadly Conceptualized)??
3.3. Disciplinary skillsDisciplinary skills– What do people who do X do?What do people who do X do?– What do people who do X need to know?What do people who do X need to know?
4.4. Course-related skillsCourse-related skills– How does this course contribute to (3)?How does this course contribute to (3)?– Beginnings of possible questionsBeginnings of possible questions
1010
Bloom ↔ Discipline ↔ CourseBloom ↔ Discipline ↔ Course
Let’s see a bit of what we’ve accomplished thus far…Let’s see a bit of what we’ve accomplished thus far…
YY--NN
Bloom’s LevelsBloom’s Levels Discipline / What Discipline / What Practitioners DoPractitioners Do
This Particular This Particular CourseCourse
KnowledgeKnowledge
ComprehensionComprehension
ApplicationApplication
AnalysisAnalysis
SynthesisSynthesis
EvaluationEvaluation
1111
HIST 1311HIST 1311
““History of the United States”History of the United States”
Professor Stephanie ColeProfessor Stephanie Cole
130 – 145 students130 – 145 students
Innovation: Personal Response SystemsInnovation: Personal Response Systems
1212
HIST 1311HIST 1311
In recent years, there has been considerable In recent years, there has been considerable debate in the United States regarding the status debate in the United States regarding the status of same sex marriage. In some states, such of same sex marriage. In some states, such unions are legal. In others, they are prohibited unions are legal. In others, they are prohibited by the state Constitution. From an historian’s by the state Constitution. From an historian’s perspective, which of the following events would perspective, which of the following events would provide the most insight when it comes to provide the most insight when it comes to understanding the sort of conflict regarding understanding the sort of conflict regarding federal vs. states rights?federal vs. states rights?a)a) The Teapot Dome scandal of the 1920’sThe Teapot Dome scandal of the 1920’sb)b) The Missouri Compromise of 1820The Missouri Compromise of 1820c)c) The ratification of the 14th Amendment theThe ratification of the 14th Amendment the
US Constitution in 1868US Constitution in 1868d)d) The Nullification Crisis of the 1830’sThe Nullification Crisis of the 1830’s
1313
POLS 2311POLS 2311
““Government of the United States”Government of the United States”
Professor Rebecca DeenProfessor Rebecca Deen
60 – 110 students60 – 110 students
Innovation: Campaign Service ModuleInnovation: Campaign Service Module
1414
POLS 2311POLS 2311
Which of the following is Which of the following is notnot a a component of the Michigan Model of component of the Michigan Model of voting behavior?voting behavior?
a) partisan identificationa) partisan identificationb) social groupsb) social groups
c) candidate evaluationsc) candidate evaluations
d) issue preferenced) issue preference
1515
LING 2301LING 2301
““Intro to the Study of Human Language”Intro to the Study of Human Language”
Professor David SilvaProfessor David Silva
25 – 30 students25 – 30 students
Innovations: Blog; Public Service ProductInnovations: Blog; Public Service Product
1616
LING 2301LING 2301
When a person suffers from the neurological When a person suffers from the neurological condition known as aphasia, not only are his condition known as aphasia, not only are his language capacities diminished in some language capacities diminished in some regard, but in addition, he presents difficulties regard, but in addition, he presents difficulties with:with:
a)a) moving the left side of his body (arm, leg, mouth, moving the left side of his body (arm, leg, mouth, etc.)etc.)
b)b) moving the right side of his body (arm, leg, mouth, moving the right side of his body (arm, leg, mouth, etc.)etc.)
c)c) his nighttime visionhis nighttime vision
d)d) remembering the names of close friends and family remembering the names of close friends and family membersmembers
e)e) breathing normally, particularly on high-ozone days breathing normally, particularly on high-ozone days
1717
LING 2301LING 2301
As a parent, you are hoping to invest wisely in the As a parent, you are hoping to invest wisely in the linguistic development of your 3-year old. More linguistic development of your 3-year old. More specifically, you want your toddler to enter school specifically, you want your toddler to enter school as an English-Chinese bilingual. Hence, you:as an English-Chinese bilingual. Hence, you:
a)a) take Chinese classes and share what you’ve learned with your take Chinese classes and share what you’ve learned with your child each daychild each day
b)b) subscribe to a Chinese-language satellite TV channel and have subscribe to a Chinese-language satellite TV channel and have your child watch children’s programming each morning from your child watch children’s programming each morning from 8:00 to 11:308:00 to 11:30
c)c) hire a Chinese housekeeper, who takes your child when hire a Chinese housekeeper, who takes your child when running errands each day, and who speaks Chinese with your running errands each day, and who speaks Chinese with your child regularlychild regularly
d)d) leave your child with a Chinese babysitter, who works from her leave your child with a Chinese babysitter, who works from her home because she is raising a 1-year old and a 3-year old of home because she is raising a 1-year old and a 3-year old of her ownher own
e)e) enroll your child in an ethnically diverse preschool, making enroll your child in an ethnically diverse preschool, making sure that there are Chinese-speaking children in the programsure that there are Chinese-speaking children in the program
1818
LING 2301: ResultsLING 2301: Results
Pre-TestPre-Test Post-TestPost-Test ΔΔ
Mean Proportion Mean Proportion Correct Correct (r(rss=0.504, p=0.039)=0.504, p=0.039)
39%39%(sd=27%)(sd=27%)
65%65%(sd=24%)(sd=24%)
+26%+26%
Mean Adjusted ScoreMean Adjusted Score(zero-baseline)(zero-baseline)
-2.18-2.18(sd=4.0)(sd=4.0)
+3.65+3.65(sd=4.3)(sd=4.3)
+5.82+5.82
Proportion below zero Proportion below zero baselinebaseline 82%82% 12%12%
Proportion withProportion withValue Added > 0Value Added > 0 82%82% (2 zero, 1 negative)(2 zero, 1 negative)
LING 2301: SurpriseLING 2301: Surprise
Which of the following Which of the following conditions is conditions is notnot considered considered a prerequisite to successful a prerequisite to successful language acquisition?language acquisition?
a)a) cognitive capacity (i.e., cognitive capacity (i.e., developmentally normal brain developmentally normal brain function)function)
b)b) access to data in the form of access to data in the form of appropriate textbooks and/or appropriate textbooks and/or electronic media (e.g., CDs)electronic media (e.g., CDs)
c)c) opportunities for authentic opportunities for authentic interactioninteraction
d)d) integrative motivationintegrative motivatione)e) instrumental motivationinstrumental motivation
92% correct (Pre-test)
53% chosen on Post-test!
2020
Still to be DoneStill to be Done
Moving pedagogy into the higher domains as Moving pedagogy into the higher domains as appropriateappropriate– Rethinking Rethinking knowledgeknowledge as something else as something else
Working on how to write the questionsWorking on how to write the questions– Learning from others, e.g. TTU’s CATLearning from others, e.g. TTU’s CAT
Validating each CBA instrumentValidating each CBA instrument
Continuing to assess the QEP as we move Continuing to assess the QEP as we move toward Phase IItoward Phase II
Fostering a culture of assessment campus-wideFostering a culture of assessment campus-wide
2121
What Do We Have for Our Efforts?What Do We Have for Our Efforts?
Some answers to “WIIFM?”Some answers to “WIIFM?”
Some more dedicated amateur assessorsSome more dedicated amateur assessors– note: the root of note: the root of amateuramateur is “love” is “love”
Some additional movement in our ongoing Some additional movement in our ongoing shift into a “culture of assessment”shift into a “culture of assessment”
“If we don't change direction soon,we'll end up where we're going.”
-Irwin Corey
Thank YouThank You
ありがとう Asante DankeDiolch Gracias Gràcies
Grazie 감사합니다 KöszönömMahalo Merci Obrigado
Спасибо Tack Terima Kasih谢谢
David J. SilvaDavid J. [email protected]@uta.edu
Be A Maverick. ™