-
INCREASING EQUITY USING CULTURALLY RELEVANT ASSESSMENT
Karen Singer-Freeman, Director of Academic Planning &
Assessment
Christine Robinson, Executive Director
Harriet Hobbs, Director of Assessment Systems & University
Accreditation
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
Office of Assessment and Accreditation
http://assessment.uncc.edu/
2018 Designee
-
OVERVIEW
•Learning Outcomes for the Workshop•Approach to analyzing
equitability of assessments•Culturally Relevant Assessment
•Theoretical model•Student feedback
•Data dive•Engaging faculty in conversation
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
WORKSHOP LEARNING OUTCOMES
By the end of the workshop, participants will be prepared
to:
1) Identify questions and practices about inclusive assessment
and student learning that can be addressed using existing sources
of campus data.
2) Identify strengths and weaknesses of assignment types.
3) Understand the importance of disaggregating data by student
characteristics.
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Equity in Assessment and Data Sources• What questions could we
ask about assessments and
equity?• Do different assignments measure similar abilities?• Do
assignment formats influence students’ abilities to
demonstrate learning?• Do student learning outcome reports
capture important
differences in student success?
• Utilize software systems’ data • Learning Management Systems
(LMS)• Assessment Management Systems (AMS)• Student Information
Systems (SIS)
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
STUDENT LEARNING OUTCOME (SLO)
REPORTS
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Do student learning outcomes reports capture important
differences?
•Student Learning Outcomes Reporting can compress data•Full
assignment grade
•5 point ratings on multiple rubric dimensions•Average across
rubric grades
•Above or below a criterion cut off
•Using Learning Management Systems it is possible to
disaggregate data to see a more complete picture
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
SLO Report ExampleITCS 2300 Web-Based Application
Development-Computer Science• SLO- Students demonstrate ability to
create static web pages…
• Assignments - 4-6 webpages using a common theme: • 4 Criteria:
Understanding target demographic, Effective use of HTML and
JavaScript,
Browser compatibility, and Aesthetics.• Target: 80% will receive
average rubric score of 3 (acceptable) or higher• Result: 97%
received average rubric score of 3 (acceptable) or higher
• What other questions do we have?
2 3 11
47
139
0
50
100
150
1 2 3 4 5
Number of Students Receiving Each Rubric Rating
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Data Disaggregation: Is Achievement Equitable?
0% 0%4%
25%
72%
2% 3%7%
22%
67%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5
% Receiving Each Rating Disaggregated by Ethnicity
Non-URM (n = 146) URM (n = 60)
5% of URM Don’t meet but all Non-URM meet7.04 (1), p <
.01
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Does the difference in achievement matter?
3.76
3.15
3.66
3.01
0
0.5
1
1.5
2
2.5
3
3.5
4
Semester GPA Cummulative GPA
Average Semester and Cumulative GPAURM grades marginally lower
than Non-URM, p < .10
Non-URM (n = 146) URM (n = 60)PLEASE DO NOT COPY, REPRODUCE, OR
DISTRIBUTE WITHOUT PERMISSION
-
STUDENT SUCCESS: ASSIGNMENT TYPES
-
Activity1) Please pick up the paper on your seat and circle the
items
that would generally describe this sort of assignment.
2) Compare your assignment and responses with the person next to
you
3) Total the number of odd items and the number of even items
you selected
4) Discuss – What do you think these questions are trying to
measure?
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Culturally Relevant Assessment(Singer-Freeman, Hobbs, &
Robinson, 2019)
• Aligned with teaching and SLOs
• Clear instructions• Limits effects of prior knowledge and
privilege
• Scaffolding• Limits effects of prior knowledge and creates
opportunities for early
success
• Inclusive content
• High utility value• Work has meaning beyond the academic
context
• Avoids stereotype threat
• Results disaggregated
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Predictions Based on Analysis of Assignment Types
Low Risk
• Reflective writing
Moderate Risk
• Open-ended tests
• Inclusive projects
• Writing in discipline
High Risk
• Multiple-choice tests
• Formal Essays
• Homework
High Utility Value Low Utility Value
High Inclusive Content
Reflective Writing Inclusive Projects
LowInclusive Content
Writing in DisciplineFormal Essays,
Tests, HomeworkPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE
WITHOUT PERMISSION
-
What do students think about different types of assignments?
•16 items assess utility value, inclusive content, &
alignment
•Completed for extra credit (n = 162)•Lower level classes in
• Biology (2)• Theater (1)• Writing (2)
•Upper level class in • Public Health
42% 40%35%
0%10%20%30%40%50%
Non-White 1stGeneration
Male
19%10%
46%
24%
0%
10%
20%
30%
40%
50%
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
The Measure
Utility Value (Chronbach’s Alpha = .90)
• Result in something I will discuss with others.
• Help me understand myself better.
• Improve my understanding.
• Provide me with experience that will be professionally
useful.
• Have personal value.
----------------------------------
• Are only completed to earn a grade.
Inclusive Content (Chronbach’s Alpha = .93)
• Allow me to express my learning in my own words.
• Make me feel confident I can succeed.
• Include examples and materials that are familiar to me.
• Measure my true understanding.
• Allow me to relate class materials to my own experiences.
• Include clear instructions.
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Assignments RatedPlease indicate the extent to which _______
generally:Formal Papers or Essays
Reflective Writing in which you relate class materials to your
own experiences.
Inclusive Projects – Projects that allow you to write in
different styles or using familiar content.
Writing in Discipline – Writing assignments in the format used
in a career you are interested in pursuing (e.g., lab reports,
progress reports, theater reviews).
Oral Presentations
Short Answer Test Questions
Multiple-Choice Test Questions
oNot at all, A little, To a moderate extent, Very muchPLEASE DO
NOT COPY, REPRODUCE, OR DISTRIBUTE
WITHOUT PERMISSION
-
Student Responses – Utility Value
2.78 2.722.59 2.53 2.44
2.282.12.15
1.912.12 2.17
1.99 1.911.65
1
1.5
2
2.5
3
3.5
4
Writing inDiscipline
OralPresentation
InclusiveProject
ReflectiveWriting
FormalPapers
ShortAnswer Test
Mult. ChoiceTest
Average Rating on 4 point Likert ScaleCritical Value for Utility
Value Construct = .11, p < .05
Utility Completed for more than a gradePLEASE DO NOT COPY,
REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Student Responses – Inclusive Content
2.81 2.77 2.77 2.75 2.742.49 2.49
1
1.5
2
2.5
3
3.5
4
Writing inDiscipline
OralPresentation
InclusiveProject
ReflectiveWriting
ShortAnswer Test
FormalPapers
Mult. ChoiceTest
Average Rating on 4 point Likert ScaleCritical Value = .11, p
< .05
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Student Confirmation of Model
High/ModerateUtility Value
Low /Moderate Utility Value
High Inclusive Content
Reflective WritingOral Presentations
Inclusive Projects
LowInclusive Content
Writing in DisciplineFormal Essays
Tests
Possible Correction Confirmation Addition
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE
WITHOUT PERMISSION
-
Some Research Support(Singer-Freeman, Hobbs, & Robinson,
2019)
Theater Appreciation (n = 64)• 69% URM
• Achievement gap • Multiple-choice exams
• No gap • Inclusive projects
Child Development (n = 110)• 44% URM
• Achievement gap • In-class multiple choice quizzes
• No gap • Reflective writing • Online multiple choice
quizzes
9073
9483
0
50
100
Inclusive Project Multiple-Choice*
Theater AppreciationURM Non-URM
9077 85
93 85 87
0
50
100
ReflectiveWriting
In-ClassQuizzes*
OnlineQuizzes
Child Development
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Experimental Psychology
• 5 semesters
• 135 students • 30% URM
• Achievement Gap• URM students receive
lower grades on exams than on lab reports
• Non-URM students receive similar grades on exams and lab
reports
7973
80 76
0102030405060708090
100
Writing in theDiscipline
Open-Ended Exams
Experimental Psychology
URM* Non-URMPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Research Confirmation of Model
High/ModerateUtility Value
Low /Moderate Utility Value
High Inclusive Content
Reflective WritingOral Presentations
Inclusive Projects
LowInclusive Content
Writing in DisciplineFormal Essays
Tests
Possible Correction Confirmation Addition
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Discussion
•Do our measures capture utility value and inclusive
content?
•Do the predictions in the model and findings make sense given
your experience with classroom assessments?
•What else would you want to know about how different groups of
students respond to different types of assignments?
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
BREAK TIME
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
DISAGGREGATING RESULTS BY ASSESSMENT TYPE
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Our Process
•Obtained a list of 88 classes with enrollments of over 50
students and DFW rates of over 30%
•Reviewed 36 classes • Identified 9 that included different
forms of assessment and
reported grades through LMS• The majority of classes we rejected
only included testing
•Of the 9 classes with differentiated assignments • 100%
included Exams and Quizzes
• 2 Formal essays and 1 Writing in the Discipline (Papers)• 2
Projects and 1 Oral presentation (Projects)• 2 Homework problem
sets and 3 Informal writing (Homework)
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Classes•Pre-Calculus MATH 1103•Organic Chemistry Lab CHEM
2131L•Introduction to Communication Theory COMM 2100•Network Theory
ENGR 2112•Principles of Accounting ACCT 2122
(flipped)•Physiological Psychology PSYC 3113•Design and
Implementation ITCS 3112•Sociology of Health & Illness SOC
4120•Conservation Biology BIOL 4244
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
How to disaggregate data•Download grades, assignment
descriptions, and syllabi for desired course from LMS• Consider
merging data from multiple semesters if enrollments
are low
•Decide whether to include or exclude• Missing assignments•
Completion-based grades
•Aggregate grades for equivalent assignments•Convert points to
percentages •Merge file with demographic data from Institutional
Research
•Compare outcomes using independent groups t-testPLEASE DO NOT
COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
Introduction to the data tables
Partner with someone at your table and pick a dataset
p < .05 p < .10 p < .05 p < .10
Achievement Gaps Reverse Achievement Gaps
Title provides broad introduction to contents
N = Total number of students in the class
(URM, Transfer, or Aid-Based Loans) = Number of students who
were members of disaggregated group
Columns labeled with groups contain average percentage scored on
each type of assignment
sig. = Whether observed differences were statistically
significantPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Assignment Class N (URM)
Non-URM
URM sig N (Aid Loans)
No Loan
Aid Loan
sig N (Trans)
Native
Trans sig
Reading Response 20% Sociology of Health & Illness SOCY
4130
118 (63)
78 73 .05 129 (33)
75 79 ns 130 (81)
82 72 .001 Exams 25% (open book, online) 77 71 .01 72 76 .09 77
71 .01 Paper 35% 82 78 .05 79 83 .09 84 77 .001 Final Grade 77 70
.01 71 78 .001 76 70 .02 Assignment Class N
(URM) Non-URM
URM sig N (Aid Loans)
No Loan
Aid Loan
sig N (Trans)
Native
Trans sig
Quizzes (20%) Physiological Psychology PSYC 3113
51 (18)
87 83 ns 55 (17)
86 84 ns 54 (37)
90 84 .01 Exams (45%) 80 79 ns 79 78 ns 84 77 .04 Oral (15%) 97
94 ns 94 98 ns 97 96 ns Final Grade 82 81 ns 80 80 ns 82 80 ns
Assignment Class N (URM) Non-URM URM sig Quizzes 20%
Network Theory II ENGR 2112
42 (8)
85 84 ns Tests 40% 66 66 ns Final Exam 30% 72 65 ns Homework 10%
74 65 .09 Final Grade 67 65 ns Assignment Class N (URM) Non-URM URM
sig Pre-Lab Quiz
Organic Chemistry Lab CHEM 2131L
141 (43) 78 70 ns Lab reports 80 75 ns Final grade 81 76 ns
Assignment Class N (URM) Non-URM URM sig Exams (70%)
Introduction to Communication Theory COMM 2100
136(54)
73 72 ns Application Paper (10%) 88 84 .09 Minute Responses
(20%) 97 92 .07 Final Grade 72 70 ns
Assignment Class N (URM) Non-URM URM sig Quizzes (7%)
Principles of Accounting ACCT 2122 (Fli d Cl )
50 (18)
89 89 ns Exams (64%) 75 72 ns Shark Tank Group Project (14%) 85
88 .06 Homework (7%) 81 87 ns
Introductory Classes Disaggregated by URM status
Advanced Classes in Sociology and Psychology
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
Assignment
Class
N (URM)
Non-URM
URM
sig
Quizzes 20%
Network Theory II
ENGR 2112
42 (8)
85
84
ns
Tests 40%
66
66
ns
Final Exam 30%
72
65
ns
Homework 10%
74
65
.09
Final Grade
67
65
ns
Assignment
Class
N (URM)
Non-URM
URM
sig
Pre-Lab Quiz
Organic Chemistry Lab
CHEM 2131L
141 (43)
78
70
ns
Lab reports
80
75
ns
Final grade
81
76
ns
Assignment
Class
N (URM)
Non-URM
URM
sig
Exams (70%)
Introduction to Communication Theory
COMM 2100
136(54)
73
72
ns
Application Paper (10%)
88
84
.09
Minute Responses (20%)
97
92
.07
Final Grade
72
70
ns
Assignment
Class
N (URM)
Non-URM
URM
sig
Quizzes (7%)
Principles of Accounting
ACCT 2122 (Flipped Class)
50 (18)
89
89
ns
Exams (64%)
75
72
ns
Shark Tank Group Project (14%)
85
88
.06
Homework (7%)
81
87
ns
Final Grade
77
75
ns
Assignment
Class
N (URM)
Non-URM
URM
sig
Tests (80%)
Pre-Calculus
MATH 1103
101 (33)
79
78
ns
MyMathLab (20%)
79
80
ns
Final Grade
76
75
ns
-
Dive into some dataAnswer the following questions:1) For which
classes do the final grades tell the same story as
the assignment grades (look at significant and non-significant
differences)?
2) For which classes do different assignments result in
different patterns of performance than final grades (look at
significant and non-significant differences)?
3) Can you identify any particular type of assignment that might
be showing evidence of achievement gaps?
4) What else would you want to know?
5) Pick a class from the handout and outline some questions and
suggestions you might have for the faculty member who teaches the
class.
6) What other observations did you make?PLEASE DO NOT COPY,
REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION
-
URM Results
-6
-4
-2
0
2
4
6
8
10
12
Tests Papers Projects Homework Final Grade
Points Separating non-URM from URM students
MATH CHEM COMM ENGR ACCT PSYC ITCS SOC BIOLPLEASE DO NOT COPY,
REPRODUCE, OR DISTRIBUTE
WITHOUT PERMISSION
-
Transfer Student Results
-6
-4
-2
0
2
4
6
8
10
12
Tests Papers Projects Homework Final Grade
Points Separating Native fromTransfer Students
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Aid-Based Loan Student Results
-6
-4
-2
0
2
4
6
8
10
12
Tests Papers Projects Homework Final Grade
Points Separating non-Loan from Aid-Based Loan students
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Summary
FrequentAchievement Gaps
• Tests• Low stakes or high stakes• Multiple choice or open
ended• Open-book and Online
• Low-Stakes Homework, Writing, and Projects
• Formal Writing
Fewer Achievement Gaps
• Writing• Writing in the discipline• Reflective writing•
Inclusive projects
• Oral Report• Group Project
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
CONCLUSIONS &
NEXT STEPS
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
Next Steps• Disaggregation of responses to different forms of
assessment
reveals importance of• Aligning assessments with teaching and
SLOs• Attending to context
• Faculty Workshops• Differentiated Assignments/Assessments
• Utility Value and Inclusive Content• Implicit Bias
• Scale project to other courses and programs
• Determine if achievement gaps reflect assessments or
competence
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
DISCUSSION
If achievement gaps exist on your campus, after hearing this
talk, what new ways might you use to investigate these gaps?
• Who would you include in the conversation?
• What are approaches you could take to determine whether gaps
reflect assessments or competence?
• What would next steps include to address gaps that result from
each?
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
SPECIAL THANKS
UNC Charlotte Academic Affairs
Arna Erega
Students and Faculty
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
REFERENCESAssociation of American Colleges & Universities
(2018). A vision for
equity: results from AAC&U’s project committing to equity
and inclusive excellence: Campus-based strategies for student
success. Retrieved from
https://aacu.org/publications/vision-equity.
Eccles J. S. et al. Expectations, values and academic behaviors.
In: Spence JT, editor. Perspective on achievement and achievement
motivation. San Francisco: W. H. Freeman; 1983. pp. 75–146.
Harackiewicz et al. (2016). Achievement gaps with a
utility-value intervention: Disentangling race and social class.
Retrieved from
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4853302/
Montenegro, E., & Jankowski, N. A. (2017). Equity and
assessment: Moving towards culturally responsive assessment
(Occasional Paper No. 29). Urbana, IL: University of Illinois and
Indiana University, National Institute for Learning Outcomes
Assessment (NILOA).
Singer-Freeman, K., Hobbs, H., & Robinson, C. (2019).
Theoretical matrix of culturally relevant assessment. Assessment
Update.
PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT
PERMISSION
-
C o n t a c tInformation
Christine Robinson, [email protected] Harriet Hobbs,
[email protected]
Karen Singer-Freeman, [email protected] DO NOT COPY,
REPRODUCE, OR DISTRIBUTE
WITHOUT PERMISSION
Increasing Equity using culturally Relevant AssessmentOverview
Workshop Learning outcomesSlide Number 4Student Learning Outcome
(SLO) ReportsDo student learning outcomes reports capture important
differences?SLO Report ExampleData Disaggregation: �Is Achievement
Equitable?Does the difference in achievement matter?Student
Success: ASSIGNMENT TypesActivityCulturally Relevant
Assessment�(Singer-Freeman, Hobbs, & Robinson, 2019)
Predictions Based on Analysis of �Assignment TypesWhat do students
think about �different types of assignments?The MeasureAssignments
RatedStudent Responses – Utility ValueStudent Responses – Inclusive
ContentStudent Confirmation of ModelSome Research
Support�(Singer-Freeman, Hobbs, & Robinson, 2019)Experimental
PsychologyResearch Confirmation of ModelDiscussionBreak
timeDisaggregating Results by assessment typeOur ProcessClassesHow
to disaggregate dataIntroduction to the data tablesSlide Number
30Dive into some dataURM ResultsTransfer Student ResultsAid-Based
Loan Student ResultsSummaryConclusions �& �Next StepsNext
StepsDiscussionSpecial thanksReferencesContact Information