Jim Julius SDSU Course Design Institute May 27, 2009
Mar 29, 2015
Jim JuliusSDSU Course Design InstituteMay 27, 2009
Guiding Questions
Why collect formative feedback on course design?
How should one decide what kind of feedback to seek?
What tools are available to collect feedback?
What do I do with the data?
What (and why) are you measuring?
What (and why) are you measuring?
Outcomes: tell you what you got, not how or why
Inputs Processes
Seeking continuous improvement Approaching course design from
an inquiry mindset
Outcomes
Satisfaction Retention Success Achievement External proficiencies Real-world performance
Inputs
Learner characteristics Context Design Learning resources Faculty development
Processes
Pedagogies Presentation media Assignments/assessments Student use of technologies Community of Inquiry model
(social, cognitive, teaching presence) Interactions
(content, peers, instructor, technology itself)
Community of Inquiry Model
CoI - Interactions
Narrowing Your Inquiry Do you want to evaluate your course
according to “best practices”, i.e. standard course design quality criteria?
Do you want to know more about your learners in general: needs, preferences, motivation, satisfaction?
Do you want to focus on student achievement?
Do you want feedback on your facilitation of learning?
Do you want feedback on specific course elements and/or technologies?
Course Design Quality Criteria Chico rubric Quality Matters Related to Chickering and Gamson’s
“7 Principles for Good Practice in Undergraduate Education” From Indiana University, 2001 From VCU, 2009
Paid tool: Flashlight
Learning about Learners
Direct Indirect
Learning styles surveys Parallel faculty-student
surveys ELI – student and faculty SDSU’s LRS faculty and
student surveys, adapted from LITRE (NC State)
Distance Education Learning Environment faculty and student surveys
National and institutional data (aggregate)
Institutional data (for your learners)
LMS data
Student Achievement
Direct Indirect
Low-stakes: muddiest point, minute papers, clickers, discussion boards
Pre- and post- tests
Grade data Attendance/participation Outcome comparisons
(Different technology/pedagogy and same outcome, or Same technology/pedagogy and different outcomes)
Teacher Behaviors/Overall Direct Indirect
Community of Inquiry Survey
Small Group Analysis Mid-semester surveys End of course
evaluations Assessing online
facilitation Paid: IDEA survey of
student ratings of instruction
Observation Protocols
Course Elements
Direct Indirect
Student Assessment of Learning Gains: SALG
Clicker opinions survey
Examine usage data from Blackboard
Data from M. Laumakis
pICT fellow in 2005 Began teaching parallel 500-student
sections of PSYCH 101 in 2006, one traditional and one hybrid
First fully online PSYCH 101, Summer 2008
Evaluating the Face-to-Face Class Evaluated Fall 2005 innovations via the
Student Assessment of Learning Gains (SALG)
How much did the following aspects of the class help your learning?
Rated from 1 (no help) to 5 (great help)
Evaluating the Face-to-Face Class What did the data show?
Question MWF Section
TTH Section
ConceptCheck Questions 4.1 4.1
Discussion Boards 2.9 3.1
19
Evaluation Findings: IDEA Diagnostic Survey
20
Evaluation Findings: IDEA Diagnostic Survey
Fall 2006 Blended
Fall 2006 Traditional
Spring 2007
Blended
Spring 2007
Traditional
Progress on objectives
70 73 77 77
Excellent teacher
65 68 69 68
Excellent course
62 72 73 71
Note: Top 10% = 63 or more
21
Evaluation Findings:Departmental Course Evaluations
22
Evaluation Findings: Course GradesFall 2007
Fall 2007 Course Grades
12.8
15
34.6
35.8
3.9
15
12.1
33.1
31
8.9
0 10 20 30 40
F
D
C
B
A
Gra
de
% in Category
Blended
Traditional
Clicker Data: Spring 2007
Question % Agree or Strongly
Agree
Class clicker usage makes me more likely to attend class. 93%
Class clicker usage helps me to feel more involved in class. 84%
Class clicker usage makes it more likely for me to respond to a question from the professor.
91%
I understand why my professor is using clickers in this course. 90%
My professor asks clicker questions which are important to my learning.
90%
Summer 2008 Fully Online: SALG Data How much did the following aspects
of the class help your learning? Rated from 1 (no help) to 5 (great
help)
Summer 2008 Fully Online: SALG Data
Question Summer 2008 Online
Taking the test online 4.27
Discussion Forums 3.00
Introduction e-mail that explained the basics of the course
4.50
SALG Data over time
Question Fall 2007
Blended
Fall 2007
F2F
Spring 2008
Blended
Spring 2008 F2F
Summer 2008
Online
Questions, answers, and discussions in class
3.96 4.04 4.10 4.01 4.36
Live online class sessions
3.39 4.20 4.15
Archives of live online class sessions
4.15 4.50 4.44
Quality of contact with the teacher
3.41 3.48 3.94 3.90 4.26
Working with peers outside of class/online
3.12 3.22 3.31 3.39 3.82
Summer 2008: Community of Inquiry Survey Statements rated from 1 (strongly
disagree) to 5 (strongly agree) Based on the Community of Inquiry
framework’s three elements:1. Social Presence2. Cognitive Presence3. Teaching Presence
Summer 2008:Community of Inquiry SurveyCoI Dimension Student Ratings
Social Presence 3.94Affective Expression 3.56Open Communication 4.29Group Cohesion 3.97
Cognitive Presence 3.96Triggering Event 3.91Exploration 3.73Integration 4.09Resolution 4.10
Teaching Presence 4.38Design and Organization 4.50Facilitation 4.38Direct Instruction 4.23
So, what would you like to further explore?