Page 1
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 1/48
Seeking Evidence of Impact for Technology-Based Innovations in
Continually Dynamic Settings
Chuck Dziuban
Patsy Moskal
Research Initiative for Teaching Effectiveness
Center for Distributed Learning
University of Central Florida
Page 2
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 2/48
Changing Baselines and
Landscape Amnesia
Page 3
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 3/48
The Unnatural History of the Sea
Page 4
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 4/48
A Jessica Hagy Inspired Model of
Changing Baselines
“You look like somebody that I used to know”
Ability toremember what itwas like back then
Your real knowledgeof how it is today
Data you can get your hands on
Page 5
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 5/48
A Jessica Hagy Inspired Model of
Success & Withdrawal
Attention spansare short
A little A lot
How often you show success and withdrawal
P e o p l e
s ’ I n t e r e s
t
L e v e l
That isfascinating
Whatever
Page 6
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 6/48
The Evolving Evidence of Impact
Model at UCF
Page 7
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 7/48
UCF Distributed Learning Impact
Evaluation – Iteration (1)
Students Faculty
Success
SatisfactionDemographic
profiles
Retention
Strategies for
success
Reactive behavior
patternsWill
Scholarship of
Teaching and
Learning
Work?
Page 8
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 8/48
UCF Distributed Learning Impact
Evaluation – Iteration (2) (Green is new)
Students Faculty
Success
SatisfactionDemographic
profiles
Retention
Strategies for
success
Reactive behavior
patterns
Online programs
Writing project model
Web 2.0 teaching
Higher order
evaluation models
Student evaluation of
instruction
Theater
Informationfluency
Generational
comparisons
Page 9
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 9/48
UCF Distributed Learning Impact
Evaluation – Iteration (3) (Green is new)
Students Faculty
Success
SatisfactionDemographic
profiles
Retention
Strategies for
success
Reactive behavior
patterns
Generational
comparisons Informationfluency
Chemistry
Psychology
Writing Center
Education
Mathematics
Communications
Access
Page 10
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 10/48
UCF Distributed Learning Impact
Evaluation – Iteration (4) (Green is new)
Students Faculty
Success
SatisfactionDemographic
profiles
Retention
Strategies for
success
Reactive behavior
patterns
Generational
comparisons Informationfluency
Scholarship of
Teaching and
Learning
The faculty
research culturePedagogyAmbivalence
Engagement
Access
Page 11
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 11/48
UCF Distributed Learning Impact Evaluation –
Iteration (5): A Second Generation Research Agenda
Students Faculty
HELPEnter your second generationresearch ideas in the text box
Page 12
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 12/48
The Impact of Online Courses
Page 13
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 13/48
UCF SCH Growth
600000
700000
800000
900000
1000000
1100000
1200000
1300000
1400000
1500000
1600000
02-03 03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 12-13
F2F OTHER VIDEO BLENDED (WEB) ONLINE (WEB)
Page 14
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 14/48
34,999
59.8% 11,304
19.3% 4,435
7.6%
504
0.9%
1,363
2.3%1,485
2.5%1,825
3.1%
802
1.4%
744
1.3%
828
1.4%
209
0.4%
“Live” Main
Campus Students49,852 “Live” Rosen
Campus Students
2,604
Web
Students
18,565
“Live”
Regional
Students5,198
UCF Fall 2011 Headcount
Page 15
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 15/48
Student Success
Page 16
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 16/48
Student Success
88 88 87 8791
8791 91 90 90
949088 88 88 88 89 88
0
10
2030
40
50
60
70
80
90
100
Spring 10 Sum 10 Fall 10 Spring 11 Sum 11 Fall 11
F2F (n=669,518) Blended (n=60,309) Fully Online (n=157,922)
Page 17
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 17/48
Student Withdrawal
3 2 3 3 2 43 1 3 3 2 45 4 5 4 4 5
0
10
2030
40
50
60
70
80
90
100
Spring 10 Sum 10 Fall 10 Spring 11 Sum 11 Fall 11
F2F (n=749,656) Blended (n=60,958) Fully Online (n=158,031)
Page 18
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 18/48
Student Satisfaction
Page 19
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 19/48
Student satisfaction with fully online
courses
46 45
5 3 1
4347
5 4 20
10
20
30
40
50
60
70
80
90
100
Very Satisfied Satisfied Neither Unsatisfied Very
Unsatisified
P e r c e n t
Fully Online (n=1368) Blended (n=877)
Page 20
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 20/48
Students’ preference for taking future
online courses
69
21
5 4 1
63
21
7 5 30
10
20
30
40
50
60
70
80
90
100
Definitely Probably Not Sure Probably Not Definitely Not
P e r c e n t
Fully Online (n=1373) Blended (n=906)
Page 21
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 21/48
Students’ preference for taking future
blended courses
29 2925
5 2
42 39
116
20
10
20
30
40
50
60
70
80
90
100
Definitely Probably Not Sure Probably Not Definitely Not
P e r c e n t
Fully Online (n=1250) Blended (n=908)
Page 22
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 22/48
Students’ preference for class modalities
34
11 188
12 15
28
12
24
11 11 13
0
10
20
30
40
50
60
70
80
90
100
Fully
Online
Mostly
Online
Equal Mostly F2F Fully F2F No
preference
P e r c e n t
Fully Online (n=1370) Blended (n=907)
Page 23
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 23/48
Faculty Satisfaction
Page 24
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 24/48
Sections previously taught by faculty
23 24
77
14 19
3439
12
15
24
1117
5
11
19
8
10
7
8
2410 5
53
3016+ sections
11-15 sections
6-10 sections
1-5 sections
None
Online
(n=176)
Blended
(n=163)
V/RV
(n=142)
F2F, no Web
(n=160)
F2F, with Web
(n=168)
Page 25
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 25/48
Number of students faculty feel they
can effectively teach
3627 28 34
25
3242 41
3742
1116
13
1918
68 8 5
147 11 8 10
100+ students
76-100 students
51-75 students
31-50 students
<30 students
Online
(n=176)
Blended
(n=162)
V/RV
(n=120)
F2F, no Web
(n=167)
F2F, with Web
(n=170)
Page 26
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 26/48
Faculty satisfaction by modality
Online
(n=153)
Blended
(n=143)
V/RV
(n=72)
F2F, no Web
(n=150)
F2F, with Web
(n=152)
20
41
18
18
31
36
25
1018
57
7
39
45
2511
8
45
38
15
S a t i s f i e d
N e u t r a l o r u n s a t i s f i e d
Very satisfied
Satisfied
Neutral
Unsatisfied
Very
unsatisfied
Page 27
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 27/48
Positive aspects of teaching an
online course (n=196)
48
19 1611 10 8
0
10
2030
40
50
6070
80
90
100
P e r c e n
t
Accessibility
Flexibility Resources Depth
Interaction Organization
Page 28
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 28/48
Less positive aspects of teaching an
online course (n=196)
46
2312 11
3
0
10
2030
40
50
6070
80
90
100
P e r c e n
t
Disengaged
students
Disconnect
with students
Integrity
issues
Technology
issues
Time
consuming
Page 29
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 29/48
What are the students saying
about me?
Evidence that held rock solid.
“This guy is so
boring that my pillowneeds a pillow.”
-Rate My Professors
Page 30
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 30/48
Ambivalence and the Structure of
Political Opinion
Page 31
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 31/48
Facilitation of learning
Communication of ideas
Excellent Very Good Good Fair Poor
Then...
The probability of an overall rating of Excellent = .97 &
The probability of an overall rating of Fair or Poor =.00
If...
A decision rule for the probability of faculty member
receiving an overall rating of Excellent (n=1,280,890)
Respect and concern for students
A comparison of excellent ratings by course
Page 32
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 32/48
A comparison of excellent ratings by course
modality--unadjusted and adjusted for instructors
satisfying the rule (n=1,171,664)
Blended 52 97Fully Online 48 97
F2F 48 97
Blended LC 43 97Lecture Capture 42 97
Course Overall If Rule 1
Modality % Excellent % Excellent
A comparison of excellent ratings by top ten most
Page 33
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 33/48
A comparison of excellent ratings by top ten most
heavily populated departments--unadjusted and adjusted
for instructors satisfying the rule (n= 1,286,890)
Overall If Rule 1Department % Excellent % Excellent
Political Science 56 97
History 56 97
Crim. Justice 55 97
Psychology 53 97
Finance 51 98
Microbiology 50 98English 48 97
Biology 44 97
Mathematics 38 97
A comparison of excellent ratings by college unadjusted
Page 34
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 34/48
A comparison of excellent ratings by college unadjusted
and adjusted for instructors satisfying the rule
(n=1,280,890)
Overall If Rule 1College % Excellent % Excellent
Education 60 98
Nursing 54 98
Health & Public Affairs 53 98
Arts & Humanities 51 97
Optics & Photonics 49 94
Hospitality Management 48 97Medicine 46 98
Sciences 45 97
Business Administration 44 97
Engineering & Comp Sci 39 97
A comparison of excellent ratings by class level
Page 35
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 35/48
A comparison of excellent ratings by class level
unadjusted and adjusted for instructors satisfying the
rule (n=1,171,664)
Freshman 46 97Sophomore 45 97
Junior 48 97
Senior 51 97
Class Overall If Rule 1
Level % Excellent % Excellent
A comparison of excellent ratings by class size decile
Page 36
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 36/48
A comparison of excellent ratings by class size decile
unadjusted and adjusted for instructors satisfying the rule
(n=1,17,664)
Overall If Rule 1Decile % Excellent % Excellent
1 54 97
2 51 97
3 54 98
4 52 98
5 46 97
6 47 977 48 97
8 44 97
9 44 97
10 41 97
Page 37
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 37/48
Principles that Guide Evidence
of Impact
Page 38
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 38/48
There is good, fast, and cheap –
Pick any two
Page 39
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 39/48
Don’t let technology drive the
evaluation or assessment
Page 40
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 40/48
Statistics and measurement are
great but they are not everything
Page 41
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 41/48
Push access and flexibility out as
far as possible
Page 42
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 42/48
A great product makes a huge
difference
Page 43
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 43/48
Just because you can doesn’t
mean you should
Page 44
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 44/48
Progress made is better made in
small steps
Page 45
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 45/48
Simple really is better
Page 46
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 46/48
Decision makers don’t knowwhat you are talking about.
Make it relevant to their world
Page 47
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 47/48
One more chance!
Students Faculty
HELPEnter your second generationresearch ideas in the text box
Page 48
7/29/2019 Seeking Evidence of Impact for Technology-Based Innovations in Continually Dynamic Settings (166251811)
http://slidepdf.com/reader/full/seeking-evidence-of-impact-for-technology-based-innovations-in-continually 48/48
For more information contact:
Dr. Chuck Dziuban
(407) 823-5478
[email protected]
Dr. Patsy Moskal
(407) 823-0283
[email protected]
http://rite.ucf.edu
http://www.if.ucf.edu/