Strategy for Educational Improvement

Continuous Improvement in Teaching and Learning

Candace Thille, Director OLICarnegie Mellonwww.cmu.edu/oli

Strategy for Educational Improvement

Goal Directed Practice & Targeted

Feedback are Critical to Learning

Synthesize and Apply Skills and

Concepts to Solve

Authentic Problems

“The Killer App” feedback loops for continuous improvement

OLI Review:• Apply learning science research and scientific method

to course development, implementation and evaluation

• Environments are developed by teams of content experts (and novices), learning scientists, HCI, software engineers

• Feedback loops for continuous improvement

What Difference Does This Make?

Accelerated Learning Results

7

OLI students completed course in half the time with half the number of in-person course meeting

OLI students showed significantly greater learning gains (on the national standard “CAOS” test for statistics knowledge) and similar exam scores

No significant difference between OLI and traditional students in follow-up measures given 1+ semesters later

“The Killer App” feedback loops for continuous improvement

Learning Curve Analysis on Stoichiometry Data

www.learnlab.org

PSLC: www.learnlab.orgLearnLab: Like a research

hospital for learning• Tech enhanced courses

– Intelligent tutors, virtual lab simulations, language dialogs, multimedia, …

• Data on what works best

Researchers Schools

Chemistry virtual labChemistry virtual lab (David Yaron, Candace Thille)(David Yaron, Candace Thille)

Physics intelligent tutorPhysics intelligent tutor(VanLehn, van de Sande)(VanLehn, van de Sande)

REAP vocabulary tutorREAP vocabulary tutor (Maxine Eskenazi+)(Maxine Eskenazi+)

Researchers Schools

Learn Lab

Carnegie

Learning

Cognitive

Tutor:

Carnegie Learning Cognitive Tutor:More than 2,600 Schools & 500,000 students

www.carnegielearning.com

Many studies of Cognitive Tutor Algebra effectiveness

• 11 study reports available – From 1994 to present, 11 different districts– More than 8000 students in these studies– Most run independently of Carnegie

• Significant positive results in 9 of 11– Two found no difference

• All study results available at carnegielearning.com

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8.

www.carnegielearning.com

Field Studies: 1000s of grade 6-10 math students in urban district outside Boston

Instant reports to teachers

Students get problems online

Students get instant feedback

www.assistments.org

Assistments Results• Better at predicting student knowledge by considering # of attempts, amount

of tutoring assistance and how fast a student solves a problem. Feng, M., Heffernan, N.T., &

Koedinger, K.R. (2009). Addressing the assessment challenge in an Intelligent Tutoring System that tutors as it assesses. The Journal of User Modeling and User-Adapted Interaction. Vol 19: p243-266.

• Control for time and ASSISTments is a better assessor than traditional paper and pencil. Feng, M. & Heffernan, N. (2010) Can We Get Better Assessment From A Tutoring System Compared to Traditional Paper Testing? Can We Have Our

Cake (Better Assessment) and Eat It too (Student Learning During the Test)? Educational Data Mining 2010.

www.assistments.org

Virtual Assessment

Products of Inquiry- Create conclusions and select evidence

Multiple Forms of Complex MeasuresProcesses of Inquiry- Gather data and interview people

Harvard School of Education, Chris Dedehttp://virtualassessment.org

Links to other views

A noteA build-on note

An opened note

Knowledge building: Engage

students in creative

knowledge work, with literacy as a

by-product.Institute for Knowledge Innovation and Technology, OISE/UT

Marlene Scardamaliawww.ikit.org

Legend A red-circled node = a contributor

A blue-squared node = a view

Tie= strength of knowledge-building (note contribution from a contributor to a view) as measured by the unit of analysis:

Weak (periphery)

Collaboration

Strong(Core)

Collaboration

Weak (periphery)Knowledge-interaction

Strong (core)Knowledge-interaction

Institute for Knowledge Innovation and Technology, OISE/UTMarlene Scardamalia

www.ikit.org

Pasteur’s Quadrant• Stokes argues basic/applied goals need not trade off

Low Emphasis on Applied Work

High Emphasis on Applied Work

High Emphasis on Basic Science

How to translate to the real world?

Low Emphasis on Basic Science

What principle can be derived?

X

URLs for more information:• Open Learning Initiative (Carnegie Mellon):

www.cmu.edu/oli• Pittsburgh Science of Learning Center (Carnegie Mellon):

www.learnlab.org• Algebra & Geometry Cognitive Tutors (Carnegie Learning):

www.carnegielearning.com• Assistments (Worcester Polytechnic Institute):

www.assistments.org• Virtual Assessments (Harvard):

http://virtualassessment.org• Knowledge Building (University of Toronto):

www.ikit.org

“Improvement in Post Secondary Education will require converting teaching from a ‘solo sport’ to a community based research activity.”

—Herbert Simon

Open Learning Initiative: www.cmu.edu/oliCandace Thille: [email protected]