Shute, V. J., Levy, R., Baker, R., Zapata, D., & Beck, J. (2009). Assessment and learning in intelligent educational systems: A peek into the future. In S. D. Craig, & D. Dicheva (Eds.), Proceedings of the Artificial Intelligence and Education (AIED '09) Workshop on Intelligent Educational Games (pp. 99 – 109). Brighton, UK. http://myweb.fsu.edu/vshute/pdf/peek.pdf
Education is a social process. Education is growth. Education is not a preparation for life; education is life itself. –Dewey
Education, Writ LargeEducation, Writ Large
The world has changed a lot in the past 100 years. Education has not.
Classroom photo, 1910. Classroom photo, 2010.
Why Education Needs to ChangeWhy Education Needs to Change
Why Assessment Needs to ChangeWhy Assessment Needs to Change
Learning & Assessment DefinitionsLearning & Assessment Definitions
CurrentCurrent
Assessment
Assessment
PracticesPractices
Student
Student
Learning
Learning
Teach. Stop. Test. Repeat. Teach. Stop. Test. Repeat.
AssessmentAssessment
Collaborative LearningCollaborative Learning
Fuse Learning & AssessmentFuse Learning & Assessment
ThemesThemes
Comprehensive Learner Models
Seamless & Ubiquitous Assessment
Assessment Info for Decision Making
Aligned along 3 dimensions: vertically, horizontally, and temporally.
Theme 1 Theme 2 Theme 3
On June 26, 1974, the first product with a bar code was scanned at a check-out counter.
It was a 10-pack of Wrigley's Juicy Fruit chewing gum.
Today, the pack of gum is on display at the Smithsonian Institution's National Museum of American History.
19741974
Setting Up A Metaphor for Theme 1Setting Up A Metaphor for Theme 1
Before 1974, businesses had to close down once a year to take inventory.
Barcodes enabled access to a continuous stream of information to monitor inventory.
Now businesses continue without interruption, supporting real-time inventory management.
Theme 1 MetaphorTheme 1 Metaphor(via Pellegrino, Chudowsky, & Glaser, 2001)
Business
Schools shouldn’t have to interrupt normal instruction to give external tests.
Instead, assessment should be continual & invisible, supporting real-time, JIT learning.
This is the first of 3 themes envisioned, 20 years hence (seamless & ubiquitous assessment).
Theme 1 MetaphorTheme 1 Metaphor(via Pellegrino, Chudowsky, & Glaser, 2001)
School
Advances in measurement let us administer evidence-based assessments to: Extract ongoing information from a learner Make accurate inferences of competencies React in immediate and helpful ways.
Accomplished via automated scoring and machine-based reasoning techniques.
Seamlessly woven into the fabric of the learning environment.
Stealth AssessmentStealth Assessment
Stealth assessment can be used to improve learning: Gathering evidence for accurate
estimates of competencies
Info on competencies used by teachers, the system/game, and/or students
Current estimates of competency levels integrated into the game and displayed as progress indicators.
This elevates valued competencies to the same level as health and weapons!
How Does It Work?How Does It Work?
Theme 1: SummaryTheme 1: Summary“Seamless and Ubiquitous Assessment”
IdeaIdea: : A continuous process fusing assessment and A continuous process fusing assessment and learning. learning.
SeamlessSeamless: : Removal of false boundaries between Removal of false boundaries between learning and assessment. learning and assessment.
UbiquitousUbiquitous: : Constant nature and need to Constant nature and need to feedback back results of assessment into learning. feedback back results of assessment into learning.
Current State
Theme 2: Comprehensive ModelsTheme 2: Comprehensive Models
Goal State
Theme 2: Comprehensive ModelsTheme 2: Comprehensive Models
Research needed to identify, model, and assess 21st century competencies as well as other attributes, like affective variables, values, dispositions, etc.
These will comprise rich evolving profiles representing a synopsis of what’s known, what can be done, what’s believed, preferred/valued, and felt.
ECD focuses on the evidence of competencies. It provides a way to reason about assessment design, and
a way to reason about learner performance. As an assessment framework, it can support both Theme 1 (seamless/ubiquitous
assessment) and Theme 2 (comprehensive models).
Competency Model
What do you want to say about the What do you want to say about the person?person?
Evidence ModelWhat observations would provide What observations would provide
best evidence for what you want to best evidence for what you want to say?say?
Task/Action Model Model
What kinds of tasks let you make What kinds of tasks let you make the necessary observations?the necessary observations?
Assessment DesignAssessment Design
For more information about ECD, please see ECD for Dummies.
Design and DiagnosisDesign and Diagnosis
Example Conceptual ModelExample Conceptual Model
Example Computational ModelExample Computational Model
Summary of ECDSummary of ECD
ECD should be used as the framework for new assessments because it can:
Yield valid assessments for different purposes
Provide accurate estimates of complex competencies and dynamic performances
Aggregate information from various sources
Afford transparency to stakeholders via evidentiary reasoning to support claims.
New competencies need to be identified, modeled, assessed and supported.
Assessments need to be valid and reliable to meet our educational needs (e.g., ECD and stealth assessment).
Immersive games are fun, and enable learning within complex and relevant environments.
Wrapping It UpWrapping It Up
Themes over Themes over timetime
Theme 20 years…
Seamless assessment
Learning-related programs/devices that students use (games, learning environments, applications) employ seamless assessment.
Ubiquitous assessment
Different platforms are interoperable re: delivery, analysis, and support of key comps, relevant data, and inferences linking them altogether.
Comprehensive Model: Identifying key comps
Wide range of important skills and other attributes identified and justified (e.g., causal reasoning, collaboration, critical thinking, reflexivity, self-regulation, adaptability, focused attention, systems thinking, interdisciplinary thinking, problem solving, compassion, etc.)
Comprehensive Model: Modeling
Robust (conceptual & computational) models of all comps, as well as their interactions. Vertical scales for key competencies (e.g., trajectories).
Comprehensive Model: Assessing
Most assts intimately combined within learning experiences thus rarely need summative asst (e.g., only for auditing purposes). Assessment data accumulated into lifelong portfolios/profiles.
Th
eme
1T
hem
e 2
Fusing assessment and learning
Understanding full complement of characteristics involved with learning
Rendering assessments useful to all parties
Major Funding TargetsMajor Funding Targets
Thank you!Thank you!
Questions?Questions?
Email: Email: [email protected]@fsu.eduWebsiteWebsite: http://myweb.fsu.edu/vshute: http://myweb.fsu.edu/vshute
Curiosity
Critical Thinking
Collaboration
Grit
Systems Thinking
Empathy
Creativity
Go Back
BarriersBarriers(or Opportunities!)(or Opportunities!)