Shute, V. J., Levy, R., Baker, R., Zapata, D., & Beck, J. (2009). Assessment and learning in intelligent educational systems: A peek into the future.

Shute, V. J., Levy, R., Baker, R., Zapata, D., & Beck, J. (2009). Assessment and learning in intelligent educational systems: A peek into the future. In S. D. Craig, & D. Dicheva (Eds.), Proceedings of the Artificial Intelligence and Education (AIED '09) Workshop on Intelligent Educational Games (pp. 99 – 109). Brighton, UK. http://myweb.fsu.edu/vshute/pdf/peek.pdf

Education is a social process. Education is growth. Education is not a preparation for life; education is life itself. –Dewey

Education, Writ LargeEducation, Writ Large

The world has changed a lot in the past 100 years. Education has not.

Classroom photo, 1910. Classroom photo, 2010.

Why Education Needs to ChangeWhy Education Needs to Change

Why Assessment Needs to ChangeWhy Assessment Needs to Change

Learning & Assessment DefinitionsLearning & Assessment Definitions

CurrentCurrent

Assessment

Assessment

PracticesPractices

Student

Student

Learning

Learning

Teach. Stop. Test. Repeat. Teach. Stop. Test. Repeat.

AssessmentAssessment

Collaborative LearningCollaborative Learning

Fuse Learning & AssessmentFuse Learning & Assessment

ThemesThemes

Comprehensive Learner Models

Seamless & Ubiquitous Assessment

Assessment Info for Decision Making

Aligned along 3 dimensions: vertically, horizontally, and temporally.

Theme 1 Theme 2 Theme 3

On June 26, 1974, the first product with a bar code was scanned at a check-out counter.

It was a 10-pack of Wrigley's Juicy Fruit chewing gum.

Today, the pack of gum is on display at the Smithsonian Institution's National Museum of American History.

19741974

Setting Up A Metaphor for Theme 1Setting Up A Metaphor for Theme 1

Before 1974, businesses had to close down once a year to take inventory.

Barcodes enabled access to a continuous stream of information to monitor inventory.

Now businesses continue without interruption, supporting real-time inventory management.

Theme 1 MetaphorTheme 1 Metaphor(via Pellegrino, Chudowsky, & Glaser, 2001)

Business

Schools shouldn’t have to interrupt normal instruction to give external tests.

Instead, assessment should be continual & invisible, supporting real-time, JIT learning.

This is the first of 3 themes envisioned, 20 years hence (seamless & ubiquitous assessment).

Theme 1 MetaphorTheme 1 Metaphor(via Pellegrino, Chudowsky, & Glaser, 2001)

School

Advances in measurement let us administer evidence-based assessments to: Extract ongoing information from a learner Make accurate inferences of competencies React in immediate and helpful ways.

Accomplished via automated scoring and machine-based reasoning techniques.

Seamlessly woven into the fabric of the learning environment.

Stealth AssessmentStealth Assessment

Stealth assessment can be used to improve learning: Gathering evidence for accurate

estimates of competencies

Info on competencies used by teachers, the system/game, and/or students

Current estimates of competency levels integrated into the game and displayed as progress indicators.

This elevates valued competencies to the same level as health and weapons!

How Does It Work?How Does It Work?

Theme 1: SummaryTheme 1: Summary“Seamless and Ubiquitous Assessment”

IdeaIdea: : A continuous process fusing assessment and A continuous process fusing assessment and learning. learning.

SeamlessSeamless: : Removal of false boundaries between Removal of false boundaries between learning and assessment. learning and assessment.

UbiquitousUbiquitous: : Constant nature and need to Constant nature and need to feedback back results of assessment into learning. feedback back results of assessment into learning.

Current State

Theme 2: Comprehensive ModelsTheme 2: Comprehensive Models

Goal State

Theme 2: Comprehensive ModelsTheme 2: Comprehensive Models

Research needed to identify, model, and assess 21st century competencies as well as other attributes, like affective variables, values, dispositions, etc.

These will comprise rich evolving profiles representing a synopsis of what’s known, what can be done, what’s believed, preferred/valued, and felt.

ECD focuses on the evidence of competencies. It provides a way to reason about assessment design, and

a way to reason about learner performance. As an assessment framework, it can support both Theme 1 (seamless/ubiquitous

assessment) and Theme 2 (comprehensive models).

Competency Model

What do you want to say about the What do you want to say about the person?person?

Evidence ModelWhat observations would provide What observations would provide

best evidence for what you want to best evidence for what you want to say?say?

Task/Action Model Model

What kinds of tasks let you make What kinds of tasks let you make the necessary observations?the necessary observations?

Assessment DesignAssessment Design

For more information about ECD, please see ECD for Dummies.

Design and DiagnosisDesign and Diagnosis

Example Conceptual ModelExample Conceptual Model

Example Computational ModelExample Computational Model

Summary of ECDSummary of ECD

ECD should be used as the framework for new assessments because it can:

Yield valid assessments for different purposes

Provide accurate estimates of complex competencies and dynamic performances

Aggregate information from various sources

Afford transparency to stakeholders via evidentiary reasoning to support claims.

New competencies need to be identified, modeled, assessed and supported.

Assessments need to be valid and reliable to meet our educational needs (e.g., ECD and stealth assessment).

Immersive games are fun, and enable learning within complex and relevant environments.

Wrapping It UpWrapping It Up

Themes over Themes over timetime

Theme 20 years…

Seamless assessment

Learning-related programs/devices that students use (games, learning environments, applications) employ seamless assessment.

Ubiquitous assessment

Different platforms are interoperable re: delivery, analysis, and support of key comps, relevant data, and inferences linking them altogether.

Comprehensive Model: Identifying key comps

Wide range of important skills and other attributes identified and justified (e.g., causal reasoning, collaboration, critical thinking, reflexivity, self-regulation, adaptability, focused attention, systems thinking, interdisciplinary thinking, problem solving, compassion, etc.)

Comprehensive Model: Modeling

Robust (conceptual & computational) models of all comps, as well as their interactions. Vertical scales for key competencies (e.g., trajectories).

Comprehensive Model: Assessing

Most assts intimately combined within learning experiences thus rarely need summative asst (e.g., only for auditing purposes). Assessment data accumulated into lifelong portfolios/profiles.

Th

eme

1T

hem

e 2

Fusing assessment and learning

Understanding full complement of characteristics involved with learning

Rendering assessments useful to all parties

Major Funding TargetsMajor Funding Targets

Thank you!Thank you!

Questions?Questions?

Email: Email: vshute@fsu.eduvshute@fsu.eduWebsiteWebsite: http://myweb.fsu.edu/vshute: http://myweb.fsu.edu/vshute

Curiosity

Critical Thinking

Collaboration

Grit

Systems Thinking

Empathy

Creativity

Go Back

BarriersBarriers(or Opportunities!)(or Opportunities!)