Top Banner
Data-Driven Learning Strategy Jessie Chuang Classroom Aid Inc.
58

Data-Driven Learning Strategy

Oct 19, 2014

Download

Education

Education must capitalize on the trend within technology toward big data. New types of data are becoming available. From evidence approaches to xAPI and the whole Training and Learning Architecture(TLA) big data is the foundation of all.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Data-Driven Learning Strategy

Data-Driven Learning Strategy

Jessie ChuangClassroom Aid Inc.

Page 2: Data-Driven Learning Strategy

Data is

the only hard evidence of what works in education,

and

the compass for personalized learning journey.

Page 3: Data-Driven Learning Strategy

Data, Data, Data

Amazon: data driven engine accounts for 1/3 of all sales

Netflix: recommendation engine accounts for 3/4 of all new orders

How about learning and training segment?

Page 4: Data-Driven Learning Strategy

Evidence Framework A report by U.S. Dep. of Ed.

- discusses the promise of sophisticated digital learning systems for collecting and analyzing very large amounts of fine-grained data (“big data”) as users interact with the systems.

- describes an iterative R&D process, with rapid design cycles and built-in feedback loops—one familiar in industry but less so in education (however, the report provides numerous examples of applications in education).

Page 5: Data-Driven Learning Strategy

Two critical steps - 1● Education must capitalize on the trend within

technology toward big data.

● New types of data are becoming available. ● broad, static categories(demographic, end-of-year grades and test

scores) => more dynamic, fine-grained data in the process of learning

● data in a single context => linked across different parts of learning

● collected through different methodologies and reported in isolation =>

aggregating evidences from different data sources

Page 6: Data-Driven Learning Strategy

Two critical steps - 2● A revitalized framework for analyzing and using evidence

○ Gold standard evidence - slow and expensive, poor matches to the

rapid pace of digital development practices

○ An evidence framework should help educational stakeholders align their

methods of obtaining evidence with their goals, the risks involved, the

level of confidence needed, and the resources available.

Confidence

Risk

The need of evidencebefore implementation

collection of data should be an ongoing task

Page 7: Data-Driven Learning Strategy

Challenges addressed by evidence approaches

● Making sure learning resources promote deeper learning● Building adaptive learning systems that supports

personalized learning● Combining data to create support systems more

responsive to student needs● Improving the content and process of assessment with

technology● Finding appropriate learning resources and making

informed choices

Page 8: Data-Driven Learning Strategy

Evidence Approaches w/i Great Potential1. Educational data mining and learning analytics applied to data gathered from digital

learning systems implemented at scale

2. Rapid A/B testing conducted with large numbers of users within digital learning systems

(example: Khan Academy)

3. Design-based implementation research (DBIR) supported by data gathered from digital

learning systems (collaborative design from developers, researcher, teachers)

4. Large datasets of different types from multiple sources, combined and shared across

projects and organizations

5. Technology-supported evidence-centered design (ECD) of measures of student learning

6. Data gathered from users about a learning resource, how users have used it and their

experiences using it. => Example: enable “Rapid Prototyping” by making early version of a

product freely available online, collect data and mine data for insights; ratings and reviews

from users …...

Page 9: Data-Driven Learning Strategy

What Works Clearinghouse (by U.S. Department of Education)- “The high volume of research on different programs, products, practices, and policies in education can make it difficult to interpret and apply the results. We review the research. Then, by focusing on the results from high-quality research, we try to answer the question “What works in education?”

- The goal is to provide educators with the information they need to make evidence-based decisions.

Page 10: Data-Driven Learning Strategy

iZone - Researchers and Schools collaborate on an Education Test Bed● Partnership from 250 schools, Research Alliance (based at NYU), EdSurge,

ChallengePost, and IDEO● Supporting from U.S. Dep. of Education, Bill&Melinda Gates Foundation, New

York City Dep. of Education● To support developers in rapidly developing and testing selected technology

based instructional supports and featuring test results on EdSurge● To address unmet needs identified by a diverse group of school stakeholders● Launch prize competitions for developers, winners will be invited to beta-test

products in iZone classrooms● Researcher have access to resources necessary to a given study; test bed data

are available for secondary analysis to continue relevant study (also to develop analytic capability and let good algorithms bubble up)

● The collaboration will produce Consumer Reports-style guide for learning tech.● Silicon Valley also initiated an iZone to create a similar ecosystem

Page 11: Data-Driven Learning Strategy
Page 12: Data-Driven Learning Strategy

collegestats.org

Page 13: Data-Driven Learning Strategy

Learning Analytics Applications ● Learner modeling: model a learner’s knowledge, behaviors, motivation,

experience…● Learner profiling: cluster users into similar groups● Domain modeling: decompose content into components and sequences● Effectiveness: test learning principles, pedagogies etc.● Trend analysis: understand changes over time● Recommendation and adaptation: suggest resources and actions to learners

or instructors, adapt systems to learners...

Page 14: Data-Driven Learning Strategy

Research examplesCombine observation data with click streams to detect boredom, frustration, gaming the system…

● Off-task behaviors patterns can be identified like completing module too fast or too slow …

● Evidences of gaming the systems like systematic and rapid incorrect answers, systematic guessing …

Experience API, xAPI, can give us more because …

Page 15: Data-Driven Learning Strategy

Learner

Learning DesignerData Analyst

StandardsInteroperabilityAnalysis (data mining)

Learning theoriesPsychologyDesign (experience design, UI)Subject Matter Expertise

*Motivator*Navigator (content brokering)*e-Portfolio(or digital badge backpack)

inform iterateco-design

integrate

(developers are constructors)

*Connecting Learner Community*Learner Attribute (style)

*Narrator (learner created context, reflection)

Gamified layeron data from across platforms

Big Data

[Data-Driven Learning Strategy]

Page 16: Data-Driven Learning Strategy

Intelligent learning environment is to match a learner with the right content at the right time.

Page 17: Data-Driven Learning Strategy

About Learner Profiling

Image credit: DARPA

Page 18: Data-Driven Learning Strategy

Recording Learning EventsLearning happens in interactions:

openclipart.org

Instructors, Peers, Experts….

Contents: Courses, Books, Web pages, Games, AR ….

Activities(making, exercises, researching, online, offline ….)

Learner

Page 19: Data-Driven Learning Strategy

Recording Learning Events

Social LearningGroup Learning

image credit: Search Engine People Blog

Page 20: Data-Driven Learning Strategy

Learning

is

different

Page 21: Data-Driven Learning Strategy
Page 22: Data-Driven Learning Strategy

image credit: “A New Architecture for Learning”, published by Educause

Page 23: Data-Driven Learning Strategy

xAPI tracking all kinds of learning experiences

Page 24: Data-Driven Learning Strategy

The xAPI specification has two primary parts

1. defines the syntax of the xAPI data format a. the vocabularies should be community-drivenb. all activities and context can be trackedc. any enabled application/device can send statements

2. defines the characteristics of “learning record stores” (LRS) - a crucial component of xAPIa. data can be exchanged between LRSs (set free from LMS)b. learner can have life-long “personal learning locker”c. LRSs need to validate xAPI statements

Page 25: Data-Driven Learning Strategy

Course Webpage Game Simulator

CoachingSocial LearningProjectMobile

Apps

LRS Learning data is sent to LRS

Other activities

Page 26: Data-Driven Learning Strategy

LMS LRS

LRS

ReportingTool

Learning records can be delivered to LMSs,LRSs or Reporting Tools.

Page 27: Data-Driven Learning Strategy

Recording Learning Events

This is only the basic idea. Crafting the statements with more context and related information is necessary to support analytics and reporting.

ActivityStreams: stream of activity data statements, borrowed from social analytics

Page 28: Data-Driven Learning Strategy

11 Attributes in xAPI Data Format★ Unique Identifier

★ Actor (required)

★ Verb (required)

★ Object (required)

★ Result

★ Context

★ Timestamp

★ Stored (internal recording

timestamp)

★ Authority

★ (Protocol) Version

★ Attachments

All information in XAPI statements can

be separated into :

● meta-data,

● descriptive information, and

● complementary data.

Page 29: Data-Driven Learning Strategy

Syntax-1

Actor:

Agent (= persona) or group (multiple IDs allowed)

Verb:

ID = an IRI(URL) = a specific semantic meaning +

human readable display

Object:

an agent, a group, a statement or an activity(most common)

Page 30: Data-Driven Learning Strategy

Syntax-2

Activity:

ID = an IRI(URL) = with a specific boundary (granularity)

Definition:

Name

Description

objectType

Extentions (useful to customize reporting)

Page 31: Data-Driven Learning Strategy
Page 32: Data-Driven Learning Strategy

Syntax-3

Result:

Score

Success

Duration

Completion

Response (learner’s response to the experience)

Extentions

Page 33: Data-Driven Learning Strategy

Syntax-4

Context:

Registration (differentiate multiple attempts)

Instructor

Team

ContextActivities (parent, grouping, category, other - like related lesson)

Revision

Platform

Language

Statement (refer to one other statement for a whole experience)

Extentions

Page 34: Data-Driven Learning Strategy

Context, Context, ContextThe XAPI differentiates between the core context and the wider context.

The core context includes the instructor(s), the direct peers involved in an activity (team), the learning environment (platform), the language that was used in the performance, and a framing statement for an activity (e.g., the course that relates to the activity).

The extended context includes a set of data-records about the wider context of a learning activity. This wider context is not explicitly specified and can include the location of the learner, the wider (social) relations, the duration of an activity, environmental factors (e.g., temperature or noise level) etc. The format and the content of the wider context is specific to the AP and is not subject to the interoperability of the data format.

Page 35: Data-Driven Learning Strategy

Ask the right questions!

Page 36: Data-Driven Learning Strategy

LRSAn LRS is defined by two interfaces:

● Statement interface (statement API)● Document interface - this interface handles three types of documents

● State interface (state API)● Activity profile interface (activity API)● Agent profile interface (agent API)

The LRS is responsible for

1. validating that the system sending data is authorized,2. checking that the data being sent is xAPI-compliant,3. storing the data properly,4. making that data available to any other authorized system or activity provider

when asked.

Page 37: Data-Driven Learning Strategy

Advanced Applications of xAPI - Data Transfer based on RESTful HTTP w/i LRS● Agent Profile API

○ personal info., learner profile and modeling, user settings, learning journal, career plan & goal

○ an integrated picture of a learner activities across systems and devices with multiple identities

● Activity Profile API (for activity provider)○ interactions between learners (collaboration, social or competition)○ learning planning tool (access to or update the LRS internal definition of

a given activity id, even before the activity sends any statement)● State API

○ persist state across devices● Authentication services, querying services, visualization services, and

personal data services are some examples.

Page 38: Data-Driven Learning Strategy

xAPI tracking is...

Semantic

Contextualized

From any device and sensor

LRS frees the learning data so they can be put together, analyzed, modeled, reused, carried with learners and accumulated life-long

Page 39: Data-Driven Learning Strategy

xAPI + Open Badges => Learner ePortfolio

Both are representing learner data by exploiting HTTP, JSON, and REST - simple, lightweight method that lowers entry barrier for developers.

Together, they offer a new way to think about constructing interoperable learner model data!

Page 40: Data-Driven Learning Strategy

JSON vs XML● JSON is bandwidth-non-intensive● JSON is better adapted (than XML) to devices with limited capabilities such as

smart things● JSON possesses a very limited set of data types. Restricting itself to primitive

data types makes it deeply and immediately interoperable with pretty much any programming language that exists out there.

● JSON is a better data exchange format, XML is a better document exchange format

● JSON is a preferred format in NoSQL database - data not seamlessly conform to a columnar/relational model. Since JSON objects may be heterogeneous in terms of number and types of fields, this allows for tremendous flexibility in storing and retrieving objects as compared to relational databases.

(Should you like to know more about JSON, here is a very simple interactive tutorial. )

Page 41: Data-Driven Learning Strategy

Bootstrapping Learner Model Data● Making data contribution as easy as possible● Bootstrapping heterogeneous learner model

data ○ the raw experience data stored in a LRS referenced by an Open Badge,

could be analyzed to perform an application’s own interpretations of that evidence, as suggested by Carmagnola, Cena, and Gena (2011)

○ Guo & Greer, 2007;

○ Tiroshi, Kuflik, Kay & Kummerfeld, 2011 ) summarized various boostrapping methods

○ ADL developed an open source project/resource called “lr-data”(Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)

Page 42: Data-Driven Learning Strategy

Sharing and Analysis of DataIn the case of sharing and making sense of learner data, a similar model can be followed where any contributing application (e.g., an ITS or game) could easily publish activity data to a LRS, Open Badges to a learner’s badge backpack, or other learner model data to their shared learner profile as JSON over a RESTful HTTP connection.

The LRS, badge backpack, or learner profile can exist anywhere (e.g., an employer organization, a commercial provider, or even self-hosted by a learner) – all the contributing applications need to know are the URLs and the learners credentials.

If a learning application wants to make sense of all this data, it too just needs the URLs and the learner’s credentials to get started. The learner could set permissions concerning what applications can access their data. (Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)

Page 43: Data-Driven Learning Strategy

TLA Services

TLA will also include services for managing learner profiles. Open Badges can be referenced by learner profiles, which will likely contain other learner data such as goals, reflection, etc.

The TLA will also include services for creating and accessing competency definitions to serve as a common way to reference educational standards, learning objectives, and competency definitions through web APIs(Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)

Page 44: Data-Driven Learning Strategy

The whole picture =Training and Learning Architecture(TLA)

● ePortfolio● Learner modeling● Machine readable

● Competency standards

● Knowledge map● Standard

alignment

● xAPI COP● Common

vocabulary● Learning

Design

● Sharing of metadata & paradata (LR)

● Re-usability● Semantic analysis

Page 45: Data-Driven Learning Strategy
Page 46: Data-Driven Learning Strategy

A taxonomy of education standards

Redd, Brandt

Page 47: Data-Driven Learning Strategy

Education Standards for Interoperability

❏ Academic standards ❏ Data Standards (consistent definition)

❏ Student data❏ Data Dictionary (data element definition)❏ Logical Data Model (entity definition)

❏ Educator data❏ Content data (metadata, paradata)

❏ xAPI vocabulary/activity profile

CEDS

Page 48: Data-Driven Learning Strategy

Major Learner Data Categories

● Educational records● Competencies(skills, knowledge, abilities,

outcomes…) and domain learning objectives● Data in affective, motivational(disposition) and

social dimensions● Data for learning style modeling● Data for pedagogical purposes

Page 49: Data-Driven Learning Strategy
Page 50: Data-Driven Learning Strategy

About Content

Image credit: DARPA

Page 51: Data-Driven Learning Strategy

Learning Registry - Social Network sharing metadata and paradata of Learning Resources

Make contributing as easy as possible !

Page 52: Data-Driven Learning Strategy

ParadataWhile learning analytics generally refers to analysis of data about learners, paradata refers to data about learning resources.

Paradata can also record contextual information by linking resources with educational standards and curricula, pedagogic approaches and methodologies.

Paradata can be regarded as an extended and altered version of JSON ActivityStreams.

Paradata differs from ActivityStreams in that it enables complex aggregations of activities to be recorded; e.g. High school English teachers taught using this resource 15 times during the month of May 2011.

Page 53: Data-Driven Learning Strategy

Content StrategyxAPI modernize the SCORM runtime, what about a modernized content strategy?

IEEE LTSC : the answer is HTML5 + EPUB 3

W3C standards define an Open Web Platform for application development that has the unprecedented potential to enable developers to build rich interactive experiences, powered by vast data stores, that are available on any device.

=> HTML5 is the cornerstone.

In addition to the classic “Web of documents” W3C is helping to build a technology stack to support a “Web of data,” the sort of data you find in databases. The ultimate goal of the Web of data is to enable computers to do more useful work and to develop systems that can support trusted interactions over the network. The term “Semantic Web” refers to W3C’s vision of the Web of linked data. Semantic Web technologies enable people to create data stores on the Web, build vocabularies,

and write rules for handling data. Linked data are empowered by technologies such as RDF, SPARQL, OWL, and SKOS.

Page 54: Data-Driven Learning Strategy

RUSSEL - Re-Usability Support System for eLearning

The RUSSEL project includes:

● a novel user interface (UI), ● drag-and-drop SCORM disaggregation, ● a framework for automated metadata generation, ● a new approach to supporting instructional design best

practices while remixing existing materials, ● paradata capture from use of content in instructional

designs.

Page 55: Data-Driven Learning Strategy

The Whole Picture

Page 56: Data-Driven Learning Strategy

xAPI and Open Ecosystem Picture- Based on standardized tracking and communicating language - xAPI, and the standardization for learner's data

- Modularized ecosystem, new apps could always integrate in and with each other (through standard web service call), a product could have several modular components shown here

Knowledge Map / Topic graph / Competency network

Learning Objects / Resources / Tools / Widgets

Adaptive engine / Intervention engine / Recommendation algorithm

Analytics / Data mining tools

User Interface Dashboard / Reporting / Visualization

Learning Community Certifications

Page 57: Data-Driven Learning Strategy

Next Gen Learning Environment

from ADL Andy Johnson

Page 58: Data-Driven Learning Strategy

Building “Community of Practice(COP)” is the crucial

foundation of a standard, and the culture of using data for making instructional decisions.

xAPI is an open and flexible framework standard for tracking learning experiences, it needs further

“community-driven” rules - top-down rules won’t work - for different domains to

ensure data quality and interoperability.