Nyc open data presentation by Knewton Data Scientist, Chaitu Ekanadham,

NYC Open Data, March 18, 2015

Chaitanya Ekanadham

Representing

learning

experiences

personalize learning experiences

for students around the world

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

Knewton’s mission

content inventory

student goals

interaction data

recommendations

analytics

content insights

provide high quality adaptive

education to everyone in the world

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2014 KNEWTON, INC.

Knewton’s mission

leverage past learning experiences

to improve future learning

experiences

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

data science team’s mission

when a learner is presented some content and

acquires knowledge as a result

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2014 KNEWTON, INC.

what’s a “learning experience”?

learner

s

content

alice

sharo

n

... ...

video lecture on limits

derivatives definition

integrals quiz

limits exercise

related rates word problem

bob

pastfuture

leverage past learning

experiences to improve future

learning experiences

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

past learning

experiences

modelslearner

representation

content

representation

models

past learning

experiences

learner

representation

proficiency

learning speed

time investment

preferred mode

content

representation

similarities

difficulty

length

effectiveness

leverage past learning experiences

to improve future learning

experiences

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

modelslearner

representation

content

representation

hypothetical

future learning

experience

score

path

dependent

effects

difficult to

quantify

do not have

final grades

leverage past learning experiences to

improve future learning experiences

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

unique challenges

content

representation

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

video lecture on

limits

limits problem set

derivatives

definition

common derivatives

differentiation

exercises

integrals as limits of

Riemann sums

fundamental

theorem of calculus

summation notation

reference

not scalable!

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

video lecture on

limits

limits problem set

derivatives

definition

common derivatives

differentiation

exercises

integrals as limits of

Riemann sums

fundamental

theorem of calculus

summation notation

reference

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

video lecture on limits

limits problem set

derivatives definition

common derivatives

differentiation exercises

integrals as limits of

Riemann sums

fundamental theorem of

calculus

summation notation

reference

instruction

assessment

both

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

measuring

learner ability

modelslearner

representation

content

representation

hypothetical

future

learning

experience

score

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

learner

proficiency

item

difficulty

item

discrimination

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

learner

s

assessment

content

alice

sharon

...

...

limits exercise

integrals quiz

limits challenge problem

related rates word problem

bob

differentiation problem

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

learner

s

assessment

content

alice

sharon

...

...

limits exercise

differentiation problem

integrals quiz

limits challenge problem

related rates word problem

bob

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

learner

s

alice

sharon

...

bob

assessment

contentlimits exercise

differentiation problem

integrals quiz

limits challenge problem

related rates word problem

...

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

learner

s

alice

sharon

...

bob

assessment

contentlimits exercise

differentiation problem

integrals quiz

limits challenge problem

related rates word problem

...

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

? ?

likelihood

prior

posterior

? ?

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

likelihood

prior

?

?

?

?

?

?? ?

?

t=3? ? t=2

posterior

? ? t=1

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

differentiation

related rates

word problem

3D shape volume

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

assessing multiple concepts

compensatory likelihood: knowing 1 concept is good enough

differentiation

related rates

word problem

3D shape volume

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

assessing multiple concepts

non-compensatory likelihood: have to know both concepts

differentiation

related rates

word problem

3D shape volume

learner

timing patterns

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

time

timesession break

response times

histogram of log-taus

with highlighting!

48K students

9.6K modules

1M interactions

histogram of log-taus

with highlighting!

48K students

9.6K modules

1M interactions

histogram of log-taus

with highlighting!

session

breaks

response times

?

48K students

9.6K modules

1M interactions

log time

pro

bab

ility response times

session breaks

normalized response time

no

rma

lized

qu

ittin

g r

ate

normalized response time

no

rma

lized

qu

ittin

g r

ate

high

engagement

normalized response time

no

rma

lized

qu

ittin

g r

ate

low

engagement

high

engagement

normalized response time

no

rma

lized

qu

ittin

g r

ate

non-sticky low

engagement

high

engagement

normalized response time

no

rma

lized

qu

ittin

g r

ate

non-stickyboredom?

low

engagement

high

engagement

normalized response time

no

rma

lized

qu

ittin

g r

ate

non-stickyboredom?

low

engagement

high

engagement

slow

normalized response time

no

rma

lized

qu

ittin

g r

ate

non-stickyboredom?

low

engagement

high

engagement

slowfrustration?

deep thought?

PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

exciting challenges

● offline learning

● learner affinity for pedagogical strategies

● automating content graphing

● student agency

● controlled experiments

● “adaptivity-ready” content design

chaitanya ekanadham

managing data scientist

Knewton, Inc.

[email protected]

Twitter: @knewton

thank you.

appendix