Improving the OER Experience: Enabling Rich Media Notebooks of OER Video and Audio

Improving the OER Experience: Enabling Rich Media Notebooks of

OER Video and Audio

Brandon Muramatsu [email protected] McKinney [email protected]

Peter Wilkins [email protected]

MIT, Office of Educational Innovation and Technology

Citation: Muramatsu, B., McKinney, A., Wilkins, P. (2010). Improving the OER Experience: Enabling Rich Media Notebooks of OER Video and Audio.Presented at OER10: Cambridge, UK, March 23, 2010.

Unless otherwise specified, this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Why are we doing this?

• More & more videos on the Web– Universities recording

course lectures– Students relying upon

Web video for courses

2

MIT OCW 8.01: Professor Lewin puts his life on the line in Lecture 11 by demonstrating his faith

in the Conservation of Mechanical Energy.

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

What are the challenges?

• Search– Volume– Segmented

by Web, Video

3

Google Search for “angular momentum”Performed April 2009

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

What are the challenges?

• Description– Course and

Lecture Title– Summary– Metadata?

4

YouTube, MIT OCW Physics 8.01 - Lecture 20Retrieved August 2009

webcast.berkeley, Physics 8A, 002, Spring 2009

Retrieved August 2009

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

What are the challenges?

• Interaction & Use– Transcripts /

captions• Do they exist?• Cost?

– Full video vs.segments

5

Lewin, W. (1999). Lec 20 | 8.01 Physics I: Classical Mechanics, Fall 1999.Retrieved August 1, 2009 from YouTube Website:

http://www.youtube.com/watch?v=ibePFvo22x4

“GOD!!!51 MINUTES!!i think i'll pass.. “

– slourdas, YouTube

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Search thru the Static

We’re living in a video world…but only have text to use for search…

6

flickr @futureatlas.com

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Why do we need these tools?

• Improve search and retrieval

• Improve user experience

• Captioning for accessibility? With correction?

• Facilitate translation?

7

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

8

B R E A K I N G N E W S

…YouTube announces captions on all videos…News at 11……YouTube announces captions on all videos…News at 11…

Comparing SpokenMedia and YouTube Auto-Caption?

YouTube

• Scale ✔• Research-basis ✔• For all videos (soon)✔

• No transcript/caption export (?)

• YouTube hosted• Accuracy based on

general patterns (?)• No transcript editing (?)

SpokenMedia

• Limited

• Research-basis ✔

• Service by request

• Transcript/caption export available ✔

• Hosted anywhere ✔

• Accuracy based on custom models (soon)✔

• Transcript editing (soon)✔

9

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Developing SpokenMedia…

• What do we have at MIT?– Existing videos & audio, new video– Lecture notes, slides, etc. (descriptive text)– Multiple videos/audio by same lecturer– Diverse topics/disciplines

• Research…

10

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Enabling Research

• Speech recognition research– Automated transcription of lectures

• Why lectures?– Conversational, spontaneous, starts/stops– Different from broadcast news, other types

of speech recognition– Specialized vocabularies

11

James [email protected]

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Spoken Lecture Project

• Processor, browser, workflow

• Prototyped with lecture & seminar video– MIT OCW (~300 hours, lectures)– MIT World (~80 hours, seminar speakers)

Supported with iCampus MIT/Microsoft Alliance funding

12

James [email protected]

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

SpokenMedia Player and Demo

13

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

What works today?

14

Lecture Transcription Workflow

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Recognizer Accuracy? Up to 85%

• Accuracy– Domain Model and

Acoustic Model– Internal validity

measure – Single 100% accurate

transcript for a full course

15

Ongoing research by Jim Glass and his team

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Transcript “Errors”

• “angular momentum and forks it’s extremely non intuitive”– “folks”?

– “torques”?

• “introduce both fork an angular momentum”– “torque”!

16

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

That’s what we have today…

• Features– Video linked transcripts– “Bouncing Ball” follow along– Search within a video– Multiple transcript language support

• Challenges– Accuracy (partial toolset)

17

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Where are we heading?

• Improved accuracy

• Search across multiple video transcripts

• Automate and improve processing– > Starting a lecture transcription service

18

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Goals of theLecture Transcription Service

• Integrate with media production workflows– At MIT, University of Queensland

• Stand-alone service– Test with external content (video) producers

19

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

A Lecture Transcription Service? Caveats

• Lecture-style content (technology optimized)• Up to 85% accuracy

– (good for search, not sure about accessibility)

• English-language audio– (need much more research for other languages)

• Processing hosted at MIT (current thinking)– Submit jobs via MIT-run service– Contribute audio, models, transcript for further

research

20

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Toward Rich Media NotebooksImproving the User Experience

• Innovative player interfaces (prototypes)– Bookmarking and annotation– Clip creation and authoring

• Transcript editing (prototypes)

• Searching across collections of videos (soon-ish)

21

In Collaboration with the Univesite de Lyon 1

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Player with Annotation Mockup

22

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

Snapshots and Transcripts

23

spokenmedia.mit.edu

Brandon Muramatsu [email protected] McKinney [email protected]

Peter Wilkins [email protected]

MIT, Office of Educational Innovation and Technology

Citation: Muramatsu, B., McKinney, A., Wilkins, P. (2010). Improving the OER Experience: Enabling Rich Media Notebooks of OER Video and Audio.Presented at OER10: Cambridge, UK, March 23, 2010.

Unless otherwise specified, this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

Try it for yourself !

sm.mit.edu/upload