Study-Element Based Adaptation of Lecture Videos to Mobile Devices Ganesh Narayana Murthy (M.Tech IIT Bombay) Sridhar Iyer (Associate Professor, IIT Bombay)

Study-Element Based Adaptation of Lecture Videos to Mobile Devices

Ganesh Narayana Murthy (M.Tech IIT Bombay)Sridhar Iyer (Associate Professor, IIT Bombay)

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 1

Problem Definition

• Adapt CDEEP videos to be viewable on mobile devices:– Viewable at low network bandwidths (like GPRS)– Viewable at low cost

• Video bit-rate – Size of video stream over time– Total size = bit-rate * total time– CDEEP video bit-rate: 1150kbps– GPRS bit-rate: 40kbps

• The problem:– Video playing incurs delays if available network bandwidth is less than

video-bit rate

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 2

Video Transcoding

• Converting from one video format to another– Changing video bit rate– Changing other parameters like frame rate, screen

resolution

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 3

Format Name Typical Bit Rate Application

MPEG-1 1.5Mbps or less CD-ROM

MPEG-2 5-8Mbps DVD, HDTV

H.263 Typically low bit rates Low bit-rate video conferencing

MPEG-4 / H.264 40Kbps to 10Mbps and above

Internet Streaming, Video Telephony

Flash Video (FLV) Typically low bit rates Embedded video in websites

Video Quality at low-bit rates

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 4

(a) MPEG-1 (b) MPEG-2

Images from transcoded videos (Target bit rate : 40kbps, No audio)

Video Quality at low bit-rates (contd.)

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 5

(c) H.264 (mp4) (d) H.263 (3gpp)

Images from transcoded videos (Target bit rate : 40kbps, No audio)

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 6

(e) Flash Video (flv)

Images from transcoded videos (Target bit rate : 40kbps, No audio)

Comparison of Video CodecsFormat Name

Original VideoSize

Converted VideoSize

Video Quality at low-bit rates

Remarks

MPEG-1 432MB 26MB Poor Cannot be used at low bit-rates

MPEG-2 432MB 29.12MB Poor Cannot be used at low bit rates

H.263 432MB 38.3MB Poor Cannot be used at low bit rates

H.264 432MB 16.9MB Good Processing power / Decoding complexity is high.[1]

Flash 432MB 20.5MB Good Can be used, but cost is still high.

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 7

(Note: Video bit rate = 1150kbps, No audio, Target bit rate = 40kbps, No audio)

Video Sizes are still high forviewing over GPRS

Study-Element Based Adaptation

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 8

Motivation

• CDEEP video usually consists of– Presentation slides– Instructor explaining on white paper– Video of instructor talking

• Presentation slide is usually not changing– Video of slide is not required. One image is sufficient

• Idea– Extract one image every ‘n’ seconds and send to client.– This would reduce amount of data sent for showing one

slide.

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 9

Method-1

• Send one image every ‘n’ seconds– Server sends one image every ‘n’ seconds to client– Audio is simultaneously streamed

• Network bandwidth and Size – Network Overhead (NO) = Image Size / n– Size Overhead (SO) = Total size of images

• What is the user experience?

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 10

User Experience Basis

• Presentation Study Element– Portion of video showing one slide

• White Paper Study Element– Portion of video showing instructor writing on

white paper

• Instructor Study Element– Portion of video showing instructor talking

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 11

…………..

0 5 10 15

Presentation Slide

Delay in start of slide

3

………..

25 30 35

White Paper

Video Time(secs)

User Experience• Presentation Element

– Delay Experienced (D2) = • Delay in start of slide as compared to audio

• White Paper Element– Delay Experienced (D1) =

• Delay between any two consecutive images = Sending Rate

• Instructor Element– Only audio important. No image need be sent.

• User Experience is assumed to be one

• User Experience (Ui) = 1 sec / Di

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 12

Method-2• Trade-off for user experience

• Cost incurred in terms of number of images sent

• Same sending interval for all elements, cannot balance user experience and cost.

• Choose different sending interval for each study element

• Probably:– Higher user experience for white paper

element– Lower user experience for presentation

element

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 13

User Experience

Cost

Sending Interval

Trade-Off Relation

System Overview

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 14

Building the index

• Corpus of 10 videos– Representative of various departments

• Consider different sending intervals ‘r’– For each ‘r’ find NO,SO and U for every study element in a

video. – Repeat for all videos and take average.

• This relation can be used backwards:– For calculating sending interval, given network bandwidth

and user experience.

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 15

Graphs of User Experience

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 16

Graphs of overheads

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 17

Results

Original Video Size(MB)

Images Size (MB)

Reduction (%)

U1 U2 SupportedNetwork Bandwidth

432 2.85 97% 0.2 0.38 20kbps and above

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 18

Achieved Size Reduction

Fig: Video stream size reduction (note: Original video bit-rate = 1150kbps, No audio)

Results (contd.)

Sending Interval U1 U2 NO1(kbps)

S01(MB)

NO2(kbps)

SO2(MB)

Total Size (SO)

White Paper Element

Presentation Element

5 5 0.2 0.337 15.12 1.25 23.43 1.6 5.46

5 15 0.2 0.115 15.12 1.25 7.81 0.53 1.78

15 15 0.067 0.115 5.254 0.781 7.63 1.03 1.81

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 19

Balance User Experience and Cost

Required Network Bandwidth =max(NO1,NO2) is reduced

Reduction in size user experience for white paper element remaining same

Conclusion

• Large size reduction can be achieved by using the concept of slideshows

• Identifiying study-elements within the video helps define user-experience of the slideshow.

• CDEEP Lecture videos can be adapted to low network bandwidths and in a cost-controlled manner.

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 20

Future Work

Automated tagging– Identifying study element boundaries – Shot detection techniques

User Experience Correlation– Identifying relation between obtained user

experience and actual user values

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 21

References1. H.264 white paper.

http://ati.amd.com/products/pdf/h264_whitepaper.pdf.2. Real-time Content-Based Adaptive Streaming of Sports Videos.

Shih-Fu Chang, Di Zhong, and Raj Kumar. In CBAIVL '01: Proceedings of the IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL'01), page 139, Washington, DC, USA, 2001. IEEE Computer Society.

3. Content-aware video adaptation under low-bitrate constraint. Ming-Ho Hsiao, Yi-Wen Chen, Hua-Tsung Chen, Kuan-Hung Chou, and Suh-Yin Lee. EURASIP J. Adv. Signal Process,2007(2):27-27, 2007.

4. A Characteristics-Based Bandwidth Reduction Technique for Pre-recorded Videos. Wallapak Tavanapong and Srikanth Krishnamohan. In IEEE International Conference on Multimedia and Expo (III), pages 1751-1754, 2000.

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 22

Questions?

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 23

Content-Aware Adaptation

Ganesh Narayana Murthy NCC 2010, Chennai, 30/01/2010 24

Method Name Adaption Mechanism Video Quality Remarks

Hsiao et.al.[2] Identify visual attention regions in a frame. Encode them at high quality.

Poor Quality of important objects still depends on network bandwidth

Chang.et.al [3] Identify events in sports videos at high quality. Other regions as slideshows.

Good Slideshow of images reduces network bandwidth and size

Tavanapong. et. Al. [4]

Identify non-changing portions of lecture video and extract one image from them

Good Exploits redundancy in lecture videos