Doc.:IEEE 802.11-05/1194r0 Submission November 2005 Philip J. Corriveau - IntelSlide 1 Video Testing Methodology Notice: This document has been prepared.

November 2005

Philip J. Corriveau - IntelSlide 1

doc.:IEEE 802.11-05/1194r0

Submission

Video Testing Methodology

Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.

Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802.11.

Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures <http:// ieee802.org/guides/bylaws/sb-bylaws.pdf>, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair <[email protected]> as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802.11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at <[email protected]>.

Authors:Name Company Address Phone email

Philip Corriveau Intel HF3-96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696-1837 [email protected]

Audrey Younkin Intel HF3-96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696-3947 [email protected]

Chris Olsen Intel HF3-96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696-7548 [email protected]

Fernald Royce Intel HF3-96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696-4318 [email protected]

Rik Logan Intel 5200 NE Elam Young Pkwy Hillsboro, OR 87124 (503)-712-1675 [email protected]

Uriel Lemberger Intel PO Box 1659, Matam Industrial Park, Haifa 31015 Israel +972-4-865-5701 [email protected]

Neeraj Sharma Intel 13290 Evening Creek Drive, San Diego, CA 92128 (858)-385-4112 [email protected]

Sasha Tolpin Intel PO Box 1659, Matam Industrial Park, Haifa 31015 Israel +972-4-865-5430 [email protected]

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Agenda

• Video Assessment

• GED Process Overview

• Purpose

• Objective Outputs

• Experimental Design

• Methodology Overview

• Summary

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

• Philip Corriveau & Audrey Younkin• User Centered Design • Media and Acoustics Perception Lab

Intel Video Testing Methodology Concepts and Methods

• Olsen Christopher & Fernald Royce

• Platform Systems Technology

• Wireless Integration Team

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Video Assessment~Not a One Step Process~

• The Gross Error Detector (GED) is one of the first steps in assessing video experience. – The GED presents a simple and quick way to measure large errors,

such as dropped or repeated frames or video playback at an incorrect frame rate.

• The GED is designed to communicate an Experiential Value that can be directly correlated to a real end-user.

• Other tools like VQM, then can capture and measure video quality (artifacts).

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Why Use GED?

• The Video Gross Error Detector is a high-level video performance analysis application

• Determines whether a video stream arrives intact, in sequence and at the correct frame rate

• Enables automated, quantitative, repeatable measurements of video quality

• Significantly less expensive and more accurate than manual video quality assessment

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

GED Process Overview

0. Source Material 1. Marked Source Material

2. Compressed and Marked

4. GED Analysis

GED Encode WMV Encode

System Under Test

3. Capture Results

GED Decode

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

GED Encode

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Purpose

• Currently the Gross Error Detector (GED) measures video performance based of a set of objective metrics.

• However, this does not present the entire picture for video performance and assessment. For more accurate end user analysis, there needs to be a subjective mean opinion score (MOS) coupled with the objective metrics.

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Current Objective Outputs

• Dropped Frames

• Repeated Frames

• Delayed Frames

• Non-sequential Frames

Future Objective Outputs

• Audio Error Detection (end of Q4)

• AV Sync (end of Q4)

• Frame Hashing - detection of blur and video corruption (Q1 ’06)

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Experimental Design

• Currently the GED experimental design is broken into two Phases– Phase I: Expert Subjective Assessment

– Phase II: Non-Expert Subjective Assessment

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Phase I Expert Assessment

• Using a SD (720 x 486) compiled collection of 10 second clips adding up to a total of 321 seconds (~5 minutes).

• The GED tool will do an average over the 10 second clips with respect to metrics.

• The GED is independent of the codec, streaming server and hardware platform.

• Using Windows Media Services with 1-6 Mbps WMVs and the East Fork server with 1-8 Mbps MPEG2 SD files.

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

General Methodology

• To create a Subjective Score (1-5) there needs to be an evaluation of what real life situations produce in respect to a wireless connection.

• The wireless path loss and other effects will cause different amounts of frames dropped/repeated that can be captured and compared to the original GED.

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Example of Expected Scoring Value

• However, the temporal factor also needs to be put in place, not only the number of dropped frames, but where they occur in the sequence and how long each error occurs for.

• Each clip is 260 frames long.

Expert Score

Number Frames

Dropped

Percentage Frames

Dropped

5 >4 1.5%

4 >10 3.8%

3 >50 19%

2 >100 38%

1 >200 77%

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Phase II Non-Expert Subjective Assessment

• Subjects:

40-70 non-expert external participants will rate video quality on a scale of 1-5 (half male/half female). Using Market Decisions Corporation for outsourcing of participants.

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

General Methodology

• Based on the Expert Analysis, five new sequences (one for each score) will be developed based on the GED outputs and temporal factors.

• One or two participants per test session.

• Upon arrival to the study location, participants will be asked to read through a brief overview of the study consisting of a summary of the test procedure requirements and will be asked to sign a confidentiality agreement and consent form.

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

How to Rate

• Prior to each session, participants will be instructed to rate each video sequence based on its real-time smoothness and flow.

• Do not rate experience based on content quality but rather the flow of presentation.

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Practice Trial

• A familiarization and practice trial will carried out to begin the test session.

• Here, participants will be shown the best and the worst quality for frame of reference.

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Randomization

• The presentation order of the video content will be randomized using a pseudo-random number generator tool to prevent ordering effects, i.e. to mask any tendency for a participant observer to rate a clip in relation to the previous one.

• Randomization is a key element of psycho-visual testing to ensure that participants do not see the material in a repeated fashion that would allow a learning effect.

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Voting Opinion

• After each video sequence, participants will be presented with a pop-up window requiring their input of their opinion of the video experience on a subjective rating scale. This scale ranged from excellent (numerical equivalent of five) to bad (numerical equivalent of one).

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Numerical Values

Value Presented to Participant Actual Numerical Value

Excellent 5

Good 4

Fair 3

Poor 2

Bad 1

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Participants will sit at a predetermined viewing distance

20°

5HH

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Post Test

• At the end of the testing session, participants will be asked to fill out a post test questionnaire based on their experience.

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Methodology Overview

Tutorial

Shown “Bad” Video

Shown “Excellent” Video

Practice Trial

Clips Randomized

Subjective Vote

Post-Test Survey

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Summary

• Video Assessment not a one step process.

• We presented one of the tools GED, which is quantitative, repeatable measurements of video quality.

• Experiments concepts were described that help to correlate subjective test results with objective measurements.

• Results from the experiment described within will be presented in the next meeting in January

November 2005


doc.:IEEE 802.11-05/1194r0

Submission

Next Steps

• Intel will correlate GED and Subjective data to Link (Phy and Mac) Layer statistics for 802.11

• Develop a model for 802.11 for Video Quality

• Deliver results at January meeting

Doc.:IEEE 802.11-05/1194r0 Submission November 2005 Philip J. Corriveau - IntelSlide 1 Video Testing Methodology Notice: This document has been prepared.

Documents