Quantitative Comparison of H.264/AVC and VP8 Video Codecs for Video Conferencing Scott Kristjanson CMPT-820 Multimedia Systems Instructor: Dr. Mohamed Hefeeda Dec 2 nd 2010
Feb 22, 2016
Quantitative Comparison of H.264/AVC and VP8 Video
Codecs for Video ConferencingScott KristjansonCMPT-820Multimedia SystemsInstructor: Dr. Mohamed Hefeeda Dec 2nd 2010
Scott Kristjanson – CMPT 820 - SFU
2
Slide 2
Presentation Overview
Project Overview Methods Results Conclusions Future Work References
Scott Kristjanson – CMPT 820 - SFU
3
Slide 3
Project Overview
Compare Video Codecs for use with Video Conferencing:• VP8 – by Google’s WebM project • H.264/AVC – used x256 Open Source Codec
Scott Kristjanson – CMPT 820 - SFU
4
Slide 4
VP8 Overview
Video Codec owned by Google and made open Source• WebM Project: Matroska container with VP8 video and Vorbis audio • VP8 - Earlier versions VP7 (Skype), VP6 (Flash Video), VP3 (Theora)
Key Features:• Adaptive Loop Filtering – Identifies Fast/Slow/Static regions and
adapts encoding to match for improved quality/performance• Golden Frames – for storing reference point to background layer. Can
be used for reference without display them.• Faster encoding time by avoiding use of Arithemetic VLE (CABAC).• Dynamically scales resolution. No need to involve application,
encoder decides. No I-frame required as a transition point.
Structure:• Frame same as H.264, Segment like a Slice• Partitions used to separate critical data from video data• I-Frame, P-Frames, Golden-Frames, no B-Frames due to Patents
For more information, see Google’s “Intro to VP8” slides at:http://www.slideshare.net/DSPIP/google-vp8
Scott Kristjanson – CMPT 820 - SFU
5
Slide 5
Methods
Quantitative Comparison: Invoke encoders/decoders with various presets:• Manually from DOS, then use YUV_Compare to compute Y-PSNR • ViCoS Codec Comparison utility from MSU – gives ranking only• Used sequences foreman and salesman to simulate video conference calls• Encode with bit rates 100, 800, 1500 kbps to produce bit rate curves
Extract Quantitative Data• Y-PSNR – Signal to Noise Ratio for Luminosity• Y-SSIM – Structural Simularity Data for Luminosity• Encoding Time• Frame Rate• Bit Rate
Qualitative Comparison
• View original YUV file and encoded/decoded files using YUVviewer• Test VP8 in Video Conferencing System to verify Quantitative Data
Scott Kristjanson – CMPT 820 - SFU
6
Slide 6
Reference Video Sequences
Salesman • Ideal choice for simulating video conference• Not used since not available in CIF format• Tried to convert from .avi to CIF format but ran out of time
Foreman• Good choice, but with some background movement• Used in detailed quantitative analysis• Used with Vicos for codec over-all evaluation
Mobile• Slow motion with detailed cartoon-like background• Used with Vicos for codec over-all evaluation
Stefan• Lots of motion and lots of background detail• Used with Vicos for codec over-all evaluation
Scott Kristjanson – CMPT 820 - SFU
7
Slide 7
DOS Command Line Testing
x264 encode/decode:
vp8 encode/decode:
Visually compare original video toencoded video
with YUV_Viewer
Scott Kristjanson – CMPT 820 - SFU
8
Slide 8
Computing Y-PSNR with AviSynth
AVS script to read and compare two YUV files:• Encoded/Decoded YUV video from DOS command line testing• Original YUV video
yuv_compare_foreman.avs
foreman_yuv_compare.log
loadplugin("C:\megui\tools\avisynth_plugin\rawsource.dll")decodedClip = RawSource("C:\foreman_352x288_decoded_vp8.yuv",352, 288, "I420")originalClip = RawSource("C:\foreman_352x288.yuv", 352, 288, "I420")Compare(decodedClip, originalClip, "YUV", "foreman_yuv_compare.log")
Mean Max Max Absolute Mean Pos. Neg. Frame Dev. Dev. Dev. Dev. PSNR (dB) ----------------------------------------------------- 0 0.7591 +0.0252 5 -3 47.6277 1 0.7566 -0.0050 5 -4 47.6655 … … … … … … 299 0.8597 +0.0070 4 -4 46.8949Total frames processed: 300 Minimum Average MaximumMean Absolute Deviation: 0.6941 1.2438 3.7930 Mean Deviation: -0.2703 -0.0194 +0.0527 PSNR: 33.2940 44.9776 48.2338 Overall PSNR: 41.3285
PSNR
Scott Kristjanson – CMPT 820 - SFU
9
Slide 9
MSU’s Video Codecs Scoring System (ViCoS)
Allows custom Codecs to be bound in and tested• bind in encoders and decoders (x264 and vp8)• define encoding and decoding parameters and presets to be tested• encodes/decodes test sequences automatically and computes distortion• generates relative ranking and relative scores for PSNR and SSIM only
Versions:• Demo version used – provides ranking scores only • Professional version – creates PNSR/SSIM charts automatically
Scott Kristjanson – CMPT 820 - SFU
10
Slide 10
Video Quality – Peak Signal to Noise Ratio (Y-PSNR)
25
27
29
31
33
35
37
39
41
43
100 800 1500
Bit Rate (kbps)
Y-P
NS
R (d
b)
VP8 Bestx264 PlaceboVP8 Good1VP8 Good2VP8 Goodx264 FastVP8 Good4x264 Defaultvp8 1Pass Fast VBRVP8 RT CBR w DLVP8 RealTime CBRx264 fast zero latencyx264 Baselinex264 ultraFast
Bit-Rate/Quality Trade-Off:• 100kbps – Poor Quality• 800kbps – Acceptible to good• 1500kbps – Very Good to Excellent
PSNR Ranges:38+ Excellent 36-37 Very good34-35 Good26-33 Poor25 or less Garbage
Scott Kristjanson – CMPT 820 - SFU
11
Slide 11
Video Quality – Structural Simularity (SSIM)
0.60
0.70
0.80
0.90
1.00
100 800 1500
Bit Rate kbps
Y-SS
IM
VP8 Bestx264 PlaceboVP8 Good1VP8 Good2VP8 Goodx264 FastVP8 Good4x264 DefaultVP8 Fast VBR 1PassVP8 RT CBR w DLVP8 RealTime CBRx264 fast zero latencyx264 Baselinex264 ultraFast
SSIM score under 0.95 is quite noticible:• 100kbps – Poor Quality for all codecs
(especially x264 Fast Zero-Latency & Ultrafast)• 800kbps – most codecs Acceptible to good
(except VP8 RT CBR and x264 Ultrafast)• 1500kbps – all Very Good
(except x264 Ultrafast)
Scott Kristjanson – CMPT 820 - SFU
12
Slide 12
Encoder Frame Rates
x264 Encoder significantly faster than VP8 for similar Quality
0
15
30
45
60
75
90
105
120
135
150
165
180
195
210
225
240
255
VP8 100 x264 VP8 800 x264 VP8 1500 x264
Bit Rate (kbps)
Fram
es p
er S
econ
d
VP8 BestVP8 GoodVP8 Good1VP8 Good2VP8 Good4VP8 Fast VBR 1PassVP8 RT CBR w DLVP8 RealTime CBR---------------------x264 Placebox264 Fastx264 Defaultx264 fast zero latencyx264 Baselinex264 ultraFast
Scott Kristjanson – CMPT 820 - SFU
13
Slide 13
Good Codec Presets for Video Conferencing
Need min of 15 fps for Video Conferencing, prefer 30 fpsWant decent quality – Y-PSNR of 34 or betterLow Latency: under 150ms required4 Presets met these requirements:
Codec Preset
Scott Kristjanson – CMPT 820 - SFU
14
Slide 14
Conclusion
VP8 and x264 provide similar video quality• For higher bit rates, x264 PSNR better by 2%• For lower rates, VP8 wins by almost 10%
x264 provides better performance in some cases• x264 is between 36% and up to 2x faster than VP8• A big improvement! VP8 used to be 5-20 times slower!
Scott Kristjanson – CMPT 820 - SFU
15
Slide 15
References:
[1] Image Quality Assessment: From Error Visibility to Structural SimilarityZhou Wang, Alan Conrad Bovik, Hamid, Rahim Sheikh, and Eero P. SimoncelliIEEE Transactions on Image Processing, Vol 13, No. 4, April 2004http://ieeexplore.ieee.org.proxy.lib.sfu.ca/stamp/stamp.jsp?tp=&arnumber=1284395
[2] 6th Annual MPEG-4 AVC H.264 video codecs comparisonD. Vatolin, D. Kulikov, and A. Parshin, April 2010http://compression.ru/video/codec_comparison/h264_2010/
Scott Kristjanson – CMPT 820 - SFU
16
Slide 16
Backup Slides
Scott Kristjanson – CMPT 820 - SFU
17
Slide 17
H.264/AVC Presets Used
Codecs tested with various presets and tunings• Looking for good balance of quality and speed for video conferencing• Each tested with 3 target Encoding bit rates: 100, 800, 1500 kbps
x264 Presets:• Placebo
• 2-Pass Slower Encoding Speed, higher quality• Fast (tested with 2 tunings)
• 2-Pass Faster Encoding Speed, good quality• 2-Pass tuned with zero latency specified
• Default • 1-pass with default settings
• Baseline• 1-pass fast encoding but poor quality at lower bit rates
• Ultrafast• When speed is all that matters!
Scott Kristjanson – CMPT 820 - SFU
18
Slide 18
VP8 Presets Used
Best• 2-Pass encoding• Maximimizes Quality but slower encode times
Good 2-Pass• 2-Pass encoding, less quality but faster encode times • measured with 1, 2, and 4 cpu cores specified
Good 1-Pass• 1-Pass encoding, less quality but faster encode times
Fast VBR• 1-Pass variable bit-rate fast encoding
RealTime CBR • Encoder Frame buffer size 6• with and without deadline specified
Scott Kristjanson – CMPT 820 - SFU
19
Slide 19
VP8 Support for MultiCore Processors
VP8 Encoding Rate vs # of CPUs Preset Used: "Good"
0
10
20
30
40
50
60
70
80
90
100 800 1500
Bit Rate (kbps)
Fram
es P
er S
econ
d
1 CPU2 CPUs4 CPUs
Scott Kristjanson – CMPT 820 - SFU
20
Slide 20
Encoding Bit Rate vs Target Bit Rate
Both Codecs appear to over-run target slightlyVP8 tracks to target bit-rate slightly better
Actual versus Target Bit Rate
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
1900
1 2 3 4 5 6 7 8
Codec
Targ
et B
it R
ate
VP8 1500 kbpsVP8 800 kbpsVP8 100 kbpsx264 1500 kbpsx264 800 kpbsx264 100 kbps
Scott Kristjanson – CMPT 820 - SFU
21
Slide 21
Video Quality – Structural Simularity (SSIM)
0.6
0.7
0.8
0.9
1.0
Y-S
SIM
100 kbps800 kbps1500 kbps
SSIM score under 0.95 is quite noticible:• 100kbps – Poor Quality for all codecs (especially x264 Fast 0-Latency and Ultrafast)• 800kbps – most codecs Acceptible to good (except VP8 RT CBR and x264 Ultrafast)• 1500kbps – all Very Good