Video delivery methods

Video Delivery Methods

Yossi Cohen

Intro: “Legacy” Methods

Streaming Streaming methods

RTP/RTSP RTMP Real MS WMV/ASF

Proprietary and Fragmented Seek-ability – supported Server side – proprietary technology (FMS) Cache-ability – requires special servers for streaming User experience – very good Cost – high

Progressive DownloadSimple Utilizing existing protocols & servers (HTTP) Media file is prepared: metadata up front Playback - after metadata is received Cache-ability - supported Seek-ability – very limited support Poor user experience - seek, multi-rate Waste of bandwidth when not watched fully Low cost

Pseudo Streaming Media is sent as a regular file like Progressive DW Server must understand how the media is structured Playback: after metadata is received Existing protocols

Non standard server Non standard client component

Cache-ability – Limited ! Seek-ability – supported User experience – better than PD, support seek. Waste of bandwidth when not watched full

HTTP Streaming Intro HTTP Streaming offers the advantages of:

Progressive download in terms of Cost Standard Server Scalability Standard client components (OSMF)

Streaming in terms of User experience Seek-ability of streaming

Delivery Methods Summary

Progressive Download Session Analysis

Flash Video

Progressive Download Uses file download from an HTTP web server. Uses HTTP GET request Flash player enables file playback while the download is

still in progress. The ability to be played while the file is being

downloaded is in the wrapper (container) of the file.

PROGRESSIVE DW FLASH SESSION

Request from YouTube First stage is to connect to YouTube Server with the

HTML page request. Source: 192.168.2.102 Destination: 72.125.39.138 GET /watch?v=lAl28d6tbko&feature=popt00il00

Notice that the request passes throw server 72.125.39.138 witch is identified as a Google server

Request from YouTube The Request itself is pointed to the host name (from the

HTTP message), wich is YouTube HTTP server. The HTTP server provides only the HTML page of the

request Host: www.youtube.com

1) Request HTML Page2) HTML response

Browser192.168.2.102

Web Server72.125.39.138

Session Progress

Requesting the Flash Video Player The Player is then requested automatically by the Flash

client – the information is retrieved from the query string (in bold).

<script> (function() {var isIE = /*@cc_on!@*/false;var swfHTML = (isIE) ? "<object height=\"38" + "5\" width=\"64" + "0\" classid=\"clsid:D27CDB6E-AE6D-11cf-96B8-444553540000\" id=\"movie_player\" ><param name=\"movie\" value=\"http:\/\/s.ytimg.com\/yt\/swf\/watch_as3-vfl157163.swf\"><param

name=\"flashvars ….})();</script>

*note this is only a fragment of the query string (located in var swfHTML )

Session Progress

Browser192.168.2.102

Web Server72.125.39.138

File Server82.80.222.83

1) Request HTML Page2) HTML response3) Request the Flash

Player4) Flash Player response

Requesting of Player Models The swf file that contains TAGS to information that the

player needs. http://s.ytimg.com/yt/swf/watch_as3-vfl157163.swf

The SWF can be analyzed with swfmill. This can help locating the TAG’s inside the file.

For example: the tags containcom.google.youtube.ui.QualityButton_HqOffIcon_dataClass 1 com.google.youtube.ui.WatchEndScreen_replayIcon_dataClass 2 com.google.youtube.ui.QualityButton_HqOffIcon 3

Request from YouTube The Flash Player requests the model information from

the CDN server. YouTube uses many CDN around the world. Source: 192.168.2.102 Destination: 212.179.74.181

GET/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor%2Coc%3AU0dWRVZSUl9FSkNNNl9OTFZB&fexp=905700%2C902904&algorithm=throttle-factor&itag=34&ipbits=0&burst=40&sver=3&expire=1270674000&key=yt1&signature=B20512CB5B62A0FCEA5D15EFF56AB35B35524614.15CDEF403DDDC9716E51EBB108CBA7D7CEFD7D98&factor=1.25&id=940976f1dead6e4a&

Request from YouTube The host name is different too. Notice that this is a IP

located in Israel (host: bzq-82-80-222-82.red.bezeqint.net) – this is a Proxy.

Request from YouTube Notice that this is a different server: IP: 82.80.222.82 The host name is different too. Notice that this is a IP

located in Israel (host: bzq-82-80-222-82.red.bezeqint.net) – this is a Proxy.

Request from YouTube The original host of the message is Googles CDN

server: Host : video-stats.video.google.com IP: 74.125.79.139

Complete Session

Browser192.168.2.102

Web Server72.125.39.138

File Server82.80.222.83

FLV Server / CDN212.179.74.181

1) Request HTML Page

2) HTML response3) Request Flash

Player4) Flash Player

response5) FLV File Request

Flash Player192.168.2.102

Modern Capture (Byte Ranges)

Changes from Previous

Decision on request type (Standard / byte range( based on Request 204 replay

TCP timing using Tval Tsecr Algorithm definition in request “cburst”, “throttle” Byte range requests

204 Request

Measures latency -> Input for request type mechanism

Algorithm Request

Cburst / throttle depending on content requested Client side instead of server side control

Byte Ranges

HTTP Streaming common

Over HTTP Media Segments:

Complete media units Separate media types (A/V)

Segments alignment GOP Aligned Time Aligned

Description File Must hold segment time frame Must Hold bit rate information Might include Media description like resolution, codecs

HTTP Streaming AppleMicrosoftAdobe

HTTP Live Streamingby Apple

Agenda System Overview Components Session

Apple’s Note

Note: Many existing streaming services require specialized servers to distribute content to end users. It requires specialized skills to set up and maintain these servers, and in a large-scale deployment these servers can be costly. Apple has designed a system that avoids this by using standard HTTP to deliver the streams.

System Overview

Components Review Server

Encoder Segmenter

Distributer Basic HTTP Server

Client

Server Receives Digital / Analog input stream Encodes / Transcode video/audio

H.264 Video AAC audio (HE-AAC or AAC-LC)

Encodes / Transcode audio only: MPEG-2 elementary streams, HE-AAC or AAC-LC

files, or MP3 files Encapsulate in MPEG2

Transport Stream Program Stream

Segmenter All segments should be with the same duration All segments are placed in a separate file Creates Index file with references to segment files For protection, the Segmenter might encrypt each media

segment and create a key file

Distribution Distribution system is a regular HTTP Server Could be Apache or small embedded Server

Files Segments – stored as *.ts files Index files – stored as *.m3u8 Index file format example:

#EXTM3U#EXT-X-TARGETDURATION:10#EXTINF:10,http://media.example.com/segment1.ts#EXTINF:10,http://media.example.com/segment2.ts#EXTINF:10,http://media.example.com/segment3.ts#EXT-X-ENDLIST

Session types Live Stream Broadcast

Index file is continues updated Include a moving window of segments around “live”

part of the session Client should continuously refresh the Index file

VoD Session Index file static Includes ALL the segments of the file Enables “Seek” operation

Multi-bitrate multi–device support Multi-bitrate is enabled via multiple index files Index files are pointed by a global index files Client can select a stream according to:

Device properties Available bit rate

This method is less efficientthan Silverlight Global File

Test yourself1. What are the two Live streaming file types?2. What is the role of the Segmenter?3. On which delivery protocol is the live streaming based?

Silverlight Smooth Streaming

Movi

ng to

DASH

SILVERLIGHT INTRODUCTION

Smooth Streaming

•Microsoft’s implementation of HTTP-based adaptive streaming

•A hybrid media delivery method that acts like streaming but is in fact a series of short progressive downloads

•Leverages existing HTTP caches•Client can seamlessly switch video quality

and bit rate based on perceived network bandwidth and CPU resources

Smooth Streaming Design

•Smooth Streaming File Format based on MP4 (ISO Base Media File Format)

•Video is encoded and stored on disk as one contiguous MP4 file▫Separate file for each bit rate

•Each segment is composed of a 1 or more GOP This allows easy fragmentation at key frames

•Contiguous file is virtually split up into chunks when responding to a client request

Evolution•Previous versions of MS streaming divide

the file into many chunkc 0001.vid 0002.vid etc

•Problematic in caching, CDNs, CMS etc•Today all fragments of a file are contained

in a single bitstream container. Typically 1 fragment = 1 video GOP.

SILVERLIGHT FILES

Containers & Configuration files

MP4 File format MP4 has two format types

Disk Format - for file storage Wire format - for transport

Wire format enables easy CDN support and integration

Smooth Streaming File Format

Smooth Streaming Wire Format

File extensions Media Files

*.ismv - Audio & Video *.isma – Audio only

Manifest Files *.ism – Server manifest. Describes to the server

Relation between tracks, bitrates & files on disk. Based on SMIL 2.0 XML format specification

*.ismc – Describes to the client the available streams, CODECS used, bitrates encoded, video resolutions, markers, captions. First file delivered to client. It’s the first file delivered to client (“SDP” like).

Directory Structure

Manifest Files

Media file in different bitrates

Manifest files VC-1, WMA, H.264 and AAC codecs Text streams Multi-language audio tracks Alternate video & audio tracks (i.e. multiple camera

angles, director’s commentary, etc.) Multiple hardware profiles (i.e. same bitrates targeted at

different playback devices) Script commands, markers/chapters, captions Client manifest Gzip compression URL obfuscation Live encoding and streaming

ISM file sample<?xml version="1.0" encoding="utf-16" ?> - <!-- Created with Expression Encoder version 2.1.1206.0  --> - <smil xmlns="http://www.w3.org/2001/SMIL20/Language">- <head>  <meta name="clientManifestRelativePath" content="NBA.ismc" />   </head>- <body>- <switch>- <video src="NBA_3000000.ismv" systemBitrate="3000000">  <param name="trackID" value="2" valuetype="data" />   </video>- <video src="NBA_2400000.ismv" systemBitrate="2400000">  <param name="trackID" value="2" valuetype="data" />   </video>- <video src="NBA_1800000.ismv" systemBitrate="1800000">  <param name="trackID" value="2" valuetype="data" />   </video>

ISM file sample- <video src="NBA_1300000.ismv" systemBitrate="1300000">  <param name="trackID" value="2" valuetype="data" />   </video>- <video src="NBA_800000.ismv" systemBitrate="800000">  <param name="trackID" value="2" valuetype="data" />   </video>- <video src="NBA_500000.ismv" systemBitrate="500000">  <param name="trackID" value="2" valuetype="data" />   </video>- <audio src="NBA_3000000.ismv" systemBitrate="64000">  <param name="trackID" value="1" valuetype="data" />   </audio>  </switch>  </body>  </smil>

*.ISMC sample<?xml version="1.0" encoding="utf-16" ?> - <!-- Created with Expression Encoder version 2.1.1206.0  --> - <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="4084405506">- <StreamIndex Type="video" Subtype="WVC1" Chunks="208"

Url="QualityLevels({bitrate})/Fragments(video={start time})">  <QualityLevel Bitrate="3000000" FourCC="WVC1" Width="1280" Height="720"

CodecPrivateData="250000010FD3FE27F1678A27F859E80C9082DB8D44A9C00000010E5A67F840" />

  <QualityLevel Bitrate="2400000" FourCC="WVC1" Width="1056" Height="592" CodecPrivateData="250000010FD3FE20F1278A20F849E80C9082493DEDDCC00000010E5A67F840" />

<QualityLevel Bitrate="1800000" FourCC="WVC1" Width="848" Height="480" CodecPrivateData="250000010FCBF81A70EF8A1A783BE80C908236EE5265400000010E5A67F840" />

  <QualityLevel Bitrate="1300000" FourCC="WVC1" Width="640" Height="352" CodecPrivateData="250000010FCBE813F0AF8A13F82BE80C9081A7ABF704400000010E5A67F840" />

ISMC File - 2- <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999">- <StreamIndex Type="video" Subtype="WVC1" Chunks="299"

Url="QualityLevels({bitrate})/Fragments(video={start time})">  <QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720"

CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" /> …..

<c n="0" d="20000000" />   <c n="1" d="20000000" /> .....  <c n="298" d="5000001" />   </StreamIndex>- <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299"

Url="QualityLevels({bitrate})/Fragments(audio={start time})">  <QualityLevel Bitrate="64000"

WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000000E00042C0" />

  <c n="0" d="20433560" /> ....  <c n="297" d="20433560" />   <c n="298" d="4393197" />   </StreamIndex>  </SmoothStreamingMedia>

SILVERLIGHT SESSION

Initiation and Flow

Smooth Streaming Protocol Smooth Streaming Protocol uses HTTP [RFC2616] as its

underlying transport . The Server role in the protocol is stateless

Enabling (potentially) different instance of the server to handle client requests

Request can utilize any generic HTTP caches/proxies - > Lowering CDN costs

Messages Smooth Streaming Protocol uses 4 different messages:

Manifest Request Manifest Response Fragment Request Fragment Response

All messages follow the HTTP/1.1 specification

Messages FlowServer

Client

Manifest Request

Manifest Response

Fragment Request

Fragment Response

Fragment Request(s)

Messages Manifest Request and Fragment Request message

MUST use the HTTP "GET" method, generated by the client.

Manifest Request and Fragment Request message use the HTTP Response messages.

Status-Code SHOULD be 200.

Smooth Streaming Transport Protocol Session Details

Manifest RequestManifest Request

Manifest ResponseManifest ResponseVideo Fragment RequestVideo Fragment Request

Fragment ResponseFragment Response

Audio Fragment RequestAudio Fragment Request

Session Details - Manifest Request

In order to initiate a presentation the Client MUST send the server a Manifest Request using the HTTP GET method.

Session Details - Manifest Response

The Response is a ISMC Manifest file describing the session. - <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999">

- <StreamIndex Type="video" Subtype="WVC1" Chunks="299" Url="QualityLevels({bitrate})/Fragments(video={start time})">

  <QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" />

…..

<c n="0" d="20000000" />

  <c n="1" d="20000000" />

.....

<c n="297" d="20000000" />

  <c n="298" d="5000001" />

  </StreamIndex>

- <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299" Url="QualityLevels({bitrate})/Fragments(audio={start time})">

  <QualityLevel Bitrate="64000" WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000000E00042C0" />

  <c n="0" d="20433560" />

....

  <c n="297" d="20433560" />

  <c n="298" d="4393197" />

  </StreamIndex>

  </SmoothStreamingMedia>

Manifest Response reviewed We can see in the ISMC file that the server can support 8 different levels

of quality (bitrate) for the client can chose from between 2.75Mbit to 0.35 Mbit.

- <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999">

- <StreamIndex Type="video" Subtype="WVC1" Chunks="299" Url="QualityLevels({bitrate})/Fragments(video={start time})">

  <QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" />

  <QualityLevel Bitrate="2040000" FourCC="WVC1" Width="1056" Height="592" CodecPrivateData="250000010FD3BE20F1278A20F849E80450823E414DD1400000010E5AE7F840" />

  <QualityLevel Bitrate="1520000" FourCC="WVC1" Width="848" Height="480" CodecPrivateData="250000010FCBAE1A70EF8A1A783BE8045081AE62F3F7400000010E5AE7F840" />

  <QualityLevel Bitrate="1130000" FourCC="WVC1" Width="704" Height="400" CodecPrivateData="250000010FCBA215F0C78A15F831E8045081A27BD635C00000010E5AE7F840" />

  <QualityLevel Bitrate="845000" FourCC="WVC1" Width="576" Height="320" CodecPrivateData="250000010FCB9A11F09F8A11F827E804508199C94077400000010E5AE7F840" />

  <QualityLevel Bitrate="630000" FourCC="WVC1" Width="448" Height="256" CodecPrivateData="250000010FCB920DF07F8A0DF81FE804508113396020C00000010E5AE7F840" />

  <QualityLevel Bitrate="470000" FourCC="WVC1" Width="368" Height="208" CodecPrivateData="250000010FC38E0B70678A0B7819E80450810E5747B6C00000010E5AE7F840" />

  <QualityLevel Bitrate="350000" FourCC="WVC1" Width="320" Height="176" CodecPrivateData="250000010FC38A09F0578A09F815E80450808AADEACF400000010E5AE7F840" />

Manifest Response – reviewed The client also receives the number of chunks for audio and video tracks

and the duration of each chunk so it can request the chunk which fits the desired position in the file

<c n="0" d="20000000" />

  <c n="1" d="20000000" />

  <c n="2" d="20000000" />

  <c n="3" d="20000000" />

....

  <c n="297" d="20000000" />

  <c n="298" d="5000001" />

  </StreamIndex>

- <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299" Url="QualityLevels({bitrate})/Fragments(audio={start time})">

  <QualityLevel Bitrate="64000" WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000000E00042C0" />

  <c n="0" d="20433560" />

  <c n="1" d="19969161" />

  <c n="2" d="19969161" />

  <c n="3" d="20433560" />

  <c n="4" d="20433560" />

<c n="297" d="20433560" />

  <c n="298" d="4393197" />

  </StreamIndex>

  </SmoothStreamingMedia>

Session Details – Fragment Request

Client-Server requests are based on RESTFull URLs:GET /mediadl/iisnet/smoothmedia/Experience/BigBuckBunny_720p.ism/QualityLevels(350000)/Fragments(video=0)

The URL includes reference to: Bitrate as QualityLevels which maps to a media file Fragment number

Session Details – Fragment Response

The Server: checks “BigBuckBunny_720p.ism” server manifest file to find the

media file associated with the quality level(350000) Opens and parses the associated media file to get the chunk with

requested time offset (0). Sends the requested media fragment to the client as HTTP

response with status code set to 200

Examplehttp://www.iis.net/media/experiencesmoothstreaming

Adobe HTTP Streaming

Movi

ng to

DASH

Thank you!

• Video Expert

• Lectures on Video / Android / VoIP

• Android Native Developer

More About me:

Yossi CohenYossi [email protected] [email protected]

+972-545-313092+972-545-313092

Backup slides

Adobe Solution main components

Agenda Overview Components Files Session

HTTP Streaming Intro HTTP Streaming offers the advantages of:

Progressive download in terms of Cost Standard Server Scalability Standard client components (OSMF)

Streaming in terms of User experience Seek-ability of streaming

OverviewAdobe HTTP Dynamic Streaming is a solution that allows you to stream live and on-demand content over HTTP to Adobe Flash Player. When content streams over HTTP, clients can seek quickly to any location. HTTP Dynamic Streaming supports adaptive streaming, DVR functionality, and Adobe Flash Access protection (DRM).

COMPONENTS

• Content Ingest• Server• Client

Main components Preparation

File Packager Live Packager for HTTP Dynamic Streaming

Server Apache module (HTTP Origin Module) Flash Access

Client Player with OSMF classes Flash Player version 10.1+ Air 2.0+

Ingest- File Packager•A command-line tool •Used for converting offline content to formats

required for Adobe HTTP streaming •Translates on-demand media files into

fragments and writes the fragments to F4F files.

Ingest - Live Packager• The Live Packager for HTTP Dynamic

Streaming is part of Adobe Flash Media Server.

•The server ingests a live stream over RTMP and translates it into F4F files in real-time.

•The built-in Apache HTTP Server uses the HTTP Origin Module to deliver the live content over HTTP.

Server - Apache module (HTTP Origin)•Extension to Apache HTTP Server version

2.2. •Enables processes of Adobe Files:

▫F4F, F4M,F4X ▫.bootstrap and ▫.drmmeta

•Flash Media Interactive Server 3.8 includes Apache HTTP Server.

Server - Flash Access

• DRM Server • Flash Access delivers protected

media to Flash Player• For content protection, both File

Packager and Flash Media Server are required to package and encrypt the content

Client - OSMF classes• The OSMF Player uses the

ActionScript 3.0 NetStream.appendBytes() API to deliver data to Flash Player.

•OSMF is a robust framework designed to deliver high-quality video.

•Adobe strongly recommends using OSMF to build HTTP Dynamic Streaming players.

ADOBE HTTP STREAMING FILESF4F, F4M,F4X .bootstrap and .drmmeta

Files The files required for HTTP streaming are:

F4F - MPEG4 media format. Holds the media F4M – Media description file(codec, resolution) F4X - Fragments location file .bootstrap – bootstrap information for each

segment .drmmeta – DRM encryption information

*.F4F File

•Standard MP4 format with open file specification

•Each file contains a segment of the source file.

•Each segment contains one or more fragments of content.

•The file formats stores any flash supported codec

•A player can use a URL to address each fragment.

HTTP Streaming file types

•*.F4X File▫Flash Index file. ▫Contains the location of specific

fragments within a stream. •*.F4M File▫Flash Media Manifest file. ▫Contains information about the media

codecs, resolution, and the availability of multi-bitrate files.

F4M File Sample<?xml version="1.0" encoding="utf-8" ?> - <manifest xmlns="http://ns.adobe.com/f4m/1.0">  <id>Dynamic Streaming</id>   <duration>253</duration>   <mimeType>video/mp4</mimeType>   <baseURL>rtmp://cp67126.edgefcs.net/ondemand</baseURL>   <media

url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_dream_768x428_24.0fps_408kbps" bitrate="408" width="768" height="428" />

  <media url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_dream_768x428_24.0fps_608kbps" bitrate="608" width="768" height="428" />

  <media url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_dream_1024x522_24.0fps_908kbps" bitrate="908" width="1024" height="522" />

  </manifest>

Folder example Example for File “foo” Folder will include :

fooSeg#.f4f (many fragments) foo.f4x foo.meta foo.bootstrap foo.drmmeta (if the stream is configured for

encryption)

Session The player fetches foo.f4m manifest. The player request fragment# HTTP Origin Module locates the required f4f files by

looking at F4X index file and send to client. If the client seeks further, the Origin module searches at

the f4m and again locates the f4f file that fits, starts streaming again.

If the player constantly monitor bandwidth and performance and ask the Origin Module for the proper bit rate associated f4f.