Object Placement in Video Content Distribution Networks

Object Placement in Video Content Distribution Networks

Object Placement in Video Content Distribution Networks

Mohammad Faraji, Kianoosh Mokhtarian

Department of Electrical and Computer EngineeringUniversity of Toronto

December 2011

BackgroundBackground

8 years of video content added to YouTube every day

Terabytes a day; Petabytes a year

Trend is to further accelerate

Higher-quality video streams (currently only 10% are HD)

Content distribution infrastructure

Several datacentres around the world

User request sent to closest datacentre (DNS/HTTP redirect)

MotivationMotivation

Store video files across datacentres (DCs)

Generously replicate all videos on DCs?

Not viable

Growth of data volume >> storage cost

Good News from Measurement Studies

Good News from Measurement Studies

Popularity of video depends on geographical location

More than half of the time, only a fraction from the beginning of video is downloaded

=> Place (partial) video files in selected locations

ModelingModeling

Input: history of user requests (video v for IP address i)

Distance of i to any of datacentres?

Use an Internet Coordinate System (ICS)

Delay(i, j) = Eucledian_distance[ ICS(i), ICS(j) ]

Make tracking of requests scalable

Cluster user IPs into regions in the Eucledian space of ICS

Popularity matrix P[region, video]

Distance matrix D[region, datacentre]

Partial Video FilesPartial Video Files

First minute of video downloaded many more times

Store partial video files

More effective caching

Lower start-up delay

Partial popularity assumed independent of region

Download reports: (v, 1MB), (v, 2.3MB), (v, 0.5 MB), ...

Compress into a few entries for each video (dynamic alg)

PP[v] = (0...1MB, 100 times), (1MB...end, 50 times)

Problem StatementProblem Statement

Assign (part of) each video to one or more DC

Minimize distance of video to user (region), given:

The distance matrix D[region, datacentre]

The expected download pattern P[region, video]

Partial popularity PP[video]

The storage limitation of each DC

Problem HardnessProblem Hardness

Simpler alternatives

Store one video file on a few selected DCs

NP-Complete (min set cover, max coverage)

Store multiple video files on one DC

NP-Complete (knapsack)

SolutionSolution

Maintain a utility matirx U[v, d]

Utility of replicating "the next chunk of" video v on DC d

Auxiliary priority queues

1. Find the highest-utility video v*:

2. Place the next chunk of v* on the best DC d*

3. Update row v* of U, and what the next chunk is for v*

Complexity: O[ (total video replicas) x

(log[# videos] + log[# DCs] + log[max

chunks/video]) ]

Evaluation (in Progress): DataEvaluation (in Progress): Data

File size and length of ~200K videos from [Cheng 2010]

Distances in Internet

Pairwise delay between 2500 nodes from [Wong 2005]

Video popularities

Global: Zipf-distributed (as repeatedly reported)

Local: synthetic

Partial video popularities

Generated according to [Qiu 2010]

Evaluation (in Progress): Results

Evaluation (in Progress): Results

Total delay, given our placement

Delay w/ and wo/ partial file storage

Comparison to simple threshold based distributed caching

Running time

Estimated communication overhead

Take-AwayTake-Away

Benefits of storing partial video files on selected DCs

Future work

Sevral further details for a complete working system ...

Low-overhead collection of (sub-samples of) downloads

Estimate near-future download patterns

Carefully cluster users in a limited num of regions

Solving video placement by multiple nodes

Incremental algorithm; can't shuffle everything every night

Appendix: Previous WorksAppendix: Previous Works

Cooperative web caching

Hierarchical, distributed, hybrid

CDN design (various flavors)

Video caching

On a single cache

To optimize for VCR-like functions