Evolving Operations for BBC iPlayer

Evolving Operations for BBC iPlayer

Robert Shield Principal Software Developer, BBC Media Services

Introduction

•  Traditional capital investment model •  Monolithic codebase •  Centralised relational database •  Few releases per year •  Dedicated Ops team •  Storage!

Video Factory - Before

•  Many small components •  Stateless & scalable •  AWS CloudFormation •  Continuous Delivery •  New components simple to deliver •  Storage!

Video Factory - Today

SD VIDEO •  1.3 mb / sec / channel •  21 channels •  2.3 TB per day

Video Factory – Some Numbers

SD VIDEO •  1.3 mb / sec / channel •  21 channels •  2.3 TB per day

Video Factory – Some Numbers HD VIDEO •  4.2 mb / sec / channel •  10 channels •  3.6 TB per day

6 TB per day

Video Factory – Some Numbers

6 TB per day

Video Factory – Some Numbers

In 2 locations

6 TB per day

Video Factory – Some Numbers

In 2 locations

With 2 copies in each location

24TB per day

COSMOS Hand spun Deployment manager One touch deployment Uses AWS CloudFormation on a per component basis GUI & API

Video Factory – Tools

GON SQS front end View on multiple queues Drain, re-spool & download

Video Factory – Tools

Jenkins Continuous Integration Unit & BDD Testing Kick off Cosmos deploys to Integration

Video Factory – Tools

Daskboard Developer led Many different views Uses API of other tools Constantly updated

Video Factory – Tools

•  CloudFormation abstraction •  Stack fetcher / updater •  iSpy •  Splunk •  Chaos Monkey from @NetflixOSS

Video Factory – Tools

Splunk All components iSpy View pulse of workflow Constantly evolving views on data

Video Factory – Monitoring

Simulcast Console EC2 instance Fed by 4 components Management capabilities

Video Factory – Monitoring

Zenoss Traditional monitoring Now available direct via SQS Tied in to Amazon CloudWatch & Splunk

Video Factory – Monitoring

•  Integration between devs & ops •  More supportable •  Documentation •  Standardisation & automation •  Smaller releases, more often •  Increased buy-in

Video Factory - Operations

•  Single region dependency •  Single account limits •  3rd party software •  SDKs •  Monitoring

Video Factory – Areas for Improvement

•  Break up monolithic code base •  Asynchronous messaging based architecture •  Create supporting tools early •  Think about monitoring early

–  Adapt old tools or implement new ones •  Continuous Delivery •  Smaller releases more often

–  35 times to Live per week

Video Factory – A Quick Summary

AWS Enterprise Summit

@AWS_UKI

#AWSSummit