Evolving Operations for BBC iPlayer
Robert Shield Principal Software Developer, BBC Media Services
Introduction
• Traditional capital investment model • Monolithic codebase • Centralised relational database • Few releases per year • Dedicated Ops team • Storage!
Video Factory - Before
• Many small components • Stateless & scalable • AWS CloudFormation • Continuous Delivery • New components simple to deliver • Storage!
Video Factory - Today
SD VIDEO • 1.3 mb / sec / channel • 21 channels • 2.3 TB per day
Video Factory – Some Numbers
SD VIDEO • 1.3 mb / sec / channel • 21 channels • 2.3 TB per day
Video Factory – Some Numbers HD VIDEO • 4.2 mb / sec / channel • 10 channels • 3.6 TB per day
6 TB per day
Video Factory – Some Numbers
6 TB per day
Video Factory – Some Numbers
In 2 locations
6 TB per day
Video Factory – Some Numbers
In 2 locations
With 2 copies in each location
24TB per day
COSMOS Hand spun Deployment manager One touch deployment Uses AWS CloudFormation on a per component basis GUI & API
Video Factory – Tools
GON SQS front end View on multiple queues Drain, re-spool & download
Video Factory – Tools
Jenkins Continuous Integration Unit & BDD Testing Kick off Cosmos deploys to Integration
Video Factory – Tools
Daskboard Developer led Many different views Uses API of other tools Constantly updated
Video Factory – Tools
• CloudFormation abstraction • Stack fetcher / updater • iSpy • Splunk • Chaos Monkey from @NetflixOSS
Video Factory – Tools
Splunk All components iSpy View pulse of workflow Constantly evolving views on data
Video Factory – Monitoring
Simulcast Console EC2 instance Fed by 4 components Management capabilities
Video Factory – Monitoring
Zenoss Traditional monitoring Now available direct via SQS Tied in to Amazon CloudWatch & Splunk
Video Factory – Monitoring
• Integration between devs & ops • More supportable • Documentation • Standardisation & automation • Smaller releases, more often • Increased buy-in
Video Factory - Operations
• Single region dependency • Single account limits • 3rd party software • SDKs • Monitoring
Video Factory – Areas for Improvement
• Break up monolithic code base • Asynchronous messaging based architecture • Create supporting tools early • Think about monitoring early
– Adapt old tools or implement new ones • Continuous Delivery • Smaller releases more often
– 35 times to Live per week
Video Factory – A Quick Summary
AWS Enterprise Summit
@AWS_UKI
#AWSSummit
