Rovio © 2016 Confidential
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Rovio © 2016 Confidential
Rovio © 2016 Confidential
Rovio’s Infrastructure The New Path ForwardPaul Winston VillacortaSenior Cloud Engineer, Cloud OperationsRovio Entertainment
Rovio © 2016 Confidential
HybridCloud
Private
Cloud
PublicCloud
3rd Party ManagedPrivate Cloud
Rovio © 2016 Confidential
Private Cloud Public Cloud
Rovio © 2016 Confidential
ORCHESTRATION
TRAFFIC MANAGEMENTTRAFFIC MANAGEMENT
TRAFFIC MANAGEMENTSERVICES
TRAFFIC MANAGEMENTDATABASE
Amazon CloudServices
External ServiceProvider
CDN
ExternalMonitoring
Incident Management
System
CONTINUOUSINTEGRATION
CONTINUOUSDELIVERY
Rovio © 2016 Confidential
EXTERNAL TRAFFIC MANAGEMENT
SERVICES
INTERNAL TRAFFIC MANAGEMENT
MICROSERVICES SERVICES MICROSERVICES
DATABASE TIER
ORCHESTRATION / AUTOMATION LOGIC
CACHING COMPONENTS STORAGE
MON
ITOR
ING
LOG
COLL
ECTI
ON
INTE
RNAL
DNS
OPERATIONAL JOB SCHEDULER
VIRTUALIZED
Rovio © 2016 Confidential
IT IS HUGE!!
Rovio © 2016 Confidential
200+DatabaseInstances
60+Services
700 - 2000Running
Instances
{API}
45000+API Requestsper second
3BAnalyticsProcessedevery day
Rovio © 2016 Confidential
AUTOMATION
WE DON’T ALWAYS AUTOMATE STUFFBUT WHEN WE DO IT’S END-TO-END
Rovio © 2016 Confidential
AUTOMATION• Scalability – Automatic node creation
• Availability – Automatic node replacement
• Efficiency – Automatic intelligence
Load Testing, Disaster Recovery, Rolling Upgrades, Compliance, Portability, etc…
Rovio © 2016 Confidential
AUTOMATION TOOLS• Scripting / Programmable Utilities• Puppet, Fabric, Ansible• Docker, Packer• Vagrant
Rovio © 2016 Confidential
AUTOMATION
• Builder Script is written in Python• Utilizes REST APIs• 1,100 lines of code• Week of development + testing
• 5 seconds to create new project• We have saved 70 steps and 140 clicks
Continuous Integration /Continuous
Delivery
Rovio © 2016 Confidential
AUTOMATION
• It’s GUI based• It’s API driven• It supports standalone CLI• It’s cloud aware• It’s cluster aware• It’s inside the cloud• It works like a charm
Rovio’s Orchestration
Server
Rovio © 2016 Confidential
AUTOMATION
OrchestrationServer
Cloud Provide
r
New Node
Cloud Properties
Rovio © 2016 Confidential
AUTOMATION
OrchestrationServer
Internal DNS
Monitoring
Ready Node
PACKAGE
PUPPETMANIFEST
S
Rovio © 2016 Confidential
AUTOMATION
OrchestrationServer
Internal DNS
Monitoring
Ready NodeTraffic
Management
Rovio © 2016 Confidential
LESSONS LEARNED
METRICS AND VISIBILITY - SMALL FAULT MARGINS - LEARNING FROM MISTAKES
Rovio © 2016 Confidential
Metrics and Visibility• Sensible Metrics• Actionable Items• Metrics that can be visualized and aggregated
Rovio © 2016 Confidential
Metrics and Visibility
Rovio © 2016 Confidential
Metrics and Visibility
Rovio © 2016 Confidential
Small fault margins• Small mistakes build up quickly• Small mistakes are unpredictable• Small mistakes costs money!
Rovio © 2016 Confidential
Learning from mistakes• NEVER leave critical components UNATTENDED!• Do it RIGHT from the BEGINNING!• Do a proper root-cause analysis and publish the findings
in your organization.
Rovio © 2016 Confidential
CHALLENGES
Rovio © 2016 Confidential
CHALLENGES• Modernizing legacy applications• Cloud Providers configuration changes• Security• Legacy nodes• AZ Distribution• Architectural Challenges
Rovio © 2016 Confidential
Thank You!
Related Documents
