NetflixOSSSeason 1, Episode 3
Lightning talks
Pytheas
● Web based resource and rich UI framework
● Dashboards
● Web console
● Exploring / visualizing structured + unstructured data
Pytheas - based on OSS frameworks
● Guice● Jersey● FreeMarker● JQuery● DataTables● D3● JQuery-UI● Bootstrap
Pytheas - Modular Design
● Single jar per module● Bundles
○ Guice Module○ Jersey Resources○ FreeMarker Templates○ CSS + Img + JS code○ Properties configuration○ Pytheas Module class
● Minimal bootstrap code to wire modules together into a dashboard
Pytheas
What's next - Angular JS - css/js preprocessing - Scaffold generation
https://github.com/Netflix/pytheas
Conformity Monkey
Help keep your instancesand clusters followingbest practices
Conformity Monkey
● Mark and Notify approach● Customize rules● Customize rule set● Add new rules● Customize execution time and frequency● Can migrate to other cloud platforms
Sample Conformity Rules
● Security Groups● Instance Configuration In Eureka● ELB and ASG mismatch● Cluster Naming Conventions
Conformity Monkey
https://github.com/ Netflix/SimianArmy
Zuul
● Netflix's edge tier○ In place of Nginx, Apache Http or Traffic Server
● Dynamic filters that act on HTTP requests● Filters written in Groovy / Dynamically
updated○ Any JVM language Supported
Zuul within Netflix
ELB
Zuul
API Service
Netflix Services
ELB
Zuul
Streaming Service
ELB
Zuul
Website Service
Request Lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters
How we Use Zuul
● Authentication● Insights● Stress Testing● Canary Testing● Dynamic Routing● Load Shedding● Security● Static Response handling● Isthmus
Zuul Components
https://github.com/Netflix/zuul
http://techblog.netflix.com/2013/06/announcing-zuul-edge-service-in-cloud.html
ICE: AWS Usage Tool
● Background
● Design
● Usage
● Next Steps
Background
● There is increased resource usage in the cloud
● That is very decentralized
● But lacks broad and deep insights about resource usage
ICE Design Approach
● Cloud resource usage has many dimensions: account,
region, zone, product, usage type, user-defined tags
● ICE writer tags each line item in billing file with different
tags
● ICE UI provides a way for users to query cloud resource
usage freely in arbitrary dimensions
ICE Usage
● ICE provides a birds-eye view of large and complex
cloud landscape from a usage and cost perspective.
● ICE UI is easy and intended to be used by:
● Managers
● Reservation Planners
● Engineers
● ICE OSS at https://github.com/Netflix/ice
ICE Next Steps
● Accessing billing files though IAM role and allow
multiple payer accounts
● Create Wiki page and Google Group
● Configurable cost alerts
Genie - Motivation
Genie - Configuration Service
● Abstraction of physical details of back-end Hadoop clusters in the cloud○ Status: up, out of service, or
terminated○ Cluster naming/tagging for job
submissions○ Site-specific Hadoop, Hive and
Pig configurations
Genie - Job Execution Service
● Simple API to run Hadoop, Hive and Pig jobs○ Auto-magic submission of jobs to
the right Hadoop cluster○ Abstracting away cluster details
from clients
Genie - Use Cases Enabled at Netflix
● Running 100% of all SLA jobs, and a high percentage of ad-hoc jobs
● Running nightly short-lived “bonus” clusters to augment ETL processing
● Re-routing traffic between clusters● “Red/black” pushes for clusters● Attaching stand-alone gateways to clusters
http://techblog.netflix.com/2013/06/genie-is-out-of-bottle.html
https://github.com/Netflix/genie
Lipstick
Visualization of Pig workflows at a logical level.● quickly spot common errors / inefficiencies● visually monitor execution● empowers users to support themselves● facilitates communication between
infrastructure team and users
Lipstick
Lipstick in our Architecture
Lipstick UI
Lipstick UI - Pig Workflow Closeup
https://github.com/Netflix/Lipstick
http://techblog.netflix.com/2013/06/introducing-lipstick-on-apache-pig.html
Overview and Roadmap
34 Projects and counting...
Excellent mindshare and vendor adoptionToo hard to get started for individual users
State of NetflixOSS
Where to Start?
● Recipe app(s)● AMI's - Asgard available, more coming!● Hangouts● Workshop(s)● One button install.... not there yet.
Build, Deploy, Tools
Libraries Libraries, Cont-d.
Services ZK / C* Big Data Recipe Apps
Aminator Archaius Karyon Eureka Curator Genie Recipes-RSS
Asgard Blitz4J Netflix Graph
Edda Exhibitor Lipstick Flux Capacitor
GCViz Denominator Ribbon EvCache Astyanax ACME Air
Ice EvCache RxJava Turbine Priam
Simian Army
Feign Servo Zuul CassJMeter
Pytheas Governator Hystrix
Categories
3rd Party demos
Acme Air OSS Sample and Benchmark
● Fictitious airline sample application● Mobile app (iOS/Android) and desktop browser versions
● NodeJS and Java versions with multiple NoSQL data tier impls● Benchmarking framework that allows “Web Scale” runs● Previously run at 4.3 Billion customer facing REST calls per
second (50,000 req/sec) using IBM middleware and cloud
● Re-implemented using Netflix OSS● Microservices architecture● Eureka, Hystrix, Karyon, Ribbon● Asgard
● Run at (peak) ~ 13,000 req/sec – 1.1 Billion per day● Replicate this yourself in ~ 4 hours with the OSS
Learn more at http://bit.ly/acmeairblog and from @aspyker
Eucalyptus, AWS, and NetflixOSS
More details: https://github.com/eucalyptus/eucalyptus/wiki/Netflix-OSS-Tools
Asgard manages autoscaling across both clouds.Eucalyptus/AWS compatibility makes it possible!
SomeNetflixOSS
servicescan run in
either cloud
Simian Army
Edda
Aminator
Priam
Servo30+ NetflixOSS services
...and more on the way!
Confidential and Proprietary
● Multi Data Center Support● Single Dashboard to manage OpenStack & AWS ● Access multiple Tenants in Single Login● One-Stop-Shop to manage
● Compute, Storage, Network● Security● Cloud Templates
● Asgard Framework and Architecture● Why Asgard?● Janitor Monkey for OpenStack● Future Plans● Demo
HIGHLIGHTS
Riot Games rCloud●Aminator +chef-solo provisioner●Asgard●Edda +Security Audits●Simian Army●Ice●Eureka●Archaius + Configuration Service●Ribbon, Karyon, Hystrix●And more
Progress and Updates
NetflixOSS Cloud Prize
State of the Cloud Prize
Forks of github.com/netflix/cloud-prize ~700 Mailchimp signup form complete ~120Public active submissions ~20
Two more months for procrastination!Stickers in return for Mailchimp signup...
Bootcamp/WorkshopsAMIs, Charms, Cookbooks, Ansible Models
More incentives and joint prizes
Make it easier to get started
Juju Charm Championshiphttps://juju.ubuntu.com/charm-championship/Helping solve ease-of-deploy for NetflixOSS
Mark Shuttleworth joins Netflix Cloud Prize judgesAdrian Cockcroft joins Charm Championship judges
Ubuntu Juju
Citrix are offering $10K prize for best integration with NetflixOSS
Apache Cloudstack
First prototype!
Cloud Prize Trophy
Inspiration
Cloudified
Cloud Monkey
It Bleeps!
Designed by bleeplabs.com
Q+A and Demos
Food and Drink in Building B