Jan 23, 2018
Microservices: A foundational approach for fully managed cloud data analytics!Sam Lightstone!CASCON 2016!
CASCON 2016
Cloud Data Services @IBM – we have a few!
CASCON 2016
Cloud Data Services Table Stakes!
‣ Things I Need!‣ Compute, Network and Storage: plentiful and
reliable
‣ Data services that can scale from tiny to massive in users, data, and associated resources.
‣ Scale my engineering team!
‣ Service Resilience
‣ For the provider (us): System management and extensibility by mere mortals
‣ Pricing and billing that starts small and scales seamlessly
‣ Security (You can trust us with your data)
‣ Quality data movement
‣ Things I Don’t!‣ Salespeople
‣ “Build an app in 5 Minutes”
‣ The last 5% of performance
3
CASCON 2016
The good ol’ days…!
1:1 relationship between the application and the backend implementing it. The Monolith – one thing to deploy, and manage.
CASCON 2016
Microservices to the rescue!
‣ Small autonomous services that work together
Abstract implementation
behind API
Decentralize most
processing
Design for the
consumer
Deploy components independently
Constraint the scope of failure
(limit blast radius)
Monitor aggressively
Model a business domain
Automation (practice &
culture)
Microservices!
CASCON 2016
Starting with an opinion!
‣ Microservices ROCK! ‣ But not without problems and limitations
CASCON 2016
Why microservices work!
Source: “Service Discovery and Registration in a Microservices Architecture” F. Khaliq, F. Sanches.
CASCON 2016
Single Responsibility Principle!
‣ Group together things that change together.
‣ Corollary: Separate things that change for different reasons.
CASCON 2016
The truth will set you free!
‣ Prefer true stateless, virtual, small, highly available, microservices ‣ No concern for placement or latency
‣ Minimal specifics on physical layer
‣ Stateless is best
‣ Trust the infrastructure to connect microservices efficiently
You just need to believe….!
CASCON 2016
And here is the truth…. !
‣ Cloud Data Services are data intensive – and have one or more components that are massively stateful.
‣ BigData, in particular can store 10,000GB – 5,000,000GB in a single tenant
‣ Serious data makes ephemeral, virtual, redundant, replicated, …. all sound quaint. ‣ How many redundant copies of my 5 Petabytes do you want to store? ‣ 5 Petabytes randomly placed? ‣ 5 Petabytes with how much access latency?
‣ Performance makes multi-tenancy hard ‣ Share my cores with how may people? ‣ Other tenants are doing what????
CASCON 2016
An example for a generic cloud data service!
Billing Data Movement
Container Service
User Console
User Mgmt.
Service Broker
Data Engine
Spark Analytics
R Studio
Visualization
Monitoring (fault and
usage)
Key Mgmt. (encrypt)
PROBLEM
PROBLEM
CASCON 2016
PROS & CONS!
‣ Code simplicity! Easier for humans to build, understand.
‣ Update microservices independantly
‣ Enables continuous delivery
‣ Smaller, lower risk, updates to part pieces becomes possible
‣ Easier to test and deploy
‣ Enable resiliency where possible, even if other services are lacking
‣ Complicates DevOps
‣ Some microservices need to be reasonably collocated – where latency matters.
‣ Monitoring & logging more vital than ever, and harder to coordinate
‣ Small & autonomous fiction: can everything be made small?
PROS! CONS!
13