An Iterative Approach to Service Oriented Architecture

Iterative Service Oriented

Architecture

Eric Saxby@sax @ecdysone @sax

From thisFrom this

To thisTo this

Why should you listen to me?

Proprietary and Confidential



■ I have many leather bound books



■ I have many leather bound books

■ My apartment smells of rich mahogany



■ Application developer Many various technologies. Rails for ~6 years."

■ Recent focus has been operational"

■ Such acronyms, so buzzwordTDD, Agile, DevOps, SOA, Cloud, IaaS

■ Broken a surprising amount of things in a short time"

■ Work at Wanelo

Wanelo? Whaaa?

Wanelo? Whaaa?

■ Social network for shoppingIf you buy things on the Internet, you are our demographic

Wanelo? Whaaa?


■ Many millions of users

Wanelo? Whaaa?


■ Many millions of users

■ Databases with billions of records

More importantly

More importantly

■ Iterated from one large Rails codebase to services

More importantly


■ Open source as much as we can

More importantly



■ Scaled Ruby where people say Ruby can’t be scaled

More importantly



■ Scaled Ruby where people say Ruby can’t be scaled

■ Very small team

But are you really here for success?

“Good judgment comes from experience. Experience comes from bad judgment.” — Proverb"

— Theo Schlossnagle"

http://www.infoq.com/presentations/Scalable-Internet-Architectures

“The Anti-Pattern Goldmine”

■ That I may or may not have worked for"

■ One giant, tangled Rails 2 codebase our vendored code had vendored code!

■ 30 engineers racing to commit code as fast as possible"

■ Success == large project launched on specific date

A completely hypothetical company

Releases a mess

■ Code branches for months at a time"

■ Monthly releases turn to weekly releases"

■ Change list distributed and explained to executives"

■ Multi-hour deploys"

■ Annual code freeze means 3 month code bonanza

Over time, things get worse…


■ Rough estimates == Firm deadlines



■ Shortcuts only way to meet deadlines




■ As soon as projects finished, teams split to meet new deadlines




■ As soon as projects finished, teams split to meet new deadlines

■ Estimates based on worst-case timelines… never worst-case enough

Technical DebtTechnical Debt

Technical DebtTechnical Debt

What did I think I learned?

■ When programming is not fun, THERE IS A PROBLEM"

■ SOA IS SOLUTION TO ALL OUR PROBLEMS"

■ DEVOPS IS THE ANSWER

Now for a role change

■ I move into the operations team renamed to “not the operations team”BECAUSE DEVOPS"

■ Working on systems automation"

■ Engineering: SOA is the only way forward"

■ Product: SOA is that thing getting in the way of cranking out features

If at first you can’t succeed


■ Take a firm stance



■ Take some deep breaths



■ Take some deep breaths

■ Cry like a baby

New features would be services, or not built

■ New login feature as separate application"

■ Data service to manage Enterprise software home-grown Rails app

“This will be hard and we’ll screw up, but we have no other option”

Okay, so what is SOA again?

One site composed of multiple applications

Internal External

UIAPI only

HTTP

Synchronous

TCP

Asynchronous

And why would I want this?

■ Scale organization"

■ Each team manages their own code"

■ Each team deploys their own code"

■ Outsource some code without giving keys to the kingdom


■ Scale codebase"

■ Performance tightly coupled to workload"

■ Hide complexity behind clean interfaces"

■ Use tools best suited to task at hand


■ Scale tests"

■ Small codebase == small test suite"

■ but… costly learning curve to test service boundaries

At some point, the cost of not doing services outweighs the cost of doing

services

Ok… back to our protagonists

■ Data service 9 months "

■ Complex state transitions between applications"

■ No major data problems"

■ Dependent applications ambiguously successful

EngineeringEngineering

We didn’t break anything!We didn’t break anything!

ProductProduct

9 months??????9 months??????

Ok, ok, ok, but what about?…

■ 3 months 2-4 engineers + the “DevOps” team"

■ Staging servers available in 2-3 weeks 2 months later… no one had deployed to them!

■ Released with feature flagExtra few weeks to break it in production

But then…

■ Worked exactly as intended"

■ Very resilient to failures!

■ User metrics generally successful

EngineeringEngineering

Success!!!1!!Success!!!1!!

ProductProduct

3 months??????????3 months??????????

How many engineers does it take to Service Oriented Architecture?

What is success?

■ Engineering: Success"

■ Product: Failure!

■ Who wins?

Nobody

But let’s go deeper…

■ Why did we need SOA?"

■ Engineering did not trust product"

■ Engineering did not trust that we could fix the shortcuts required to do our job

And even deeper…

■ Product did not trust Engineering to deliver on promises"

■ Product accountable for features, not for quality

Trust

So what did we learn?


Fuck it, SOA FROM THE BEGINNING next time


Fuck it, SOA FROM THE BEGINNING next timeX

Agile is more than just sprinting

■ Iterate, using data to guide next iteration"

■ Deploy soon, to start collecting data"

■ Red — Green — Refactor"

■ SOA will not solve organizational problems

SOA does not “fix” your code quality

SOA is a tool to help with our technological pain points

Iteration is how we create and deploy changes

Iteration

■ Small changes deployed quickly"

■ Feature flags"

■ Each step prioritized and completed when necessary"

■ But when SOA?

Performance

Code complexity will be less than not doing SOA

New code completely unrelated to old code

With the caveat…

■ May go into old codebase short-term"

■ Deploy today is better than deploy next week"

■ Requires TRUST that this will be fixed later

So… Performance

■ Site getting slower"

■ DBs becoming I/O bound on disk"

■ User growth/activity is crazy, and increasing

■ One table completely outstripping others"

■ Reads from table killing caches for others"

■ We’re in the cloud, so max resource limits

■ Site will stop working, soon"

■ Don’t want to stop feature development"

■ 10 engineers

Step 1: Isolate the data

Remove ActiveRecord relations

class Product < ActiveRecord::Base has_many :saves end

Remove ActiveRecord relations

class Product < ActiveRecord::Base def saves Save.where(product_id: self.id) end end

Pretend it’s in a different database

class Save < ActiveRecord::Base establish_connection "saves_#{Rails.env}" end

production: database: production_db host: 10.0.0.100 saves_production: database: production_db host: 10.0.0.100

■ Each change deployable"

■ Tests + staging servers will show where things break"

■ Watch out for DB connection count!

Switch to using different database

■ Create read-only replica"

■ Site maintenance mode"

■ Push new database.yml pointing to replica"

■ Promote replica to be master"

■ Restart unicorns"

■ Disable maintenance

■ Performance may now be fine"

■ Next step can come later, whenever necessary or convenient

Step 2: Isolate the interface

■ How many ways do you access table?"

■ What is long term growth?"

■ What are long term goals?

How would you shard if you could shard?

■ Wanelo == Saves"

■ Saves are viewed by"

■ Product"

■ User

Refactor to create an API

class Product < ActiveRecord::Base def saves Save.where(product_id: self.id) end end

Refactor to create an API

class Product < ActiveRecord::Base def saves Save.by_product(self.id) end end

Refactor to reduce the API

■ Remove redundancy"

■ Add tests"

■ You will break things very soon!

Step 3: Extract DB adapter

Move finders into a module or base class

class Save include SavesClient end

module SavesClient def self.included(other) other.send(:attr_accessor, :id, :product_id) other.extend ClientClassMethods end !

module ClientClassMethods def by_product(*args) adapter.new(self).by_product(*args) end !

def adapter @adapter ||= SavesClient::DbAdapter end end end

■ Finders call an adapter"

■ Adapter wraps database calls

Why an adapter?

■ Adapter is your feature flag"

■ Deployable as in-place refactor"

■ Can be replaced later with different adapters

module SavesClient class DbAdapter def self.close_connections # Check in the database connection, # since we're shutting down this thread SavesService::Save.clear_active_connections! end !

def by_product(*args) relation :by_product, *args end !

def relation(method, *args) SavesClient::AdapterRelation.new(self, SavesService::Save.send(method, *args)) end end end

■ Because THREADS"

■ State changes should always return a new instance"

■ #all, #first, #etc are called on Relation instance"

■ Relation will delegate to adapter

def relation(method, *args) SavesClient::AdapterRelation.new(self, SavesService::Save.send(method, *args)) end

Adapter methods return a Relation

■ ActiveRecord model, which moves into the client"

■ Insert your favorite DB adapter here"

■ Important part is that this is hidden from the including class

def relation(method, *args) SavesClient::AdapterRelation.new(self, SavesService::Save.send(method, *args)) end

Scope of query is called on…

Save.by_product

DBAdapter

AdapterRelation.new

relation.all

adapter.all

ActiveRecord::Save Postgres

Instantiate client instance(s)

Critical to deploy at this step

■ 100% guaranteed to make mistakes"

■ 100% guaranteed to have missed something"

■ Cost of fixing this is low at this step

Step 4: Launch service app

Insert your framework of choice

■ HTTP? JSON? MessagePack?"

■ What is easiest to develop?"

■ Transport mechanism maps to an adapter in the client"

■ Can be replaced later

brief interlude

Q: Why should you have deployed by now?

A: Because the client is much more complex than the server

So………..

Your bugs are in the client

Server code will be dictated by the client

■ By service launch time, whiteboard diagram will have changed"

■ Understanding will have changed"

■ Don’t write your server before you understand the needs of the client

So, back to the server

■ We used Sinatra + Oj"

■ Sinatra is awesome!

Step 5: Use the service

Adapter is your feature flag

switch between DB and service"

=="

switch between DB and HTTP adapters

Save.by_product

HTTPAdapter

AdapterRelation.new

relation.all

adapter.all

HTTPClient JSON to Hash

Instantiate client instance(s)

Retrospective

■ Isolate data"

■ Isolate interface"

■ Extract DB adapter into gem"

■ Launch service"

■ Switch to service adapter

Tests?

Integration vs Unit

■ Spin up server, a la Sunspot?"

■ “Fake” adapter"

■ In-memory store (Array)"

■ Need unit + integration"

■ Can delete redundant tests later

Development

■ Need to actually run service"

■ Failures are in interactions"

■ Foreman + Subcontractor"

■ Each app needs isolated Bundler env"

■ Don’t mix dependencies

But what about a new app?

How can we iterate on something that doesn’t exist?

■ Deploy as early as possible"

■ Chef/Puppet/Salt/Capistrano/CFEngine"

■ Focus on being able to change quickly"

■ Isolate and understand differences

Feature flags are more than on/off

■ Off"

■ Asynchronously write"

■ On

Integrate deep before wide

■ Code path from beginning to end"

■ Edge cases/Errors will bubble up"

■ Avoid complexity or cleverness in design"

■ It will come on its own

Takeaways

■ Barriers to SOA are barriers to all development"

■ Figure out how to iterate"

■ Hexagonal architecture can bubble up from organization’s needs"

■ Priorities/needs change. Facilitate this.

Thanks!

sax ecdysone sax

especially to everyone involved in this work, of which I was one small part

An Iterative Approach to Service Oriented Architecture

Software

leather bound

large rails

services open

end end refactor

vendored code

social network

productiondb

teams split