1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature. Mission: imPossible Scaling in the Cloud 1 Saturday, May 11, 13
Dec 16, 2014
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Mission: imPossible
Scaling in the Cloud
1Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
The Core: Message Bus100% Cloud Native
2Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
What Cloud Translates To
3
Small to mid size instances
Lack of Control Over Physical Resources
No delay in expanding
Limited Storage Options
Redundancy is Easy Redundancy is Necessary
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
What Does Your Data Look Like?• Log Data Classic OLTP• By Nature, it’s Distributed • Long & Mid Term Storage• Eventual v. Immediate Consistency • Speed of Retrieval -Classing OLAP Warehouse
4Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Requirements
5
• No Data Loss• Ability to Pause, Promote, and Update Seamlessly• Do Not be the Bottleneck • Allow for Out of Sequence Events• Restate and Correct Analytics
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Tools• Chef• Liquibase• HornetQ• Rep Mgr• Postgres
6Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Distributed OLTP
7
Disparate event types can be sent to separate instances.
Within the same event, parallel writes don’t require synchronous multi-masters.
Batch Process Writes with Copy Into
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Vertical Sharding- Inheritance is your friend!
8
One parent per event, sharded by time, allows for efficient querying, individual backups, and general sanity.
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Vertical Sharding- Inheritance is your friend!
9
Constraints: Timestamp not DateEvent ID (pkey)Explain Analyze!
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Vertical Sharding- Inheritance is your friend!
10
WHILE startday <= endday endday LOOP tomorrow := startday + interval '1 day'; index_name := 'idx_name' || replace(startday::text, '-', ''); tname := 'event_' || to_char(startday, 'YYYY_MM_DD'); EXECUTE 'create table ' || quote_ident(tname) ||'() inherits(event)'; EXECUTE 'ALTER TABLE ' || quote_ident(tname) ||' ADD CONSTRAINT createdck CHECK (created >= ' || quote_literal(startday) ||' AND created < '|| quote_literal(tomorrow) ||')'; EXECUTE 'CREATE INDEX '|| index_name ||' ON '|| quote_ident(tname)||'(act_id) ; startday := tomorrow;END LOOP;
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Close The Shard
11
FOR tbl IN SELECT t.table_name FROM information_schema.tables t WHERE LEFT(t.table_name, 6 ) = 'event_' AND t.table_name NOT IN (SELECT table_name FROM information_schema.constraint_column_usage WHERE constraint_name = 'idcheck') AND REPLACE(REPLACE(t.table_name, 'event_', ''), '_', '-')::DATE < NOW()::DATELOOP EXECUTE 'SELECT coalesce(min(event_id), 0), coalesce(max(event_id), 0) FROM ' || quote_ident(tbl) ||';' INTO smin, smax; EXECUTE 'ALTER TABLE '|| quote_ident(tbl) || ' ADD CONSTRAINT idcheck CHECK (event_id BETWEEN '|| smin || ' AND ' || smax ||' )';END LOOP;
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Star & Snowflake OLAP
12
Because API calls can’t time out.
Partition-able on fact, account - what works for you.
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Build Your Snowflake
13
• Pull Aggregates• Add/Update base fact• Cascade Updates Through Dimensions
Quartz Triggered Calls Three Phase Process:
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Build Your Snowflake
14
Driven by Sequential IDPosition Lookup, Pull, & Update in a Single
TransactionSave State for Performance Analysis
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Aggregate on Pull
15
CREATE TABLE source_epoch AS SELECT * FROM dblink('dbname=src host=127.0.0.2 port=5432 user=rickastley password=never_gonna_give_you_up', 'SELECT some, thing, COUNT(id), DATE_TRUNC(''hour'', TO_TIMESTAMP(date)), MAX(id) AS id FROM events WHERE id > 50823461 GROUP BY 1, 2 ORDER BY id) AS t(some TEXT, thing, INT event_count INT, hour TIMESTAMP, id INT)
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
The Defense
16
Bugs for the entire system will find their way through to your warehouse. Do not always trust your data.
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
What It All Looks Like
17
Writer
Writer
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
What It All Looks Like
18
Writer
Writer
OLTPTypeA
OLTPTypeB
OLTPTypeB
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
What It All Looks Like
19
Writer
Writer
OLTPTypeA
OLTPTypeB
OLTPTypeB
OLAPWH
Type1
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
What It All Looks Like
20
Writer
Writer
OLAPWH
Type1Query Head
OLTPTypeA
OLTPTypeB
OLTPTypeB
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
What It All Looks Like
21
Writer
Writer
OLAPWH
Type1Query Head
OLTPTypeA
OLTPTypeB
OLTPTypeC
OLTPTypeAOLTPTypeA
OLTPTypeC
OLTPTypeC
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
What It All Looks Like
22
Writer
Writer
OLAPWH
Type1Query Head
OLTPTypeA
OLTPTypeB
OLTPTypeC
OLTPTypeAOLTPTypeA
OLTPTypeC
OLTPTypeC
OLAPWH
Type1Query Head
Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
http://news.messagebus.com/2013/01/03/mission-impossible/
Questions?
23Saturday, May 11, 13
1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.
Thank You!24
Saturday, May 11, 13