DevOps, CD and [Data] Microservices

A NEW PLATFORM FOR A NEW ERA

© Copyright 2014 Pivotal. All rights reserved.© Copyright 2014 Pivotal. All rights reserved.

DevOps, CD & [Data] Microservices

2

Breaking the data monolith

Fred Melo @fredmelo_br

© Copyright 2014 Pivotal. All rights reserved.

$Busin

ess

Developm

ent

QA

Ope

rations

Custo

mer

WHY?

© Copyright 2014 Pivotal. All rights reserved.

$WHY?

Concept Cash

© Copyright 2014 Pivotal. All rights reserved.

How?

© Copyright 2014 Pivotal. All rights reserved.

© Copyright 2014 Pivotal. All rights reserved.

User Mgmt

Order Mgmt

Shipping

Inventory

Payment

Payment

Order Mgmt

Inventory

User Mgmt

Shipping

Contract

Contract

Contract

ContractContract

Conway’s Law and MicroservicesArchitecture x Development teams

© Copyright 2014 Pivotal. All rights reserved.

Deploy

> cf pushCloud

Controller

BBS (message bus)

RouterApp Deployed

Order Mgmt

Contract

© Copyright 2014 Pivotal. All rights reserved.

App App AppApp AppAppScale

AppAppApp

Auto-heal

Migrate Pivotal Cloud Foundry

© Copyright 2014 Pivotal. All rights reserved.

App App AppApp AppAppScale

AppAppApp

Auto-heal

Migrate

Logging

Metrics

Platform Services

Pivotal Cloud Foundry

© Copyright 2014 Pivotal. All rights reserved.

App

Stateless

Ephemeral disk

Short-lived context

“12 Factor"

App App AppApp AppAppAppAppApp

Pivotal Cloud Foundry

© Copyright 2014 Pivotal. All rights reserved.

Statefull

Disk-based storage

Long-lived context

Data Service

???????

App App AppApp AppAppAppAppApp

Pivotal Cloud Foundry

© Copyright 2014 Pivotal. All rights reserved.

Scale

Re-deploy

Migrate

Distribute

Upgrade Update

Pivotal Cloud Foundry

© Copyright 2014 Pivotal. All rights reserved.

User Mgmt

Order Mgmt

Shipping

Inventory

Payment

Payment

Order Mgmt

Inventory

User Mgmt

Shipping

Contract

Contract

Contract

ContractContract

Polyglot Persistence"RDBMs is the king" “The right tool for each job"

One Size Fits All

© Copyright 2014 Pivotal. All rights reserved.

Polyglot PersistenceRDBMs only IMDGs, K-V Stores, Object stores, …

One Size Fits All

Scale-up only

Grid-based HADisk-based HA

Scale-out

Specific hardware Containerized

© Copyright 2014 Pivotal. All rights reserved.

Scale

Re-deploy

Migrate

Distribute

Upgrade UpdateOrder Mgmt

Contract

User Mgmt

Contract

Pivotal Cloud Foundry

© Copyright 2014 Pivotal. All rights reserved.

Order Mgmt

Contract

User Mgmt

Contract

InventoryContract

PaymentsContract

ShippingContract

SQL?Analytics?

BI?

Reporting?

© Copyright 2014 Pivotal. All rights reserved.

Order Mgmt

Contract

User Mgmt

Contract

InventoryContract

PaymentsContract

ShippingContract

Analytical System

Delive

ry F

requ

ency

Degr

ee o

f au

tom

atio

n

Manual & Tooling

Transactions

Analytics

Data Scientist

MDMAnalytical Team

© Copyright 2014 Pivotal. All rights reserved.

How about Data Pipelines?

Anatomy of a Data Pipeline

Source Destination

Pro

cess

ing

Ste

p

Pro

cess

ing

Ste

p

Pro

cess

ing

Ste

p

Pro

cess

ing

Ste

p

Data Data

Monolith

© Copyright 2014 Pivotal. All rights reserved.

Data Pipeline

Scale

Re-deploy

Upgrade

Migrate

Distribute

Update

Source Destination

Pro

cess

ing

Ste

p

Pro

cess

ing

Ste

p

Pro

cess

ing

Ste

p

Pro

cess

ing

Ste

p

© Copyright 2014 Pivotal. All rights reserved.

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

Binding Abstraction Layer

Transport Options

Data Pipeline

Spring Cloud Stream

Breaking the Data Pipeline Monolith

© Copyright 2014 Pivotal. All rights reserved.

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

Transport Options

Spring Cloud Stream

Apache YARN

Breaking the Data Pipeline Monolith

© Copyright 2014 Pivotal. All rights reserved.

From Microservices to Data Microservices

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

Cloud Services

Transport Choices

Spring Cloud Stream

Pivotal Cloud Foundry

Scale

Auto-heal

Logging

Metrics

Registry and Discovery

Circuit Breaker

Distributed Config

© Copyright 2014 Pivotal. All rights reserved.

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

Pivotal Cloud Foundry Services

Data Microservices

Spring Cloud Data Flow

Data Microservices Orchestration

Pivotal Cloud Foundry

© Copyright 2014 Pivotal. All rights reserved.

SpringBoot app

Contract

SpringBoot app

Contract

Spring Cloud Data Flow

JSON Filter Transform Aggregate

CustomHTTP

Sample pipeline

Pivotal Cloud Foundry

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

Contract

SpringBoot app

ContractSpringBoot

appContract

SpringBoot app

Contract

SpringBoot app

Contract

Deploy

© Copyright 2014 Pivotal. All rights reserved.

Your own infrastructure

Pivotal Cloud Foundry

Microservices

Spring Cloud Services

GemFire

Data Microservices

Spring Cloud Data Flow

RedisMySQL

Greenplum

Polyglot persistence

PostgreSQL

HTAP

A NEW PLATFORM FOR A NEW ERA