How to run a bank on Apache CloudStack

How to run a bank!on

Me:

Gérard de Vos

MCE @ Schuberg Philis 2008-current. Previously @ Shell, Ziggo, POIS, TNO, …

Now: “full stack”, *-lead. Then: infrastructure, hardware, HPC, Linux, provisioning, web & such

@gr4rd

!

!

!

“Schuberg Philis is an innovative business technology company. We focus on the mission critical applications that our customers and society rely on 24/7.”

Customers include:

What we had• 2009: new internet savings bank!

• Way-of-working 2009: !

• Dedicated DC space, !

• Dedicated servers, !

• Dedicated network, !

• Dedicated team!

• Growth: 0€, 0 customers -> 4B€, 120k customers!

• “Classic” application stack

Trigger1. Contract to expire in <1 year

2. Evaluated current environment:

• Dev environment(s). Not enough, clashes.

• Data refreshes. Too hard <> not done often enough.

• Different environments are different.

• And the usual suspects: lack of flexibility, underutilization of resources, huggable snowflake servers.

3. Time moved on:

• Agile development is reaching the enterprise.

• Agile infrastructure is not just for startups & unicorns anymore.

• "The Lean Startup" is for everybody.

Way-we-work now• Dedicated team (we kept something the same!)

• Shared infra

• MCC: Apache CloudStack

• Shared services

• Chef, chef cookbooks

• Github enterprise

• SBP is more Lean & Agile & Devopsy

• Contribute

• Software is eating the world

• Focus on the value chain. Reduce waste

source: Adrian Cockcroft http://www.slideshare.net/adriancockcroft/qcon-new-york-speed-and-scale

Public sitehttp://www.leaseplanbank.nl

Secure sitehttps://sparen.leaseplanbank.nl

TITLE

DRAWN BY

DESCRIPTION

FILENAME PAGE

DATE

Layer 7 application diagram 13-Sep-12

1 of 1Leaseplan bank L7 v 2.7.vsd

LPB

SBP

LeasePlan Infrastructure Services(LPIS) Dublin - Ireland

WebLogic

lpbpapp1/2active/standby

lpbpws101/102active/active

lpbpws1/2active /standby

lpbpapp101/102active/active

lpbpsan1/2

High available SAN (FCAL) via synchronous mirroring

BACK OFFICE THIRD PARTY INTERFACES

Site to Site VPNManaged by LPIS

File system FC Rep FC UBS

APPLICATION LANDSCAPE

https

Direct Banking

email2sms

Alphen a/d Rijn

First time password

FC Gateway(active/active)

FCUBS(active/standby)

Once a month postcode file is retrieved

sms

ssmtp

SFTP

Phone and emailCustomer requests

Manual reportingLogius/DigiPort interface tbd

SFTP

Transparant twin datacenter in active/active set up

Hippo

Apache

http

http

BKR FC DB

Site to Site VPN

Back office and Customer Care Center Services

Activestandby

Standbyactive

https

Operations

Legenda

...customer department server/os componentdatabase

jms

LeasePlan Infrastructure Services(LPIS) Dublin - Ireland

Direct BankingBank Admin GUI

1. Direct Banking:- Bank Admin GUI- Super Admin GUI

2. Core Banking- UBS Admin

3. CMS incl preview to content staging web site

4. OBIEE reporting

FTP-S

Email (Smtp)

WebLogic

FCDB

lpbpmx1/2active/active

Apache

(s)smtp

emailemail2sms

ssmtp

mailAlmere

Home Office users

Marketing

ICT

Finance & Control

lpbprep2/1active/standby

Apache

Postcode Table

Rensageg file transfer

CRM

Verificatie Informatie Systeem

Customer screening

KYC file

Scoring and Business rule System (SBS)

Verification of new customers

FLEXCUBE Core Banking and Gateway

Oracle databaselpbpd1/2

active/standby

Central Storage Array Network (SAN) for SFTP, application, database and some management servers

sms

http

Tomcat

Hippocontainer

Securesite

Sorrysite

KYC file Equensfiles

Back Office Front End Services

OBIEE App Server

Oracle Reporting

VPN

VPN

VPN

FLEXCUBE Direct Banking

MySQLHippo CMS

CMS and Public Web Content

Contentpublication

Data upload / KYC download

http:7002

http(s)http(s)sftp

smtp

httpmysql

SQL*Net V2 SQL*Net V2SQL*Net V2scp

http

FCAL FCAL FCAL

Email (Smtp)

/ VIS

Other files

equens putKYC get

x� equens getx� KYC putx� and other file

exchange

smtpsmtp

NMUT/betOPD/batch VerwINF

FTP-S (get + put)

equensPayment Services

For CMS + staging and OBIEE

LeasePlan Bank team

LeasePlan Bank

http

Publicsite

HTTPS

Upload list of customers

lpbprep1/2active/standby

Savings calculator XML

smtp

Antivirus + antispam

emailcustomers

LPB office

Email 2 sms

Multi homed internet acces

Hilversum

Direct BankingBank Admin GUI VPNDirect Banking

Site to Site VPN

x� BankAdmin interface for CCCx� BankAdmin + SuperAdmin

interface for LPB BackOffice

DMZ for mail, public and secure web sitesCustomersCustomer Contact Center

VPN

VPN

We came up with this• Private storage for datastores

• Private hypervisors for transaction processing systems

• Kept existing internet facing network connections & kit

• Shared cloud for

• Dev/dev2/../test(UAT) environments with anonymised data

• Admin env. monitoring, deployment, etc.

• Shared MCC zone:

• Network: I don’t care,

• Hypervisors: I don’t care

• CloudStack Primary & secondary storage: I don’t care

Shopping list

Shopping list• Private customer zone:

• Two pods -> 2 datacentres

• Network: Arista 10GbE Top-of-rack,

• Hypervisors: HP DL380G8 8core, 192GB

• CloudStack Primary & secondary storage: NetApp

• NFS storage for datavolumes: NetApp metroclustre

• Runs the production and preproduction environments

The challenges• New tech

• CloudStack & SDN

• git

• Chef

• Many others

• New thinking

• WayWeWork (highly in flux)

• Shared infra

• Shared svcs

• Design-for-failure vs Enterprisey apps

The nice things• Infra-as-code. We now think things go slow when

it takes >10 minutes to go from nothing to functioning server.

• Re-re-re-rebuilds. Process maturity, Cookbook maturity, DR/BCP maturity & confidence.

• Infra is almost a non-topic in discussions with the customer around new applications & services.

• SBP cloud HW performance. CPU/mem & IOPS/mbps

EndOfDay 2hr -> 45m

• MCC matured a lot.

• WayWeWork is maturing.

20/20 hindsight• Pushed/pulled the shared services team more. They

are providing a service, not tech.

• Sales/mgt/engineers overestimated what IAAS brings.

• Sales/mgt/engineers underestimated what IAAS brings.

• Put more of the stack into shared cloud.

• DBMS redundancy higher in the stack. (e.g. ASM vs metroclustre)

What do we need help with?

• How do we run in multitenant environments and have everything secure?

• How do we explain this to auditors so they agree?

Thank you!