StatistiX An Introduction - Oracle DWH · StatistiX An Introduction ... or on a specified date (expiration date). What ??? ... Oracle Database, Informatica, MicroStrategy, ControlM,

StatistiX

An Introduction

11. Oracle Business Analytics Big Data und Data Warehouse Konferenz 2016 15. und 16. März 2016

What do we do?

Deutsche Börse Group 1

Eurex Classic /

PNDEEXXetra / PNDEurex T7Prisma

RE / EDRE /

PNDWSS

Calypso Sec Lending

StatistiX Development

File Interface

CEF

Cash

CEF

Eurex

Eurex C7Eurex

Repo

Direct Real-Time Connection and / or File Interface

Prisma & Risk Analytics

MMPM, HypMi, PERD II

Regulatory BillingRICA & OTC

IRS Tools

External

Reports

Frontend

Reporting

Internal

Reports

Billing Input &

Reporting

Regulatory

Reporting

ODBC

Access

Ad-hoc

Analysis

Data

Vendors

Real-Time / File Interface

Business Support

EMail

EEX BDWH

StatistiX Operations

What do we do?


Eurex Classic /

PNDEEXXetra / PNDEurex T7Prisma

RE / EDRE /

PNDWSS

Calypso Sec Lending

StatistiX Development

File Interface

CEF

Cash

CEF

Eurex

Eurex C7Eurex

Repo

Direct Real-Time Connection and / or File Interface

Prisma & Risk Analytics

MMPM, HypMi, PERD II

Regulatory BillingRICA & OTC

IRS Tools

External

Reports

Frontend

Reporting

Internal

Reports

Billing Input &

Reporting

Regulatory

Reporting

ODBC

Access

Ad-hoc

Analysis

Data

Vendors

Real-Time / File Interface

Business Support

EMail

EEX BDWH

StatistiX Operations

The Challenge

4

IT and Business


Options contract

A contract that, in exchange for the option

price, gives the option buyer the right, but

not the obligation, to buy (or sell) a financial

asset at the exercise price from (or to) the

option seller within a specified time period,

or on a specified date (expiration date).

What ???

int i = 1;

i += i++ + ++i;

What's the value of i?

What ???

… ROI … Time To Market … TCO …

!!! JUST DO IT !!!

Dr. Murray Bozinsky Gordon Gekko

The Common Answer


Business<< Analysis, Usage >>

Functionality<< Analysis, Specification, Test >>

IT<< Development, Operations >>

Functional

Requirements

Technical

Requirements

Software

Systems

Business Skills Technical Skills

The StatistiX Philosophy

7

The StatistiX Philosophy


Business<< Analysis, Usage >>

Functionality<< Analysis, Specification, Test >>

IT<< Development, Operations >>

Functional

Requirements

Technical

Requirements

Software

Systems

Business Skills Technical Skills

Shifting

Development

towards

Business

Qin Dynasty(221 BC - 207 BC)

The Qin Dynasty was the first Chinese united empire

• Unification by conquest of different kingdoms

• Key success factor (simplified): Standardisation

• Technology – industrially produced easy to use weapons

• People – function based equally trained soldiers


Transform

Many Kingdoms One United Empire

Efficient division of labour

Reproducible quality

Easy to produce

Effective

Two days of training

Success Factor Standardisation: Technology


Simple Construction

Industrially manufactured

Success Factor Standardisation: People


Farmer Crossbowman

Transform

Need of a huge amount of soldiers trained for different functions

Translation to IT

Enable Business People

to develop IT Systems


The Mission


Be the data warehouse of Deutsche Börse AG.

Be our longtime memory.

Be storage and analytic/reporting facility of operational systems.

Be consumer of continuously increasing data.

Be reliable source system for operational systems.

Keep your old principles: high quality, flexibility, dynamic, innovative, no costs.

That‘s Life!


A New Warehouse Is Born!StatistiX Rising


Intention: Replace

user individual Excel analytics/calculation based on unconsolidated data

by

centralised, audit proof reporting environment based on common sense dataset.

Tools: Oracle Database, Informatica for ETL, MicroStrategy for reporting

Most important inbound data:

Trading data XETRA, EUREX, tick data CEF, masterdata WSS

Most important outbound data:

Typical standard reporting (turnaround, ranking, …), index data

Size: Small.

A New Warehouse Is Born!StatistiX Rising


Datacenter IDevelopment/Simulation

Datacenter IIProduction

Oracle

DB CEF

Fallback

DBAG

source

systems

Oracle

DB StatistiX

InformaticaInbound

fileserver

MicroStrategy

Outbound

fileserver

World

DBAG

CEF

MSI Web for

DBAG users

Oracle

DB StatistiX

Informatica

Inbound

fileserver

MicroStrategy

Outbound

fileserver

Oracle

DB CEF

WorldWorld

DBAG

source

systems

MSI Web for

DBAG users

StatistiX Being A ToddlerWhat Am I Doing On This Planet? Growing, trying out capabilities, acquire new skills - evolving!


Focus: Integrate more sources and gather more data.

Improve StatistiX infrastructure.

Improve data model.

Provide more and new services to DBAG users and DBAG customers.

Tools: Oracle Database, Informatica, MicroStrategy

New: ControlM for orchestration, Webmethods for GUI and workflow applications


Other entities from trading systems XETRA, EUREX, tick data CEF, masterdata WSS,

outside DBAG data (e.g. Reuters)


Enhanced reporting (turnaround, ranking, …), index data, content for DBAG public

website, customer individual and personalised standard information products

Size: Small.

StatistiX Being A ToddlerWhat Am I Doing On This Planet? Growing, trying out capabilities, acquiring new skills - evolving!




Oracle

DB StatistiX

InformaticaInbound

fileserver

MicroStrategy

Outbound

fileserver

Oracle

DB CEF

FallbackWorld

DBAG

source

systems

DBAG

CEF

Webmethods

ControlM

DBAG

user apps

MSI Web for

DBAG users

Oracle

DB StatistiX

InformaticaInbound

fileserver

MicroStrategy

Outbound

fileserver

Oracle

DB CEF

WorldWorld

DBAG

source

systems

ControlM

Webmethods

DBAG

user

apps

MSI Web

for DBAG

users

DBAG

source

systems

StatistiX As A ChildGrowing And Learning


Intention: Integrate more sources and more data into StatistiX.

Stabilise StatistiX infrastructure, partialy refresh it.

Further improvement of data model.

Enhance set of services to DBAG users and DBAG customers.

Tools: Oracle Database, Informatica, MicroStrategy, ControlM, Webmethods



outside DBAG data (e.g. Reuters)


Further enlargement of information product prortfolio (MMPM),

new index information products (products for other burses), content for DBAG public

website, customer individual and personalised standard information products

Size: Midsize

StatistiX As A ChildGrowing And Learning




Oracle

DB StatistiX

InformaticaInbound

fileserver

MicroStrategy

Outbound

fileserver

Oracle

DB CEF

FallbackWorld

DBAG

source

systems

DBAG

CEF

Webmethods

ControlM

DBAG

user apps

MSI Web for

DBAG users

Oracle

DB StatistiX

InformaticaInbound

fileserver

MicroStrategy

Outbound

fileserver

Oracle

DB CEF

WorldWorld

DBAG

source

systems

ControlM

Webmethods

DBAG

user

apps

MSI Web

for DBAG

users

DBAG

source

systems

Basics On StatistiX Infrastructure


Almost all StatistiX tools are running on dedicated hardware.

CPU based software licenses count sockets/cores of underlying hardware, not the

amount assigned to the virtual machine (ControlM, MicroStrategy, Oracle).

Tool requirements are extremly high, so they need huge amount of resources.

Either virtualisation software cannot provide this or it is not reasonable to put vm on

top of hardware, that will be used almost completly by one application (e.g. Informatica).

Some smaller tools and applications based on them, are running in virtual machines.

In the past, large applications were put to SUN/Solaris infrastructure, only Microsoft

software (MS Analysis Services) was running on windows.

Various types of SUN/Solaris machines were in use (V240 – M4000).

Former database server had been used together with internal disks as database storage.

In child phase of StatistiX, first techrefresh took place, mainly to replace old hardware.

Basic structures of old infrastructure were kept!

Upgrade from Oracle 8 to Oracle 9!

Basics On StatistiX Infrastructure


Focus: New challenging projects:

Clearstream Infosmart (very high data volume and up to this unknown requirements

for delivery reliability, billing project, and others

Creating own tools to provide functionality missing in commercial software products

(e.g., TFP = TargetFileProcessing – later on by ControlM AFT).

Improve services and reliability.




outside DBAG data (e.g. Reuters), data of Clearstream operational system Dataserver


Enhanced reporting (turnaround, ranking, …), index data, content for DBAG public

website, customer individual and personalised standard information products,

Clearstream operational reporting suite

Size: Large.

StatistiX Teenage YearsGrow, try different things, sometimes fail and get up again.






Oracle

DB StatistiX

InformaticaInbound

fileserver

MicroStrategy

Outbound

fileserver

Oracle

DB CEF

FallbackWorld

DBAG

source

systems

DBAG

CEF

Webmethods

ControlM

DBAG

user apps

MSI Web for

DBAG users

Oracle

DB StatistiX

InformaticaInbound

fileserver

MicroStrategy

Outbound

fileserver

Oracle

DB CEF

WorldWorld

DBAG

source

systems

ControlM

Webmethods

DBAG

user

apps

MSI Web

for DBAG

users

DBAG

source

systems



Challenge:

Continuously increasing data volume and workload bringing infrastructure to it‘s

limits.

Improve reliablity, whereas shortage of resources is obvious.

Impelmentation of desaster recovery and hardware failover are still incomplete.

Implemention of huge projects in parallel to several smaller ones, besides

operating the warehouse and maintaining it, using same staff for all of these task.

Doing major or minor upgrades on software in use.

Consequence:

Next techrefresh ahead!

Change from SUN/Solaris systems to x86 hardware and Red Hat Enterprise Linux

Partially upgrading tool software (either because of new OS or just using the opportunity

for cleanup and neat restart .

Focus on increasing resources (make things larger) on tool infrastructure.

Protection against hardware failure or loss of datacenter based on using existing hardware

and manual processes.

Database server with Direct Attached Storage (HP P2000). Upgrade to 11g R2. Using ASM

for storage management.

Protection of database with Oracle Data Guard and server, normaly used for simulation

database.

Next Techrefresh To The End Of Teenage Time


StatistiX Teenage YearsDatabase TechRefresh (Still In Use)




Oracle

DB StatistiX

P2000

P2000

P2000

P2000

P2000

P2000

P2000

P2000

production database

standby

slow disk area

production database

primary

slow disk area

simulation database

Oracle

DB SIM

Oracle

DB PRD

standby

Oracle

DB PRD

standbyData Guard

fast disk area fast disk area

P2000

P2000

P2000

P2000

P2000

P2000

P2000

P2000

Oracle

DB StatistiX

StatistiX Getting Operational.Do more, at once, at same time, realtime and at no costs.


Focus: Change StatistiX from data warehouse to operational system with realtime/near

realtime processing and integration into the production loop.

Major project: store and use data from new EUREX trading system T7,

store and report data from risk system PRISMA, receive and use data from new

EUREX clearing system C7


New: AMQP (bi-directional), Java programming, T7 client, C7 client


T7, C7, PRISMA


Reporting and analytics on T7, C7 and PRISMA data

Size: Large to very large.



Main requirements:

Downtime due to loss of service (hardware or datacenter fails) < 2h.

Separation (hardware, not only logically) of simulation and production.

Production possible in different datacenters.

No data loss on physical or logical database failure.

StatistiX Getting Operational.Overview StatistiX Environments And Basic Components As Of Today.


STX DEV PRISMA DEV STX/

PRISMA ACC

STX/PRISMA

SIM

STX PRD PRSIMA PRD

WH

Databases

StatistiX File Server

Control

M

Inbound

File Server

Source

Systems

Outbound

File Server

Target

Systems

Repository

Database

Informatica

MatLab

Mathematica

Java Engines

OS Scripts

MicroStrategy

Webmethods

Non-S

tatistix

Com

ponents

StatistiX Components in each environment

Inbound

Source

Systems

Outbound

Target

Systems

STX Environment

World

StatistiX Getting Operational.Overview StatistiX Environment Production As Of Today

Deutsche Börse Group 30N

on-S

tatistix

Co

mp

on

en

ts

W

o

r

l

d

Datacenter Hausen Datacenter Bergen-Enkheim

Applications

Hot

dwhpas4/6

dwhpbill1

ControlM Server Hot

Datacenter

DWH_PROD

Applications

Hot

dwhpas7/8

Webmethods

Hot

Dwhpmet1/3

MicroStrategy

IS Cold

frpstat1

MicroStrategy

Web Cold

frpstat3

MicroStrategy

IS Hot

frpstat2

MicroStrategy

Web Hot

frpstat4

MicroStrategy

NC Cold

frpstat8

MicroStrategy

NC Hot

frpstat7

dwhsbill1

ControlM Server

Cold Datacenter

DWH_PROD

NFS Hot +

Informatica

Options FB

dwhpnfs

Informatica

Options Hot

+ NFS FB

dwhpasrt

Veritas Cluster

Dwhpdb3 Oracle DB Primary

dtb

Storage Array P2000

Dwhsdb3 Oracle DB Phys. Standby

prisma dtb

Storage Array P2000

Morepha5/6 Oracle DB

dwhprep

MatLab

Hot

frpires03

Prisma

DTB/NTA

Data Guard

Orchestration +

file transportFile transport

Dwhpprisdb1 Oracle DB Primary

prisma

Storage Array P2000

Ndwhpdb2 Oracle DB Primary

nta

Storage Array P2000

StatistiX Getting Operational.Overview StatistiX Environment Simulation As Of Today

No

n-S

tatistix

Co

mp

on

en

ts

W

o

r

l

d

Datacenter Hausen Datacenter Bergen-Enkheim

Applications

Hot

dwhsas4/6

dwhpbill1

ControlM Server Hot

Datacenter

DWH_SIMU

Applications

Hot

dwhsas5/7

Webmethods

Hot

vdwhsmet1

MicroStrategy

IS/Web/NC

Hot

frsstat1

dwhsbill1

ControlM Server

Cold Datacenter

DWH_SIMU

NFS Hot +

Informatica

Options FB

dwhsnfs

Informatica

Options Hot

+ NFS FB

dwhsasrt

Veritas Cluster

dwhsdb4/5 Oracle DB Phys. Standby

dtbsimu

Hitachi HUSVM Hitachi HUSVM

Morepha5/6 Oracle DB

dwhsrep

Orchestration +

file transportFile transport





Oracle

DB StatistiX

production database

standby

slow disk area

production database

primary

slow disk area

simulation database

Oracle

DB SIM

Oracle

DB PRD

standby

Oracle

DB PRD

standbyData Guard

fast disk area fast disk area

Oracle

DB StatistiX

StatistiX Getting Operational.Overview StatistiX Environments And Basic Components To Be Implemented

Hita

ch

i H

US

VM

SA

N s

tora

ge

1P

T

Backup II

Backup I

Data

Hita

ch

i H

US

VM

SA

N s

tora

ge

1P

T

Backup II

Backup I

Data



Architectural issues with database(s):

Several databases for special purpose (T7, prisma).

Special databases are small or midsize, but same effort as for main database.

Projects very often need data from their own database, but additional data from

main database (database links -> poor sql).

Non-optimal usage of resources (free storage, whereas somewhere else needed,

unused capacities because usage during daytime or nighttime only).

Current infrastructure does not match needs of usage (hot-warm-cold data,

growing rappidly related to past, sandboxing, …) and scalability.

High initial costs and maintenance.

Life without appliance is possible, yet pointless?Freely adapted from Loriot’s famous quote ‘Life without a pug is possible, yet pointless!’


Where will wego from here?

Stay with

what you

know!

Something

else?

Suicide?Get a

hippster?

Life without appliance is possible, yet pointless?Freely adapted from Loriot’s famous quote ‘Life without a pug is possible, yet pointless!’


Yes you can!

Not an easy way.

Effort rises permanently.

Higly complex environment.

More and more a specialist job.

Small and midsize environment.

If it makes life easier, you need not!

Reduce complexity.

Establish consolidated homogenious environment.

Integration instead of separation.

Clearly defined responsibility.

Keep up to date with new technology and it‘s

advantages.

Life without appliance is possible, yet pointless?Future of StatistiX?


Exadata for limited in size database.

BDA for off-loading old, staging only or mass data.

One access to data using SQL and database security.

Next steps: use BDA in native way for controlled batch environment.

Future: Get business specialists

Thank You !

Axel Goldbach Manfred Wollmann

Deutsche Börse Group in-factory GmbH

[email protected] [email protected]

mailto:[email protected]

Backup

Content


04 The Challenge 20 Reaching the Goal

07 The StatistiX Philosophy 24 Abstraction and Visual Tools

13 The Pillars of StatistiX 37 StatistiX Architecture

The Pillars of StatistiX

13

The Pillars of StatistiX


Choosethe

rightPeople

Let theDatabase

be theCentreof your

Universe

UseProblemDomainspecific

Tools

StatistiX

Choose the right People


StatistiX

That's what it's all about:

The right mix !

Let the Database be the Centre of your Universe


Working with Relational Databases is Mathematics:

𝐴 𝑇𝑎𝑏𝑙𝑒 𝐶𝑜𝑙𝑢𝑚𝑛 𝑖𝑠 𝑎 𝑆𝑒𝑡 ⇒𝐴 𝑇𝑎𝑏𝑙𝑒 𝑖𝑠 𝑎 𝐶𝑟𝑜𝑠𝑠 𝑃𝑟𝑜𝑑𝑢𝑐𝑡 𝑜𝑓 𝑆𝑒𝑡𝑠 ⇒𝑇ℎ𝑒 𝐶𝑜𝑛𝑡𝑒𝑛𝑡 𝑜𝑓 𝑎 𝑇𝑎𝑏𝑙𝑒 𝑖𝑠 𝑎 𝑆𝑒𝑡 𝑜𝑓 𝑇𝑢𝑝𝑙𝑒𝑠 𝑤𝑖𝑡ℎ 𝑎𝑛 𝑂𝑟𝑑𝑒𝑟 𝑑𝑒𝑓𝑖𝑛𝑒𝑑 𝑜𝑛 𝑖𝑡 ⇒𝑇ℎ𝑒 𝑄𝑢𝑒𝑟𝑦 𝐿𝑎𝑛𝑔𝑢𝑎𝑔𝑒 𝑆𝑄𝐿 𝑖𝑠 𝑆𝑒𝑡 𝑇ℎ𝑒𝑜𝑟𝑦∧A𝑙𝑙 𝑃𝑒𝑜𝑝𝑙𝑒 𝑤𝑜𝑟𝑘𝑖𝑛𝑔 𝑓𝑜𝑟 𝑆𝑡𝑎𝑡𝑖𝑠𝑡𝑖𝑋 ℎ𝑎𝑣𝑒 𝑎𝑛 𝐴𝑐𝑎𝑑𝑒𝑚𝑖𝑐𝑎𝑙 𝐵𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑𝑖𝑛𝑐𝑙𝑢𝑑𝑖𝑛𝑔 𝑀𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑠

q.e.d.

Use Problem Domain specific Tools (1)

An Excerpt


Goal Criterion Example

Focus on “what to do”

not on “how to do”Highest possible abstraction

Don’t think about how to bring master data

together prices, just do it

Use data, don‘t think about how and where it is

stored

Multi source/target types in one processing

step

Table from Oracle, DB2, Sybase, Greenplum

etc. together with Data structures like CSV,

XML etc.

Configure technical constraints,

don‘t develop themIt‘s there, just configure it

Buffering, caching for processes and

parallelisation of them

Make business processes visible and

controllable

Realtime visual views on batch and manual

processesBatch monitoring, BPM

Cost transparency Extensive database based loggingCalculate a service’s costs on KPIs like CPU

time, processed rows, used storage etc.

Know dependencies between systems Database based repositoriesAnalyze the chain from orchestration to which

job writes to which table


An Excerpt


Problem Domain Tool

High performance data loading of huge volumes Informatica

Implementation, execution, and monitoring of processes running jobs on different servers with

different operation systemsControl-M

On business level defined business intelligence and reporting (scheduled or ad-hoc) MicroStrategy

Complex calculations with modern mathematical libraries Mathematica / MATLAB

Implementation, execution, and monitoring of business processes webMethods BPM

Integration of various systemsInformatica,

webMethods IntegrationServer



Third Party Tools In-house Development

Oracle, Java, Shell Script, Network etc.

Development

Technology

Business

Abstraction

~98%

~2%

Activities at StatistiX

Reaching the Goal

20

Reaching the Goal


Enable business people to develop systems

-

Abstraction of technology to functionality

Avoid bugs

-

Configure frameworks instead of develop code

People are eye-minded (75%)

-

Use visual development tools

People get bored by routine

-

Avoid repeating work

People should not focus on solving technical challenges

-

Use all-in-one solutions for specific problem domains

Advantages of Abstraction and Problem Domain Specific Tools


Rapidly rising learning curve

Less bugs

Systems are self-descriptive People are more motivated

People earlier become productive

Advantages of extensive Logging


Job

Where did it run?

When did it run?

How long did it run?

How many of which resources

did it consume?

Peaks in Server Usage

Determine Bottlenecks

Fulfil SLAs

Hard Disk Space, CPU Usage,

Number of Rows processed

Development

Resource Planning

Billing

Operations

Measurement Measurement Areas

Abstraction and Visual

Tools

22

Visual Tools


webMethods

Control-M

Informatica

Mapping

What is ETL?


SAbstract

Sources

Abstract

Targets

Ab

str

actio

n

Feed

Database

File

Ab

stra

ctio

n

Feed

Database

Extract Transform Load

File

Area of Development

Focus on “what to do” not on “how to do” (1/3)

Sum up the traded volume per day and instrument


Get all prices

Get listing master data

Get instrument master data

Get ISIN for listing

Bring ISIN together with the price

Aggregate price

per day and instrument

Aggregated volume

Sources are abstract – could be a database table, a CSV or XML file etc.




How to bring listing master data together with the prices




Aggregate byHow to aggregate

High Performance Data Loading

Informatica enables us to process

up to 550.000 records per second

in Eurex MMPM calculation

• Measured in a calculation step (so it is not only „simple“ loading)

• Informatica offers all capabilities of parallel processing with configuration only

• No technical knowledge about multi-threading, pipes, shared memory etc. necessary


Visual Modelling of Executable Business Processes

4-Eyes-Principle


TriggerFailed step

Successfully

completed step

Step not

yet started

Condition

Central error handlingProcess name

Visual Programming

Freefloat Calculation for DAX Indices


Projects

Service

Service implementation

Service call

Signature of the service to call

Orchestration of Batch Job Nets


Job successfully ended

Job currently running

Job waiting for executionJob net overview

Job failed

Mathematical Computation (1/2)

When was the last time you spent a pleasant evening

in a comfortable chair, reading a good program?

Jon Louis Bentley

Mathematician and Computer Scientist

Communications of the ACM, 1986


Mathematical Computation (2/2)


Isn‘t that more understandable

than the lines above?

Documentation and

„Code“ at the same

place tells morely

a story than being

technical description

of what to do

Same formula after symbolic simplification

MicroStrategy BI Platform


StatistiX Architecture

37

StatistiX Architecture – The Outer View


StatistiX

GUI

File

Feed

Inbound Outbound

GUI

Feed

File

Email Email

ESB ESB

Database

A

p

p

l

i

c

a

t

i

o

n

A

p

p

l

i

c

a

t

i

o

n

Interfaces


Type Tool Inbound Outbound

GUI

HTML

webMethods CAF

MicroStrategy Web Interface

Feed

CEF

OFI

ESB Prisma

Mail

Lotus Notes

SMTP

File

Upload via GUI

Download via GUI

ftp

Filesystem

StatistiX Architecture – The Inner View


Database Database

Tools and Layers


Orchestration

Other

Applications

Technical

Abstraction,

Native

Access

Database

Business

Abstraction

Informatica

Feature

Strength

Oracle

StatistiX

OLTP

Engine

StatistiX

Realtime

Interface

Mathematica

Control-M

Informatica

MicroStrategy

webMethods

Mathematica

StatistiX An Introduction - Oracle DWH · StatistiX An Introduction ... or on a specified date (expiration date). What ??? ... Oracle Database, Informatica, MicroStrategy, ControlM,

Documents