TIAD 2016 : Continuous Integration mesured and controlled

T I A D

Copyright©2016SplunkInc.

ContinuousIntegrationmeasured& controlledStéphaneLapiePresalesEngineer,Splunk Inc.

listentoyourdata®

T I A D

Presenters

2

StéphaneLapiePresalesEngineer– 2YearsatSplunkBasedinParis,FranceSpecializedinAppDelivery,ITOpsandCloudWentfromDev,toOpstoPresales(formercolleagueofAdegbenga atBNPParibas)

Adegbenga AmusaHPCDeveloper– BNPParibas,RiskSystemsBasedinLondon,UnitedKingdomWorkedonGridComputingforthepast10yearsHadafairshareofOpsinmycareer

T I A D

OnceUponaTimeinaFinancialInstitution…

T I A D

Service

AGroupofPeoplewasWorkingonEnhancingSome…

T I A D

• Requirements• ServiceDefinition

• Implementation• Validation&Testing

• ApplicationMonitoring• IncidentManagement

Business

Development

Operations Service

Customers

AndLikeManyServicesThereWere…

T I A D

Service

ThisGroupofPeopleWasBoundtoTransformtheService

T I A D

RiskCalculation

BusinessAnalysts

Java,C++,Cuda

Developers

ApplicationSupport

RiskSystems

StructurersRiskAnalystsQuantAnalystsDataAnalysts

T I A D

RiskCalculationCounterpartyRisk

A B

• ManySimulations• ManyParameters• ManyRegulations• …maybeslowtocalculate

T I A D

RiskCalculation

• Fresh information

• Accurateinformation

• CompliancewithMarketRegulations

• Fullyparametrizedon-demand simulations

• Longtermsimulations

• Quickresults(understandinstant)

• Easeofuse!!!

• Integratedwiththemanyothersystems

HighCustomerExpectationsStructurersRiskAnalystsQuantAnalystsDataAnalysts

T I A D

ANewProjectWasBorn

10

Goal:RebuildtheCalculationEnginetousehighlyparallelcomputingonGraphicalProcessingUnits

• MuchQuickerCalculations

• MuchMoreSimulationsCapabilities

• EasierModelsImplementation

• ReduceHardwareandMiddlewarecost

ProjectCodename:

HPCEHighPerformanceComputeEngine

T I A D

ANewProjectWasBorn

11

GPU/SIMTCPU/SMT

Complexanddifferenttasks

Notsocomplexscalabletasks

fewvery-smartcoresmanycheapercores

CPUvGPUinanutshell

T I A D12

CPUvGPUinanutshell

MonteCarloMethods

SimulationtocalculateValueatRisk

ParallelCalculatio

ns

ANewProjectWasBorn

T I A D

Project’sTimeline

13

FromZerotoHero…

Q2 Q2

Dev Production

2013 2014+

1. FullyIntegrated

2. FullyFunctional

3. IdenticalResultstoLegacy

T I A D

Project’s Timeline

14

FromZerotoHero…

Q2 Q2

Dev Production

2013 2014+

DevOps…totherescue!

FastForward◉

T I A D

Project’sTimeline

15

FromZerotoHero…

Q2 Q2

Dev Production

2013 2014+

FastForward◉

T I A D

Development AppSupport

16

DEV UAT STG PRDTEST TEST TEST

Automation

End-to-EndVisibility

VersionXYZ

VersionXYZ

Version1.0Version2.0

Version3.0.1

Version3.0.1.1

Version4.2.0.1

Business

TEST TEST TESTTEST TEST TESTTEST TEST TEST

DeliveryPipelineOverview

T I A D

Step#1

17

OpenAccesstoEnvironment’sData

T I A D

WhatisDelivered

18

High Performance Compute Engine

Non-persistent Data

PersistentData

Compute FarmREST

Services

GPUServers

CPUServers

ApacheCassandra

+SybaseIQ

+(FileSystem)

OracleCoherence

+DataServices

IBMPlatformSymphony

+CalculationServices

ApacheTomcat

T I A D

Step#2

19

Createindicatorsforeachsoftwarecomponents,oneafteranother…

T I A D

ConsistentandReusableAnalytics

20

Business AppSupport Development

HPCE

Non-persistent Data

PersistentData

Compute Farm

RESTServices

DEV/UATSTAGINGPRODUCTION

T I A D

Step#3

21

Makeitrepeatabletoimproveprogressively!

T I A D

IntegrationwithinRiskSystems

22

Non-persistent Data

PersistentData

Compute Farm

HPCERESTServices

RiskData Bus

Upstream Systems

Downstream Systems

RiskNavigator

RiskData

Warehouse

DataIn/Out

SimulationsRequests

ResultsBrowsing

T I A D

IntegrationwithinRiskSystems

23

Non-persistent Data

PersistentData

Compute Farm

HPCERESTServices

RiskData Bus

Upstream Systems

Downstream Systems

RiskNavigator

RiskData

Warehouse

DataIn/Out

SimulationsRequests

ResultsBrowsing

End-to-EndVisibility

Business

AppSupport

Development

T I A D

Step#4

24

Increasethescopetogetclosertothe“fullpicture”

T I A D

DataSourcesOverview

VersionControl

QA/TestingTools

CM/DeploymentCI/BuildAutomation

23

Applications&Systemslogs

CUSTOMLOGGING

Business AppSupport Development

T I A D26

Visualize ShareAnalyzeExplore

Schema-Freedatacollection& storage

CodeRepository

QA/TestingTools

CM/DeploymentCI/BuildAutomation

ApplicationsMonitoring

Project&IssueTracking

SystemsMonitoring

DistributedStorage&Processing

ScalestoTBs/day

=DatageneratedbyITSystems

MachineData

DataSourcesOverview

T I A D27

T I A D28

DashboardsWalkthrough

T I A D29

Compute Farm

CPUServers

GPUServers

Non-persistent Data

PersistentData

High Performance Compute Engine

RESTServices

ServiceRequestsDevelopment

Requests/Responses

T I A D30

Development

T I A D31

Development

T I A D32

Compute Farm

CPUServers

GPUServers

High Performance Compute Engine

RESTServices

Non-persistent Data

PersistentData

DataIn andOutBusiness

AppSupport

Development

RiskData Bus

T I A D34

Business

AppSupport

T I A D35

Business

AppSupport

T I A D35

AppSupport

Development

T I A D36

AppSupport

Development

T I A D37

Business

AppSupport

Development

T I A D39

Business

AppSupport

Development

T I A D39

PersistentData

Non-persistent Data

High Performance Compute Engine

Compute Farm

CPUServers

GPUServers

RESTServices

CalculationsAnalysis Business

AppSupport

Development

T I A D40

AppSupport

Development

T I A D41

Business

AppSupport

Development

T I A D42

Development

T I A D43

AppSupport

Development

T I A D44

AppSupport

Development

T I A D45

Business

Development

T I A D46

Business

Development

T I A D47

High Performance Compute Engine

PersistentData

Non-persistent Data

Compute Farm

CPUServers

GPUServers

RESTServices

InfrastructureAppSupport

Development

T I A D48

AppSupport

Development

T I A D49

AppSupport

Development

T I A D50

AppSupport

Development

T I A D51

AppSupport

Development

T I A D52

AppSupport

Development

T I A D53

AppSupport

Development

T I A D54

AppSupport

Development

T I A D55

Wrap-up

T I A D

KeyTakeAway

56

1. Don’twithholdinformation,openaccesstoeveryone(%sensitiveness).Shareyourthoughtsandanalysiswiththeotherteams.

2. TaketimetodefineIndicatorsthatmatters.RepeattheoperationwithDev,OpsandBizpeople.

3. Mostanalysisshouldbeconsistentandrepeatable.EventuallyfromtheDevcomputer’tilProductionplatform(containerscanhelpnowadays).

4. Don’trushandtrydoeverythingatonce.You’llgettherequicklyenoughifyoustreamlineyourefforts.

T I A D

LessonsLearned

57

Ø Todaywemanagedtogetan“Holistic”viewofoursystemsandwecan’timagineworkingwithoutasolutionlikeSplunk.WedidallthisveryquicklywithSplunk andinmuchmoredepththanpreviouslydevelopingcustomtools.

Ø Inthepastmostlogswereconsidered“garbage”butmadeaccessibletheyrevealedmanyUserBehaviorswedidn’texpect.Itisoftremendousvaluetounderstandwhatcustomersdo,whichfeaturestheyuseandhow.Thiskindofanalysiscantriggerchangesinthesolution.WestartedtoputLoggingBestPracticesinplacetomakesurethe“garbage”canbe“recycled”;)

Ø EverybodywithinRiskSystemsisusingSplunk forad-hocsearchesandsharingsearchlinksbutveryfewpeopleshareDashboardsandReportstoday.Goingforwardwe’dliketoenablemorepeopleandmakesuretheycancollaborateonimprovingourinternalSplunk Apps.

T I A D

Merci!