Ipaw113 Erdem.ppt

8/11/2019 Ipaw113 Erdem.ppt

http://slidepdf.com/reader/full/ipaw113-erdemppt 1/40

Data Warehouse GovernanceSimplification with Informatic

Yapi Kredi Bank

Presented by Ahmet Vefa Erdem

14.05.2014



Agenda

About Yapı Kredi Bank

Business Problem/ IT Challenge

Selecting Data Integration Technology

Solutions in Yapı Kredi

Future Plans and Vision

2- 40





Agenda

About Yapı Kredi





4- 40



New Core

Banking

System

2010

Re

for

Br

2

Campaign

Mangment

2001

Data

Mining,Customer

Segment

2002

DWH

DWH

1999

DW

Transform

Project

2003

Merger

between

YapıKredi

and

KocBank

2006

DW

Assessmnt

Project

2011

Developing Data Warehouse in YapıKredi Continous improvement in BI and Data Warehouse at YapiKre

5- 40



Big Changes are Big Challenges for Data Wareho

New Core Banking System

New Credit Card System

New Credit Risk Underwriting

New Treasury System

Bank Merger

(Koçbank & YapıKredi )

New Collection System

6- 40



Data Warehouse Life Story in 15 years

Core

BankingCreditCard ATM Others

Internet

Banking

Data Warehouse

Reporting

ProfitabilityDMart

MIS

Profitability

Business

Intelligence

ODS

ETL

Merchant

Reporting

Credit Card

DMarts

FraudDMart

Cube

CreditCard

Cartography

CreditCard

Branch

Reporting

Operations

Dashboard(Opmis)

Branch

Reporting

Center

Campaign

DMart

Campaign

Management

(Chordiant)

CRMDMart

Potential

Customer

Management

Individual

Banking CRM

CRMPrivate Banking

CRM

Corpotate

Commercial

Banking CRM

Opportun

Manageme

Presentation DataMartsETL

7- 40

http://www.google.com.tr/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=pG2XPd8jgW5sHM&tbnid=VDUfFsm1wqBBNM:&ved=0CAUQjRw&url=http://www.linux-mag.com/id/7888/&ei=m15SU8n2KsnfOdDHgWA&bvm=bv.65058239,d.ZGU&psig=AFQjCNEA-TnoOYSSNrmeXDmvJZDmvWsC9w&ust=1397993309495129

http://www.google.com.tr/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=KYRaF0kws5eINM&tbnid=MmK2BJ-jZFl-nM:&ved=0CAYQjRw&url=http://sc.ha-sys.co.jp/products/goldengate.html&ei=wZ0tU5euOYSMtQaypIHQAw&bvm=bv.62922401,d.Yms&psig=AFQjCNF58li7b6RVh1lYDXM1Sx9Id2_Fiw&ust=1395584678510978



Dmarts

(9TB)

ELTELT

Source1

Source2

Source..

Extraction

Enterprise BI, Reporting

Db Link

Replication

DWH

(75TB)ODS

(20TB)

Sizing and Usage Load Figures of Data Warehous

#of users

5,000#of tables

17,000#of

query/day

27,000

#of ETL

jobs

15,000

Size of

accumulated

data/day

8TB

#of

processed

rows/day

90 billions

#of

query/day

90 millions

8- 40



• Lower user satisfaction

• Lower performance

• High development & maintenance

• Poor Integration capability

• Poor scalability

• Compatibility problems with 3rd p

• Data quality issues

Problems knocked on the door for Data Warehou

9- 40



Diagnose & Cure:

2011 : Data Warehouse Assessment Study

• Created 5 years prog

Data Warehouse and

Assessment study findings are about;

• Data & process governance

• Data integration ( ETL )

• Data model

• Architecture

• Data quality

• Metadata management

10- 40



Agenda

About Yapı Kredi





11- 40



GOAL :

• Select an integration and data ma

tool for data warehouse and its ec

to solve major data management

adressed at the assessment stud

• Then integrate and extend it into tdomains to reinforce enterprise d

integration and governance polici


12- 40



• We evaluated worldwide leading Technol

Integration and Data Management Soluti

• Informatica

• IBM Data Stage

• Oracle ODI

• Abinitio

• SAP Business Object Data Services

• Evaluation team was established from ;

• DW Development Team• DW Administation Team

• DW ETL Operation Team

• Enterprise Data Architecture Team


13- 40



• Functional Requirements were;• Data Integration Tool ( ETL )

• Metadata Management

• Data Archiving and Data Federation

• Test Data Management ( TDM )

• Data Profiling and Data Quality

• Big Data Integration Capability

• Intregration and compatibility require

• Compatibility with Sybase IQ and O• Ability to integrate with Power Desig

Modelling Tool

• Ability to make data lineage analysi

table to Business Objects Reports


14- 40



After 6 months evaluation w


15- 40



Using Informatica

Metadata

ManagerMetadata Manager

Data ArchivingD

Data Subset D

Governance

Information Lifecycle

Management

Test Data Management

Power CenterData Integration(ETL)

16- 40



Agenda

About Yapı Kredi





17- 40





Architecture



ArchitectureInformatica Components in Data Warehouse

ETLETLODS

Source1

Source2

Sourcen

DWH

Enterprise BI, Reporting

Informatica Power Exchange

Archives

Informatica Metadata Manager, Informatica Data Services

TestData

DMart

DMart

DMart

Extraction

Db Link

Replication

Metadata

Manager

20- 40



ETL DevelopmentDefining Methods and Standarts

• Consultancy for ETL Development Best Practices from KO

• ETL development lifecycle standarts

• Documentation standarts

• Deployment methods and standarts

• Security and administration standarts

• Development best practices

• Performance tuning tips

* is the distributor and leader solution provider of Informatica

21- 40

ETL Development



ETL DevelopmentProductivity with Informatica Power Center

• We have reached a certain level of ETL development experiences

• Number of Project : 20

• Number of Developer : 55

• Number of Project Team :4

• Number of ETL Mapping : 1958

• Number of Workflow : 1428

22- 40

ETL D l t B fit f U i I f ti



Declarative Design

Define How: Built-in Templates

DefineWhat

YouWant

AutomaticallyGenerateDataflow

1 2

Conventional Hand coded ETL• Must define every step of Complex ETL Flow

• Requires specialized ETL skills• Significant development and maintenance e

Conventional ETL Design

ETL Development - Benefits from Using Informatica

Declarative Set-based Design with Info• Simplifies ETL development process

• Significantly reduce the learning curve

• Shorter implementation duration

• Enforcement of best practices and standard

• Provides Data lineage and impact analysis

• Doesn’t require specialized low level progra

23- 40

ETL D l t B fit f U i I f ti



Conventional Hand coded ETL• Difficult to organize of a large number o• Developer was not aware of a similar fu

already• There was an aversion to change and tdata sets and jobs are added rather tharefactoring the existing jobs


Declarative Set-based Design with Info

• Automatically generates the Data Flow

sources and target DB

• Reduce maintenance efforts

• Provides full functionality of ETL Autom

ILM, TDM, MM )

24- 40

ETL Development Benefits from Using Informatica

http://www.google.com.tr/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=sMTdJfmGlvcDkM&tbnid=8rg63yUsIyfPbM:&ved=0CAUQjRw&url=http://mikedellaquila.blogspot.com/2013/04/forging-identity.html&ei=kT1iU8DOHs3CtAbUqIDICw&bvm=bv.65636070,d.aWw&psig=AFQjCNHNkok7cCivEQeRoN8kdDuJeqAcbQ&ust=1399033604384207

http://www.google.com.tr/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=y85I3W_k6HoFjM&tbnid=AmNK2PRr0zxu7M:&ved=0CAUQjRw&url=http://www.attunity.com/blog/using-data-replication-handle-big-data-analytics&ei=GT1iU9SCAcretAaVyYDoAw&bvm=bv.65636070,d.aWw&psig=AFQjCNHg1cYEMMNNl_dwNpQVgOgpZcV0uw&ust=1399033396931008



Source

TableFTP Load Transform Validate

Informatica

Mapping

Old ETL Jobs per Table

New ETL Job

per Table

Extract

• Decrease in UC4 jobs

• 4,160 Jobs were deleted in UC4 job scheduler ( 1040 ODS tables

• increase operational efficiency

Job Reduction


25- 40

Test Data Management with Informatica TDM



Test Data Management with Informatica TDMBefore Informatica

Db1

Db2

Db..n

Db1

Db2

Db..n

Operational

Systems (Live)

Operational

( Test & De

DWH

Live

Admin for Test Data

Preparation

DWH

Test &

Dev

IBM Optim was

used as

Enterprise Test

Management Tool

But it could not be used

for DW because of

incompatibility with

Sybase IQ

For DWH, database

administrators were

preparing test data26- 40




est ata a age e t t o at caAfter Informatica ( for DWH Test Management)

Db1

Db2

Db..n

Db1

Db2

Db..n

Operational

Systems (Live)

Operational

( Test & De

DWH

Live

DWH

Test

DWH

Dev

Test Data Subset

Persistent Data

Masking

Informatica TDM was selected by

Data Warehouse Team because of its

rich functionality, high performance

and high compatibility with Sybase IQ

27- 40


http://www.google.com.tr/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=Lx5ehIUL3WQo9M&tbnid=ltx7s0ok4TdHfM:&ved=0CAYQjRw&url=http://dryicons.com/icon/coquette-icons-set/lock/&ei=uMQtU9udLcSUswaMuYHoCQ&psig=AFQjCNGphEAHP4cCdS93yMfh9uPHzdUaxA&ust=1395594784180743




gAfter Informatica ( for Enterprise )

Db1

Db2

Db..n

Operational

Systems (Live)

Db1

Db2

Db..n

Operational

( Test & Dev

DWH

Live

DWH

Test

DWH

Dev

Test Data Subset

Persistent Data

Masking

Replaced IBM Optim with Informatica TDM.

Informatica TDM became «Enterprise Test Data

Management Solution» for Yapı Kredi

28- 40

Using Informatica Data Archiving and Data Virtua






Using Informatica Data Archiving and Data Virtua

• In last 3 years, each year d

extended between %25-40

Warehouse

• High cost, because of usinexpensive storage systems

Warehouse.

• We decided to use data ar

under control size and cost

• We archived 10TB historica

DW data in 2013

• Now, users can access the

using Informatica DataVirtu

• Planing ~10TB archive in 2

RDBMS Compressed

Informatica

Archive File

29- 40

Informatica Metadata Manager in Enterprise Arc

http://www.google.com.tr/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=vnQ7uMhwPu9yUM&tbnid=ZpaipVBPvIop9M:&ved=0CAUQjRw&url=http://www.iconshock.com/vista_icons/12/hallowen/&ei=p35BU5CvC8WAtAaZ-IDAAQ&bvm=bv.64367178,d.Yms&psig=AFQjCNERATK945bhajpvVFaV0QO_bcEwSA&ust=1396887505174539

http://www.google.com.tr/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&docid=sTHG3JrSzsAi_M&tbnid=JNzBt76wYtA4CM:&ved=0CAUQjRw&url=http://oerworkshop.pbworks.com/w/page/File-formats-and-compression&ei=e3hBU86wOMaxtAaS_4CIAQ&bvm=bv.64367178,d.Yms&psig=AFQjCNEigfUme2SRD_NAop_65QJkY4RB0A&ust=1396885844147458



o at ca etadata a age te p se c

Db1

Db2

Db..n

Operational

Systems

Source Systems

Metadata

DataWarehouse

Metadata

DWH

ODS

DMarts

End-to-end impact analysis

MetadataManager

MetadataManager

30K

Tables

17K

Tables

30- 40



Vi i



Vision

• Having efficient, flexible, accurat

Data Warehouse solution

• Using Big Data technologies and

performance analytics methods i

• Empower and generalize Data G

policies into the company by usin

• Use Informatica platform as ente

integration and data managemen

32- 40

Plan :Evolve Data Warehouse



Actions 2011 2012 2013 2014

Make

Assessment

AdaptOrganizatio

n

Modernize

Technology

& Methods

DevelopSystems

Govern

Systems

Sybase Assesment

ETL : Start using

Informatica PC

ChangeOrganization

DW Assessment Study Data Mining Assessment

Give Trainings :TDWI , Power Designer,Informatica, SQL

Define & givedomain architectrole

Select ETL

Tool:Informatica

Change DWDevelopmentLifecycle

Start Using Exadata

for ODS & Datamarts

Upgrade DW:

Sybase IQ 15.4

Select new DW

Platform

Simplify DWStructure

ILM: Start using

Informatica ILM

TDM: Start using

Informatica TDM

Db Design : Startusing PowerDesigner

DW Re-structurin

Start UsingInformatica MetadataManager

Integrate PowerDesigner/Informatica MM /Deployment tool

Create as-is datamodels in PowerCenter

33- 40

Data Warehouse Simplification




• Data Warehouse Re-Engineering

• Platform Change ( Depends on POC results )

• Model Simplification

• Decrease number of tables

• Database consolidation between ODS + DW + Data Marts

• Near-realtime Data Warehouse

• Re-design ETL jobs with Informatica ( 15,000 jobs )

34- 40

As-is Data Warehouse



Data Warehouse

ProfitabilityDMart FraudDMart D.MiningDMart

Core

Banking

Internet

Banking ATM CreditCard

ODS

Logical

DataWarehouse

Campaign

Management

(Chordiant)

Individual

Banking CRMPrivate Banking

CRM

Corpotate

Commercial

Banking CRM

Opportun

Manageme

Potential

Customer

Management

Reporting

MIS

Profitability

Merchant

Reporting

CreditCard

Cartography

CreditCard

Branch

Reporting

Operations

Dashboard

(Opmis)

Operational Systems

Campaign

DMart

Credit Card

DMarts

PresentationDataMartsBranch

Reporting

CenterCRM DMart

OthersBusiness

Intelligence

CRM

ETL

ETL

OperationalData Store

ETL

35- 40







• Business Intelligence Re-Engineering

• Upgrade and migrate Business Objects from BOXI 3.1 into BO

• Universe consolidation ( More than 200 Universes )

• Report consolidation (12K active reports among 80K reports )

37- 40

Big Data



Big Data

• We are in the learning phase

•Planning to become Big Data Enabl

• Starting establish Hadoop platform

• Planinng to define use cases in ;

• IT of things

• Social CRM

• N-Path analysis for customer churn

• Risk management

38- 40

Social CRM



Social CRM

• Yapi Kredi is one of the most act

social media in the country

• Defining Social CRM strategies

• Working on Customer Matching

• Planning Complex Event Proces

• Planing Location Based Services

39- 40



Ahmet Vefa ErdemDataWarehouse & Data Mining Developm

Yapi Kredi Bank

[email protected]

40- 40

Ipaw113 Erdem.ppt

Documents