Data: Migrating, Distributing and Audit Tracking Michelle Ayers, Advisory Solution Consultant Email: [email protected]@visionsolutions.com.

1visionsolutions.com

Data: Migrating, Distributing and Audit Tracking

Michelle Ayers, Advisory Solution ConsultantEmail: [email protected]

Mark Scanlon, UKI Regional Director

October 2013

Vision Solution Offerings

visionsolutions.com 3

Data Migration

Traditional Approach – ‘Big Bang’AllUsersOffSystem

UsersBackOnSystem

Monday AM

EODFriday SundaySaturday

DOWNTIME

Backups Copy Data Transform Data Testing/verify

Issues:• Hope you can complete all steps in available time window…

• Data volumes becoming ever larger – takes longer to copy and convert data

• Business becoming more 24 x 7 - reduced downtime window opportunities

• Final testing/verification limited to what remaining time you have• Many stages are dependant on completion of previous stage

visionsolutions.com 4

Data Migration

Alternative Approach – ‘Continuous Replication’

visionsolutions.com 5

Data Migration

Benefits

visionsolutions.com 6

Data Migration

More Complex:

SourceData

TargetData

DT ShareCaptures Changes

DT ShareCaptures Changes

DT ShareApplies trans to Staging Database

DT ShareApplies trans to Staging Database

IBMi Oracle

Administration & Monitoring(Director, MonCon & Console)

LAN / WAN

Data path

CustomPrograms

CustomPrograms

CustomPrograms

CustomPrograms

Staging Data

Staging Data

DT ShareCaptures Changes

DT ShareCaptures Changes

DT ShareApplies trans to Staging Database

DT ShareApplies trans to Staging Database

LAN / WAN

Data path

visionsolutions.com 7

Data Sharing

Typical Approaches:

Application Database

Apps

Website

ERP Application

Business Intelligence

User Queries

• Lots of demands on the central database• All users/applications using same data therefore all should be consistentHowever:• Performance issues – too much access of same data• Users running inefficient SQL statements• Increased security risks from many different communication jobsCan lead to productivity issues for core business processes

visionsolutions.com 8

Offloaded data warehouses have been around some yearsHowever:• Data often loaded by batch overnight• May even need to reload whole tables each nightIssues:• Data is out of date quickly• Every day the database grows a little bit bigger, therefore each night the

load job takes a little bit longer…

Data Sharing

Typical Approach Using Offloaded Data Warehouse:

Application Database

ERP Application

Business Intelligence

User Queries

Data Warehouse

visionsolutions.com 9

Data Sharing

Alternative Approach – ‘CDC’ Change Data CaptureCustomer Story – Red Wing Shoe Company - USA

OFFLOAD QUERY & REPORTING PROJECTOFFLOAD QUERY & REPORTING PROJECT

Manufacturing

MS SQL Server

Query reports

Data Warehouse load

Double-Take ShareReal time CDC replicationwith transformation

Reduce CPU and I/O overheadon production systemimprove user response times

Many cost effective tools available on MS SQL serverplatform for query reports

Data is already partially‘scrubbed’ and availablefor loading data warehousesand data marts withoutperformance impact on production system

Production System Offload Query System

IBM System i DB2Lawson M3 (Movex)

visionsolutions.com 10

Data Sharing

Alternative Approach – ‘CDC’ Change Data CaptureCustomer Example – Application Interface

Bi-directional replication

IBM System i DB2Manufacturing ERP

Introduced automated manufacturing inspection tracking and also faster Data Warehouse Load.

MS SQL Server

Double-Take ShareReal time CDC replicationwith transformation

Production line information is transformed and sent to ERP

Production SystemAPPLICATION INTEGRATION PROJECTAPPLICATION INTEGRATION PROJECT

Query reports

MS SQL Server

Double-Take Share Replaced batch reload of data warehouse with DT Share creating incremental extracts of last changed row, and deleted transactions.

Third Party Inspection Application

visionsolutions.com 11

Data Auditing

Traditional Approach to Field Audit Tracking

Application Database Audit FilesBefore and after changes captured via embedded program codeOr trigger programs

• Older applications may be using embedded code to capture changes – high overhead to maintain and ensure nothing missed

• Trigger based capture can have database overhead though should ensure call changes captured

• ERP provided solution can be limited:• May only support pre-selected tables/columns• Restricted data filtering capabilities

visionsolutions.com 12

Data Auditing

Alternative Approach – ‘CDC’ Change Data Capture

FIELD AUDIT TRACKING PROJECTFIELD AUDIT TRACKING PROJECT

Application Database Audit FilesDouble-Take ShareReal time CDC replication captures before and after valuesIBMi uses journal receiver entries

• Capture changes made within or outside the ERP application• Optionally add row/column filtering• Choose columns from any database table, including local modification• Target audit files are custom designed• Add target columns to hold trans type, timestamp, user id• Access to audit data via custom queries/reports• IBMi M3 java database has journaling switched on by installation – DT

Share can use these journal receivers.

Target auditfiles can be heldin same databaseor different, sayMS SQL Server

visionsolutions.com 13

Double-Take Share

14visionsolutions.com

Thank You!

visionsolutions.com facebook.com/vsichannel

youtube.com/vsichannel

linkedin.com/companies/6542twitter.com/vsichannel

businesscontinuitytoday.com