® IBM Software Group ©IBM Corporation IBM Information Server Transform – DataStage
Jan 26, 2016
®
IBM Software Group
©IBM Corporation
IBM Information Server
Transform – DataStage
®
IBM Software Group
©IBM Corporation
Why “Transform?”
IBM Software Group
Why Transformation?
Business Driver: Single View of Corporate Data
Projects Related to Information Infrastructure Application integration
Platform migration
On-demand transformation and correction
Application re-engineering and migration (ERP to CRM)
Decision Support (BI, DW, Data Marts) Opportunity (discover new revenue sources)
Control (Fraud detection, inventory)
Regulatory compliance -SOX, BASEL, Money Laundering
Portals
Balanced scorecard dashboards, BAM
Business Goals
IT Initiatives
Information Integration
3
IBM Software Group
Transformation Pain
Multiple sources for the same entity
Lack of standards or consistent semantic meanings across systems
Embedded business intelligence
Evolving transformation requirements
Need for batch and real-time and service oriented architectures
Extreme data volumes!
Business rules for resolving data conflicts
Ownership and accountability
Zero re-use of skills and processes
4
IBM Software Group
How Is This Being Done Today?
Hand coding: Java, C, C++, VB, .NET, COBOL, 4GLs…
Spreadsheet “farms”
Early generation ETL tools
Competitive products
5
IBM Software Group
IBM Information ServerDelivering information you can trust
Understand
Cleanse Transform Deliver
Discover, model, and govern information
structure and content
Standardize, merge,and correct information
Combine and restructure
information for new uses
Synchronize, virtualize and move information for in-
line delivery
ParallelProcessing Connectivity Metadata DeploymentAdministration
Platform Services
Support for Service-Oriented Architectures
6
IBM Software Group
7
The IBM Solution: IBM Information ServerDelivering information you can trust
Understand
Cleanse Deliver
Parallel ProcessingRich Connectivity to Applications, Data, and
Content
IBM Information Server
Unified Deployment
Unified Metadata Management
Transform
WebSphere DataStageComplex transformation for simplified data
exchange and reduced coding
IBM Software Group
Implementation Examples
Uses real-time data in a financial data warehouse for intra-day analytics
Improves supply chain management by creating forecasts from POS data.
Basel II initiative will release about 40% of its minimum capital requirements
Replaced 4,000 hand-coded interfaces to create single view of ticket data
Manages 3 terabytes of store sales data for customer and product analysis
Deutsche Bahn Group
8
IBM Software Group
WebSphere DataStage
Design integration projects within a graphic, codeless environment
Integrate data from the widest range of enterprise and external data sources
Produce re-useable components
Deploy jobs in real-time, batch mode, or as services
Leverage the most scalable and adaptable parallel processing engine
9
DATASTAGE QUALITYSTAGE CLIENT
Sources Targets
PARALLEL PROCESSING
COMMON CONNECTIVITY
METADATA
COMMON SERVICES
IBM Information Server
IBM Software Group
Graphical Design Metaphor
10
IBM Software Group
Pre-Built Transformations for Productivity
11
IBM Software Group
12
Context-sensitive menu:Easy access to transforms
Extensive list of availabletransformation functionsto select from:
Graphical Design Metaphor
12
IBM Software Group
13
Error notification
Immediate notification whenthere’s a problem!
13
IBM Software Group
Extensive Re-use
Shared ContainersGraphical unit of re-useShare one developer’s (subject matter expert)
Meta data research Business rule definitions Transformation logic Special techniques
RoutinesRe-usable functions
Web ServicesDeploy jobs as web services. Invoke from other jobs or
applicationsUse Web Services
14
IBM Software Group
Enterprise ApplicationsJD Edwards
Oracle Applications
PeopleSoft
SAP BW (BAPI, IDOC)
SAP R/3 (ABAP, BAPI, IDOC)
Siebel
RDBMSIBM DB2
IBM IMS
VSAM
Oracle
Informix
RedBrick
SQL Server
Sybase
Teradata
U2 (Universe, UniData)
Tandem NON-STOP SQL
SAS
Business Exchange FormatsXMLSEXMLEDIFIXSWIFTHIPAA
Real-Time WebSphere MQ
SeeBeyond
Java Messaging Services
Java (Client & Transformer)
XML (Read / Write)
XSL-T XSL-T Transformer
Web Services (SOAP)
Enterprise Java Beans
Flat File and General Access
VSAM
VSAM CICS
IDMS
C-ISAM
Sequential File
Complex Flat File
File Set
Data Set
Named Pipe
FTP (standard, secure)
Compressed / Encoded Data
External Command Call
Parallel Wrap 3rd party applications
…And many more!
Connectivity Ensures Data Access
15
IBM Software Group
Benefits of Scalability
Number of CPUs
Pro
cess
ing
Tim
e (h
ours
)
Process the same data volume in less time
Number of CPUs
Pro
cess
ing
Vol
ume
(gig
abyt
es)
Process more data in the same amount of time
- or -
16
20
15
10
5
1 t
750
500
250
2 4 8 12 16 24 32 - - - 2 4 8 12 16 24 32 - - -
IBM Software Group
Uniprocessor SMP SystemMPP, GRID, and
Clustered Systems
Parallel Execution Enables Timely Integration
17
IBM Software Group
18
…DataStage creates “n” processes at runtimefor each Stage, where “n” is the number of logical nodes defined in a configuration file
Given a Job Design:
Enabling Parallelism
18
IBM Software Group
Metadata Driven Integration
Shared metadata across product modulesBetter and faster communication between
team members Immediate access to definitions and notes on
all objectsGreater understanding, better data
Powerful Metadata driven design toolsQuick Find and Advanced Find Impact AnalysisData Lineage reportsGreater productivity, easier maintenance,
reuse
Impact Analysis
Find Capability19
IBM Software Group
DataStage Strength Summary
Graphical, top-down design metaphor
Extensible, component based architecture
Strong Re-use capabilities
Shared Containers, Routines & Web Services
Graphical sequencing (“job flow”)
Application Deployment
Parameterization
Changed Data Capture
Ubiquitous Connectivity
Unlimited Scalability
Design serially, deploy in parallel
20
®
IBM Software Group
©IBM Corporation
Thank You