7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 1/45
<Insert Picture Here>
Oracle Data Integrator
Technical Session 3-4:
Defining The Logical Architecture. Setting up a new project
Designing Interfaces and Packages
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 2/45
<Insert Picture Here>
Agenda
• ODI Repositories
• ODI Data environments
• Reverse and enhance metadata
• Design your transformations
• Create packages workflow
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 3/45
<Insert Picture Here>
ODI Repositories:
scalable metadata
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 4/45
Example of a Repository Set-Up
Models
Projects
Execution
Work Repository(Development)
Models
Projects
Execution
Work Repository
(Test & QA)
Execution
Execution Repository
(Production)
Security
Topology
Versioning
Master
Repository
Create and archiveversions of models,projects andscenarios Import released
versions of models,projects andscenarios for testing
Import released andtested versions ofscenarios forproduction
Development – Test – Production Cycle
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 5/45
<Insert Picture Here>
ODI Data environments:
scale from development to production
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 6/45
Logical vs. Physical Architecture
• Physical Architecture
• Defines physical resources• Describes the real locations of servers and schemas
• How to access databases (Pack. Appl, etc.)
• Which data area to use, also as physical staging area
• Logical Architecture• Abstract view of the resources
• The accounting application schema is defined only once
here. It is one Logical Schema implemented in different
Contexts.
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 7/45
Mapping Logical and Physical
Resources
MS SQL Server
db_dwh
Windows
Development site: New York
Windows
MS SQL Server
db_dwh
db_purchase
Production site: Boston
Windows
MS SQL Server A
dwh
Production site: Tokyo
Datawarehouse(Logical Schema)
D e v
e l o p m e n
t
B o s t o
n
T o k
y o
Logical Architecture
Contexts
Physical Architecture
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 8/45
Our Example Infrastructure With Agents
Oracle
ACCOUNTING
SALES
MS SQL Server
db_dwh
db_purchase
Windows Windows
MS SQL Server B
purchase
Windows Linux
Oracle
ACCT
SAL
Linux
Oracle 9
ACCOUNTING
Oracle 10
SALES
Windows
MS SQL Server A
dwh
MS SQL Server
db_dwh
db_purchase
Production site: Tokyo
Production site: BostonDevelopment site:
New York
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 9/45
<Insert Picture Here>
ODI still on step 1) going to production:
Reverse and enhance metadata
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 10/45
MS SQL Server
ODI Repository
Standard vs. Customized Reverse-
Engineering
Data Model
System tables
Model (Metadata)
Oracle Data Integrator
CustomizedReverse-engineering
JDBC Driver
StandardReverse-engineering
Delimited format
Fixed formatCOBOL copybooks
File-specificreverse-engineering
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 11/45
MDM Enabled: Canonical Format Design
• Use in conjunction with
packaged MDM solution
• Design and Populate CanonicalFormat
1. Use existing metadata artifacts
to design MDM application
(entities, fields, relationships)
2. Generate and maintain MasterData structure
3. Generate and deploy
transformations using metadata
artifacts
CRM
Enterprise Service Bus
SCM Legacy ERP
Master Data
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 12/45
1. Select the Constraints nodeunder the datastore.
2. Right-click, select Insert Key .
3. Fill in the Name.
4. Select the Key or Index Type
5. Go to the Columns tab6. Add/remove columns from the
key.
Adding primary Keys
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 13/45
1. Select the Constraints node underthe datastore
2. Right-click, select Insert Reference
3. Fill in the Name
4. Select the reference type• User Reference
• Complex Reference5. Select a Parent Model and Table
• Set the model and table to<undefined> to manually enter thecatalog, schema and table name.
Creating a Reference
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 14/45
Creating Simple or Complex Reference
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 15/45
1. Right-click Constraints node,selectInsert Condition
2. Fill in the Name.
3. Select ODI Condition type.
4. Edit the condition clause• Use the Expression Editor
5. Type in the error message for thecondition.
Creating a Condition
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 16/45
<Insert Picture Here>
ODI still on step 2) going to production:
Design your transformations
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 17/45
What Is an Interface?
An Interface is an ODI object that defines the
rules to load one target datastore with datafrom one or more source datastores
A declarative design is used to implement thetransformations as mappings.
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 18/45
Using the Expression Editor
1. Click the expression
editor button ( ) in the
mapping window
2. Build your SQL
expressions from the
SQL help at the bottom,and from the Columns at
the left
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 19/45
The Scenario
Source Sybase
ORDERS
LINES
CORRECTIONS
File
Target Oracle
SALES
Mapping
- SALES = SUM(LINES.AMOUNT)+ CORRECTION.VALUE.- SALES_REP =ORDERS.SALES_REP_ID…
Filter
- ORDERS.STATUS=‘CLOSED’…
Join
- ORDERS.ORDER_ID =LINES.ORDER_ID…
About the interface
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 20/45
What Defines the Flow?
Three factors:
• Where the staging area is located• On the target, on a source or on a third server
• How mappings, filters and joins are set up• Execution location: Source, target or staging area
• Whether transformations are “active”• Choice of Knowledge Modules
• LKM: Loading Knowledge Module
• IKM: Integration Knowledge Module
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 21/45
The Basic Process
Source: Sybase
ORDERS
LINES
CORRECTIONSFile
Target: Oracle
SALES
Sequence of operations
C$_01
1
Extract/Join/Transform
C$_12
2
Extract/Transform
I$_SALES
33Join/Transform
55
Transform & Integrate
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 22/45
What Is the Staging Area?
Staging Area – A separate, dedicated area inan RDBMS where Sunopsis creates itstemporary objects and executes some of your
transformation rules.You usually put the staging area on the targetdata server.
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 23/45
Target (Oracle)
Staging Area
Case Study: Flow #1
Source (Sybase)
ORDERS
LINES
CORRECTIONSFile
SALES
The Staging Area on the target server
C$_0
C$_1
I$_SALES
11
22
33
55
Extract/Join/Transform
Extract/Transform
Join/Transform
Transform & Integrate
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 24/45
Case Study: Flow #1 in Sunopsis
Staging Area in the Target
Source Sets
Staging Area + Target
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 25/45
DB2 UDB, Sunopsis Engine, etc.
Staging Area
Case Study: Flow #2
Source (Sybase)
ORDERS
LINES
CORRECTIONSFile
Target (Oracle)
SALES
The Staging Area on a third server
C$_0
C$_1
I$_SALES
11
22
33
55
Extract/Join/Transform
Extract/Transform
Join/Transform
Transform & Integrate
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 26/45
Case Study: Flow #2 in Sunopsis
Staging Area is the Sunopsis Memory Engine
Source Sets
Target
Staging Area
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 27/45
Source (Sybase)
Staging Area
Case Study: Flow #3
ORDERS
LINES
CORRECTIONSFile
Target (Oracle)
SALES
The Staging Area on the Sybase source server
C$_0
C$_1
I$_SALES
11
22
33
55
Extract/Join/Transform
Extract/Transform
Join/Transform
Transform & Integrate
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 28/45
Case Study: Flow #3 in Sunopsis
Staging Area in the Source
Source Sets
Target
Staging Area
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 29/45
KM Description
• A Knowledge Module is
made of steps.
• Each step has a nameand a template for the
code to be generated.
• These steps are listed in
the Details tab.• The code that will be
generated by ODI will list
the same step names
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 30/45
Details of the Steps
• Details of the steps are
generic: the source and target
tables are not known, only the
technologies are known
• Substitution Methods are used
as placeholders for the tablenames and column names
• Parameters of the substitution
methods let you select which
tables or columns are used in
the KM
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 31/45
Staging areaSource TargetIntegration phaseLoading phase
Which KMs for What Flow?
• Four possible arrangements:
Multi-tech LKM Multi-tech IKM
Staging area on targetMulti-tech LKM
Single-tech IKM
Staging area on sourceMulti-tech IKM
(No LKM needed)
Source, staging area and target in same location
Single-tech IKM(No LKM needed)
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 32/45
Target (Oracle)
Staging Area
Flow #1
Source (Sybase)
ORDERS
LINES
CORRECTIONSFile
SALES
Using the Target as the Staging Area
C$_0
C$_1
I$_SALES
LKM_1LKM_1
LKM SQL to Oracle
LKM_2LKM_2
LKM File to Oracle (SQLLDR)
IKM_1IKM_1
IKM Oracle Incremental Update
IKM_1IKM_1
IKM Oracle Incremental Update
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 33/45
Sunopsis Memory Engine
Staging Area
Flow #2
Source (Sybase)
ORDERS
LINES
CORRECTIONSFile
Target (Oracle)
SALES
Using a third server as the Staging Area
C$_0
C$_1
I$_SALES
LKM_1LKM_1
LKM SQL to SQL
LKM_2LKM_2
LKM File to SQL
IKM_1IKM_1
IKM SQL to SQL Append
IKM_1IKM_1
IKM SQL to SQL Append
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 34/45
Source (Sybase)
Staging Area
Flow #3
ORDERS
LINES
CORRECTIONSFile
Target (Oracle)
SALES
Using the Source as the Staging Area
C$_0
C$_1
I$_SALES
IKM_1IKM_1
LKM_1LKM_1
IKM_1IKM_1
IKM_1IKM_1
LKM File to SQL
IKM SQL to SQL Append
IKM SQL to SQL Append
IKM SQL to SQL Append
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 35/45
Real-time enabled: Message Oriented
Integration
• Connect to Publish andSubscribe JMS MessageProviders
• Ensure messages delivery withtransaction integrity
• High-volume bulktransformations1. Design complex bulk
transformations mixing Queues,Databases and Applications
2. Use JMS Queues and topics assources or targets
JMS Provider (MOM, ESB)
CDC
Subscribe
Publish
ODI and XML files
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 36/45
ODI and XML files
• ODI XML driver can handle also very huge XML files, using Oracle and non Oracle RDBMS asworkspace for the schema.
• ODI and OWB can use also the native XML management functions in Oracle RDBMS (xml_load_f)
Source
Or Target
XMLAgent
Workspace for XML
(OME or RDBMS)
Parser JDBC JDBCJDBC
Loading
Driver
Synchronization
or Connection
Integration
XML driver management XML driver transforms the fileobject in a relational model
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 37/45
<Insert Picture Here>
ODI still on step 3) going to production:
Creating packages workflow
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 38/45
The Package Diagram
Interface step(selected)
ODI tool step
Diagram
Toolbox for ODI
tools
Toolbar
Properties ofselected step
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 39/45
How to Create an ODI Tool Step
1. In the Toolbox , expand the
group containing the tool you
want to add.
2. Click the tool.3. Click the diagram.
A step named after
the tool appears.
4. Change the Step Name
in the Properties panel.
5. Set the tool’s Properties.
6. Click Apply to save.
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 40/45
Defining a Variable and using it in a package
• Declare Variable step type:• Set Variable step type:
• Refresh Variable step type:• Evaluate Variable step type:
C t N P d
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 41/45
Create a New Procedure
1. Right-click the Procedures nodeunder a project.
2. Select Insert Procedure
3. Fill in the• Name
• Description
4. Optionally, define the default:• Source Technology
• Target Technology
5. Click Apply to save.
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 42/45
How to Create Model, Sub-Model and Datastore Steps
1. Select the model, sub-model or datastorewhich you want to work
with in the Models view.2. Drag and drop it into the package.
3. Select the type ofoperation to perform.
4. Set the Optionsfor the chosenoperation
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 43/45
A Simple Package
Step on failure
Step onsuccess
First step
• This package executes two interfaces then archives some files.
• If one of the three steps fails, an email is sent to the administrator.
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 44/45
How to Create a Loop
• Loops need a counter variable
1. Set the counter to an initial value
2. Execute the step or steps to be repeated
3. Increment the counter
4. Evaluate counter value and loop to step 2 if the goal has beenreached
Package Loop
1. 2.
3.
4.
7/23/2019 Setting Up a New Project
http://slidepdf.com/reader/full/setting-up-a-new-project 45/45
Executing a Package
1. Click the Execute buttonin the package window
2. Open Operator
• The package isexecuted as a session
• Each package step is a step
• Tool steps appearwith a single task
• Interface steps show eachcommand as a separate task