Data Stage PPT Materials

8/9/2019 Data Stage PPT Materials

1/216

Copyright Sennovate 2010. All rights

1-1Ver.1.0

IBM Information Server

InfoSphere Information Server provides a

single unified platform that enables companies

to understand, cleanse, transform, and delivertrustworthy and context-rich information.


2/216


1-2Ver.1.0

Products in IBM Inor!ation Server

IBM InfoSphere DataStage IBM InfoSphere QualityStage

IBM InfoSphere Information Services Director

IBM InfoSphere InformationAnalyzer

IBM Information Server ast!rac" IBM InfoSphere Business #lossary

$ther %ompanion &ro'ucts

IBM InfoSphere e'eration Server

(ational Data Architect InfoSphere (eplicationServer

)vent &u*lisher


3/216


1-"Ver.1.0

IBM Inor!ation Server architecture

IBM Information Server architecture is a

client+server architecture ma'e up of client+*ase''esign, a'ministration an' operation tools thataccess a set of server+*ase' 'ata integration

capa*ilities through a common services layer.


4/216


1-#Ver.1.0

IBM Inor!ation Server architecture


5/216


1-$Ver.1.0

Client tier

!he %lient tier *asically inclu'es the follo-ing

IBM InfoSphere DataStage an' QualityStage clients

A'ministrator

Director

Designer


6/216


1-%Ver.1.0

Server tier

!he Server tier inclu'es

Services

)ngine

(epository /or"ing area

/or"ing areas

Information Services Director resource provi'ers


7/216 Copyright Sennovate 2010. All rights

1-&Ver.1.0

Services tier

!hree general categories of Services

Design

)ecution

Meta'ata



1-'Ver.1.0

(epository tier

!he Share' (epository is use' to share all the IBMInformation Server pro'uct mo'ule o*ects.

!he common repository contains the follo-ing types of

meta'ata that are re2uire' to support InfoSphereDataStage

Pro)ect !etadata

*perational !etadata

+esign !etadata



1-,Ver.1.0

ngine tier

!his is a parallel engine that eecutes IBM InformationServer tas"s.



1-10Ver.1.0

or/ing areas

!hese are the temporary storage areas use' *y thecomponents.



1-11Ver.1.0

Inor!ation Service providers

Information Service provi'ers are sources ofoperations for the services li"eDataStage,QualityStage.



1-12Ver.1.0

opologies

IBM InfoSphere Information Server multiple topologiesto support variety of 'ata integration , har'-are an'*usiness re2uirements.

%onsi'er the performance nee's to select the

topology !opologies supporte' are as follo-s

!-o+tier

!hree+tier

%luster#ri'



1-1"Ver.1.0

opologies

o-tier

!he engine, application server an' the meta'atarepository are all on the same computersystems -hile client are in 'i3erent machines.

hree-tier

!he engine is on one machine , the applicationserver an' meta'ata repository is co+locate' on

other machine. %lients are in the thir' machine.



1-1#Ver.1.0

opologies

Cluster!his is a slight variation of a three tier topology.

!he engine is 'uplicate' over multiple computers.

Ina cluster environment, a single parallel o* eecution

can span multiple computer each -ith its o-n engine.!he processing of a o* on multiple machines is 'riven

*y a con4guration 4le associate' -ith the o*.


15/216



1-1%Ver.1.0

o-tier



1-1&Ver.1.0

hree tier



1-1'Ver.1.0

Clusters and rids



1-1,Ver.1.0

IBM Inosphere +ataStage

&opular )!5 tool

)na*les $rganization to 'esign 'ata 6o-s that etractinformation from multiple source systems, transform itto ma"e it more valua*le, an' then 'eliver it to one or

more target 'ata*ases. IBM InfoSphere DataStage is a part of IBM Information

Server architecture

It has

Share' components (untime Architecture


20/216

Copyright Sennovate 2010. All rights1-20

Ver.1.0

+ataStage architecture


21/216


Ver.1.0

Shared Co!ponents o +ataStage

Common User Interface

Designer

Director

dministrator

Common services

Common repository

Common parallel processing engine

Common connectors


22/216


Ver.1.0

(unti!e architecture

*S3 Script

7sing the 'esigner, o*s are create'.

!he o*s are compile' into parallel o* 6o-s an'reusa*le components that eecute on the parallel

information server engine.Designer generates the $S89$rchestrate Shell

script:.

$S8 script

7ses the familiar script of 7ni shell.


23/216

Copyright Sennovate 2010. All rights1-2"

Ver.1.0

4our core capa5ilities o +ataStage

%onnectivity to -i'e range of mainframe, legacy an'enterprise application 'ata*ases, 4le formats an'eternal information sources.

&re*uilt li*rary of more than ;00 functions inclu'ing

'ata vali'ation rules an' comple transformations. Maimum throughput using parallel, high+

performance processing architecture.

&rovi'es 'evelopment, 'eployment an' maintenancefeatures. It leverages meta'ata for analysis an'maintenance.


24/216

Copyright Sennovate 2010. All rights1-2#

Ver.1.0

InoSphere +ataStage ele!ents

he Central +ataStage ele!ents are

&roects

%reate' -ith A'ministrator.

)ach proect contains other components such as o*s,stages, lin"s, containers an' ta*le 'e4nitions.


25/216

Copyright Sennovate 2010. All rights1-2$

Ver.1.0

A 6o5


26/216

Copyright Sennovate 2010. All rights1-2%

Ver.1.0

7a!ple o a )o5


27/216


28/216

Copyright Sennovate 2010. All rights1-2'

Ver.1.0

Parallel 6o5

)ecute' *y the DataStage parallel engine.

Built+in functionality for pipeline an' partitionparallelism .

%ompile' into $S8 9$rchestrate Scripting 5anguage:.

$S8 eecutes $perators )ecuta*le %== classinstances.

(untime monitoring in DataStage Director


29/216

Copyright Sennovate 2010. All rights1-2,

Ver.1.0

Server 6o5s

)ecute' *y the DataStage server engine

%ompile' into Basic



30/216

Copyright Sennovate 2010. All rights1-"0

Ver.1.0

6o5 Se8uences

Master Server o*s that "ic"+o3 server or parallel o*san' other activities.


)ecute' *y the Server engine


31/216


Ver.1.0

Stages

Active stage

Active stages mo'el the 6o- of 'ata an' provi'emechanisms for com*ining 'ata streams,aggregating 'ata, an' converting 'ata from one

'ata type to another Alters the num*er of ro-s from source to target.

&assive Stage

A passive stage han'les access to 'ata*ases for

the etraction or -riting of 'ata. Does not alter the num*er of ro-s from sourceto target.


32/216


Ver.1.0

Parallel processing

&arallel processing is the use of multiple processors toeecute the 'i3erent parts of the same programsimultaneously.


33/216

Copyright Sennovate 2010. All rights1-""

Ver.1.0

(epresentation o )o5 ithoutparallelis!


34/216

Copyright Sennovate 2010. All rights1-"#

Ver.1.0

o type o parallel processing

&ipeline

&artitioning

%om*ining an' &artitioning


35/216

Copyright Sennovate 2010. All rights1-"$

Ver.1.0

Pipeline Parallelis!

!ransform, clean, loa' processes eecutesimultaneously

5i"e a conveyor *elt moving ro-s from process toprocess

Start 'o-nstream process -hile upstream processis running

A'vantages

(e'uces 'is" usage for staging areas

>eeps processors *usy

Still has limits on scala*ility


36/216

Copyright Sennovate 2010. All rights1-"%

Ver.1.0

Pipeline Parallelis!


37/216

Copyright Sennovate 2010. All rights1-"&

Ver.1.0

Partition Parallelis!

Divi'e the incoming stream of 'ata into su*sets to *eseparately processe' *y an operation .

Su*sets are calle' partitions 9no'es:

!his is "ey to Scala*ility

)ach partition of 'ata is processe' *y the sameoperation

).g., if operation is ilter, each partition -ill *e4ltere' in eactly the same -ay

acilitates near+linear scala*ility ? times faster on ? processors

@ times faster on @ processors

!his assumes the 'ata is evenly 'istri*ute'


38/216

Copyright Sennovate 2010. All rights1-"'

Ver.1.0

Partitioned Parallelis!


39/216

Copyright Sennovate 2010. All rights1-",

Ver.1.0

hree-9ode Partitioning


40/216

Copyright Sennovate 2010. All rights1-#0

Ver.1.0

Parallel 6o5s Co!5ine Partitioningand Pipelining


41/216


42/216


43/216

Copyright Sennovate 2010. All rights1-#"

Ver.1.0

Inor!ation in con:guration :le

9odes

It i'enti4es the num*er of no'es in the parallelprocessing.

(esource dis/

Data 4les are store' here (esource scratch dis/

8ere path is speci4e'. !he path is use' *yparallel o*s for *u3ering


44/216

Copyright Sennovate 2010. All rights1-##

Ver.1.0

Sa!ple Con:guration :le

{node "dev1"

{

fast name "etltools-dev"

pool ""

resource disk "/data/etltools-tutorial/d1" { }

resource disk "/data/etltools-tutorial/d2" { }

resource scratchdisk "/data/etltools-tutorial/temp" { }

}

node "dev2"

{

fastname "etltools-dev"

pool ""resource disk "/data/etltools-tutorial/d1" { }

resource scratchdisk "/data/etltools-tutorial/temp" { }

}

}


45/216

Copyright Sennovate 2010. All rights1-#$

Ver.1.0

Partitioning and Collecting

&artitioning *rea"s incoming ro-s into multiple streams ofro-s 9one for each no'e:

)ach partition of ro-s is processe' separately *y thestageoperator

%ollecting returns partitione' 'ata *ac" to a single stream

&artitioning %ollecting is speci4e' on stage input lin"s


46/216

Copyright Sennovate 2010. All rights1-#%

Ver.1.0

Partitioning !ethods

(oun' (o*in &artitioner

(an'om &artitioner

Same partitioner

)ntire &artitioning

8ash partitioner

Mo'ulus partitioner

(ange partitioner

DB@ &artitioner

Auto &artitioner


47/216

Copyright Sennovate 2010. All rights1-#&

Ver.1.0

(ound (o5in Partitioner

!he 4rst recor' goes to the 4rst partitioning no'e,secon' to the secon' an' so on.

/hen the 'ataStage reaches the last it starts overagain.


48/216

Copyright Sennovate 2010. All rights1-#'

Ver.1.0

(ando! Partitioner

(ecor's are ran'omly 'istri*ute' over all partitioningno'es.

5i"e roun' ro*in, ran'om partitioning can re*alancethe partitions of an input 'ata set to guarantee that

each processing no'e receives an approimatelye2ual+size' partition.

!he ran'om partitioning has a slightly higheroverhea' than roun' ro*in *ecause of the etraprocessing re2uire' to calculate a ran'om value for

each recor'.


49/216

Copyright Sennovate 2010. All rights1-#,

Ver.1.0

Sa!e partitioner

!he stage using the 'ata set as input performs norepartitioning an' ta"es as input the partitions output*y the prece'ing stage.

/ith this partitioning metho', recor's stay on the

same processing no'eE that is, they are notre'istri*ute'.

Same is the fastest partitioning metho'.

!his is normally the metho' DataStage uses -henpassing 'ata *et-een stages in your o*.


50/216

Copyright Sennovate 2010. All rights1-$0

Ver.1.0

ntire Partitioning

)very instance of a stage on every processing no'ereceives the complete 'ata set as input.

It is useful -hen you -ant the *ene4ts of paralleleecution, *ut every instance of the operator nee's

access to the entire input 'ata set.


51/216


Ver.1.0

3ash partitioner

Set *ase' on a zip co'e 4el', -here a large percentage of yourrecor's &artitioning is *ase' on a function of one or more columns9the hash partitioning "eys: in each recor'. !he hash partitionereamines one or more 4el's of each input recor' 9the hash "ey4el's:.

(ecor's -ith the same values for all hash "ey 4el's are assigne' tothe same processing no'e.

!his metho' is useful for ensuring that relate' recor's are in thesame partition, -hich might *e a prere2uisite for a processingoperation.

8ash partitioning 'oes not necessarily result in an even 'istri*ution

of 'ata *et-een partitions. or eample, if you hash partition a 'ata are from one or t-o zip

co'es, you can en' up -ith a fe- partitions containing most of yourrecor's. !his *ehavior can lea' to *ottlenec"s *ecause some no'esare re2uire' to process more recor's than other no'es.


52/216


Ver.1.0

Modulus partitioner

&artitioning is *ase' on a "ey column mo'ulo thenum*er of partitions. !his metho' is similar to hash *y4el', *ut involves simpler computation.


53/216

Copyright Sennovate 2010. All rights1-$"

Ver.1.0

(ange partitioner

Divi'es a 'ata set into approimately e2ual+size'partitions, each of -hich contains recor's -ith "eycolumns -ithin a speci4e' range. !his metho' is alsouseful for ensuring that relate' recor's are in the

same partition. A range partitioner 'ivi'es a 'ata set intoapproimately e2ual size partitions *ase' on one ormore partitioning "eys. (ange partitioning is often apreprocessing step to performing a total sort on a 'ata

set. In or'er to use a range partitioner, you have to ma"e a

range map. Fou can 'o this using the /rite (ange Mapstage.


54/216

Copyright Sennovate 2010. All rights1-$#

Ver.1.0

+B2 Partitioner

&artitions an input 'ata set in the same -ay that DB@G-oul' partition it.

or eample, if you use this metho' to partition aninput 'ata set containing up'ate information for an

eisting DB@ ta*le, recor's are assigne' to theprocessing no'e containing the correspon'ing DB@recor'. !hen, 'uring the eecution of the paralleloperator, *oth the input recor' an' the DB@ ta*lerecor' are local to the processing no'e. Any rea's an'

-rites of the DB@ ta*le -oul' entail no net-or"activity.


55/216

Copyright Sennovate 2010. All rights1-$$

Ver.1.0

Auto Partitioner

5eaving it to DataStage to 'etermine the *estpartitioning metho' to use 'epen'ing on the type ofstage, an' -hat the previous stage in the o* has'one.

!ypically DataStage -oul' use roun' ro*in -heninitially partitioning 'ata, an' same for theinterme'iate stages of a o*.


56/216

Copyright Sennovate 2010. All rights1-$%

Ver.1.0

Collecting

%ollecting the process of oining the multiple partitionsin to single 'ataset.

%ollecting metho's

(oun' ro*in

$r'ere' collector Sort merge collector

Auto collector


57/216

Copyright Sennovate 2010. All rights1-$&

Ver.1.0

(ound ro5in

(ea's a recor' from the 4rst input partition, then fromthe secon' partition, an' so on. After reaching the lastpartition, starts over.

After reaching the 4nal recor' in any partition, s"ips

that partition in the remaining roun's


58/216

Copyright Sennovate 2010. All rights1-$'

Ver.1.0

*rdered collector

(ea's all recor's from the 4rst partition, then allrecor's from the secon' partition, an' so on.

!his collection metho' preserves the or'er of totallysorte' input 'ata sets. In a totally sorte' 'ata set,

*oth the recor's in each partition an' the partitionsthemselves are or'ere'.

!his might *e useful as a preprocessing action *eforeeporting a sorte' 'ata set to a single 'ata 4le.


59/216

Copyright Sennovate 2010. All rights1-$,

Ver.1.0

Sort !erge collector

pro'uces a glo*ally sorte' se2uential stream from-ithin partition sorte' ro-s.

Sort Merge pro'uces a non+'eterministic on un+"eye'columns sorte' se2uential stream using the follo-ing

algorithmal-ays pic" the partition that pro'uces the ro- -ith

the smallest "ey value.


60/216

Copyright Sennovate 2010. All rights1-%0

Ver.1.0

Auto collector

!he 'efault algorithm rea's ro-s from a partition assoon as they are rea'y.

!his may lea' to pro'ucing 'i3erent ro- or'ers in'i3erent runs -ith i'entical 'ata. !he eecution is

non+'eterministic.


61/216


Ver.1.0

Ad!inistrator

A'ministrator is a client program use' to carry outcon4guration tas"s in DataStage.

It has ; pages

#eneral

!he general page is use' to set server+-i'eproperties.

&roect

!his lists the proects availa*le an' options toa'', e'it an' 'elete proects.

C5S

Cational 5anguage support features.


62/216


Ver.1.0

Attaching to +ataStage


63/216

Copyright Sennovate 2010. All rights1-%"

Ver.1.0

Ad!inistrator


64/216

Copyright Sennovate 2010. All rights1-%#

Ver.1.0

Pro)ect Page

A'' !o a'' Ce- DataStage &roect

Delete !o 'elete a proect. !his *utton is ena*le onlyif you have a'ministrator status.

&roperties !o set the properties of the selecte'

proect. %leanup %leans up 4les in selecte' proect

C5S !o change proect map an' locales.

%omman' !o eecute DataStage )ngine comman's

'irectly from the selecte' proect


65/216

Copyright Sennovate 2010. All rights1-%$

Ver.1.0

Pro)ect page


66/216

Copyright Sennovate 2010. All rights1-%%Ver.1.0

Add Pro)ect


67/216

Copyright Sennovate 2010. All rights1-%&Ver.1.0

Creating a pro)ect


68/216


69/216

Copyright Sennovate 2010. All rights1-%,Ver.1.0

Pro)ect Properties ; eneral ta5


70/216

Copyright Sennovate 2010. All rights1-&0Ver.1.0

eneral ta5 in Pro)ect properties

)na*le o* a'ministration from the DataStage Director 5ets DataStage operators release the resources of a o* that

has a*orte' or hung, an' so return the o* to a state in -hichit can *e rerun -hen the cause of the pro*lem has *een 4e'.

)na*le runtime column propagation for parallel o*s.

Stages in parallel o* can han'le un'e4ne' columns.

De4ne a proect+-i'e setting for auto+purge of the o* log.!o prevent o* log 4le to *ecome too large this feature is

use'.

Set up environment varia*les.%an set values to )nvironment varia*les

%reate ne- )nvironment Varia*les -hich can *e use' li"eparameters.


71/216

Copyright Sennovate 2010. All rights 1-&1Ver.1.0

Per!issions ta5


72/216

Copyright Sennovate 2010. All rights 1-&2Ver.1.0

Per!issions ta5

ssign user categories to operating system user groups, or enableoperators to view all the details of an event in a !ob log file.

"he Permissions tab is enabled only if you have logged on to

DataStage using a name that gives you administrator status.


73/216

Copyright Sennovate 2010. All rights 1-&"Ver.1.0

racing ta5


74/216

Copyright Sennovate 2010. All rights 1-Ver.1.0

racing ta5

!his is to ena*le or 'isa*le tracing on the server.


75/216

Copyright Sennovate 2010. All rights 1-&$Ver.1.0

Schedule


76/216

Copyright Sennovate 2010. All rights 1-&%Ver.1.0

Schedule ta5

Set up a user name an' pass-or' to use for runningsche'ule' DataStage o*s.

!he Sche'ule ta* is ena*le' only if you have logge'on to a /in'o-s C! server.


77/216

Copyright Sennovate 2010. All rights 1-&&Ver.1.0

Mainra!e )o5 properties


78/216

Copyright Sennovate 2010. All rights 1-&'Ver.1.0

una5le ; Con:gure cache settings


79/216

Copyright Sennovate 2010. All rights 1-&,Ver.1.0

Parallel ta5


80/216

Copyright Sennovate 2010. All rights 1-'0Ver.1.0

Se8uence ta5


81/216

Copyright Sennovate 2010. All rights 1-'1Ver.1.0

I!porting and 7porting *5)ects

rom Director choose !oolsJJ(un Manger %hoose )port JJ%omponents.


82/216


1-'2Ver.1.0

+esigner

A graphical user interface for creating DataStageapplications "no-n as


83/216


1-'"Ver.1.0

Stages

!he in'ivi'ual steps that ma"e up the o* are calle'stages.

Some of the DataStage &re*uilt stages are sort,merge, oin, 4lter, transform, loo"up an' aggregate.

Stages provi'e the ?0 to K0 percent of the applicationlogic re2uire' for enterprise 'ata integrationapplications.

)ach stage has properties that tell ho- to perform orprocess 'ata.


84/216


1-'#Ver.1.0

So!e o the Stages in +ataStage

i i


85/216


1-'$Ver.1.0

Co!!on


86/216


1-'%Ver.1.0

Stages list contd=

Sort Stage Surrogate >ey generator

!ransformer

(emove Duplicate stage

St i ti ) 5


87/216


1-'&Ver.1.0

Steps in creating a )o5

$pen 'esigner an' connect to the proect %hoose the type of o* to *e create'.

Import ta*le 'e4nition

Drag an' 'rop the stages

5in" the stages Set the properties of the stage

Save an' compile the o*

)ecute the o* *y choosing !oolsJJ(un Director

7a!pleSimple o* to group 'epartment+-ise an' sum

salary from 6at 4le

C t t th P ) t


88/216


1-''Ver.1.0

Connect to the Pro)ect

Ch th t 6 5


89/216


1-',Ver.1.0

Choose the type o 6o5

P ll l 6 5 Citory


90/216


1-,0Ver.1.0

Parallel 6o5 Canvas

Stage

s

&alat

te

(epo

sito

o*1ec

ts

%ACVAS

I t S ti l :l d : iti


91/216


1-,1Ver.1.0

I!port Se8uential :le de:nition

Se2uential 4le'e4nition

Ch +i t d :l t i t


92/216


1-,2Ver.1.0

Choose +irectory and :le to i!portImportoption

+ : C l d t


93/216


1-,"Ver.1.0

+e:ne Colu!ns and or!at

Stages and


94/216


1-,#Ver.1.0

Stages and


95/216


1-,$Ver.1.0

Se8uential :le - Source

4or!at or Se8uential :le


96/216


1-,%Ver.1.0

4or!at or Se8uential :le

$ther&roperti

es

Colu!ns ta5


97/216


1-,&Ver.1.0

Colu!ns ta5 -


98/216


1-,'Ver.1.0

Select the colu!ns needed

Colu!ns loaded


99/216


1-,,Ver.1.0

Colu!ns loaded

Se8uential :le arget Properties


100/216


1-100Ver.1.0

Se8uential :le ; arget Properties

Aggregate Stage Properties


101/216


1-

101Ver.1.0

Aggregate Stage Properties

Select#roup *ycolumn

5ist ofcolumn

s

Choose *utput colu!ns


102/216


1-

102Ver.1.0

Choose *utput colu!ns

Input page Aggregator Stage


103/216


1-

10"Ver.1.0

Input page ; Aggregator Stage

*utput page


104/216


1-

10#Ver.1.0

*utput page

6o5


105/216


1-

10$Ver.1.0

6o5

Save )o5


106/216


1-

10%Ver.1.0

Save )o5


107/216

(un +irector


108/216


1-

10'Ver.1.0

(un +irector

Status >ie


109/216


1-

10,Ver.1.0

Status >ie

(un the )o5


110/216


1-

110Ver.1.0

(un the )o5

Annotation Stage


111/216


1-

111Ver.1.0

Annotation Stage

!his stage is use' to insert notes to the 'iagram-in'o-.

!-o types of Annotation

Annotation

Description Annotation


112/216


1-

112Ver.1.0

Stages in co!5ining data

Co!5ining +ata 5ased on ?eyl


113/216


1-

11"Ver.1.0

colu!n

5oo"up Stage Merge Stage


114/216


1-

11#Ver.1.0


115/216


1-

11$Ver.1.0


116/216


1-

11%Ver.1.0


117/216


1-

11&Ver.1.0


118/216


1-

11'Ver.1.0

oo up S age

6oin Stage


119/216


1-

11,Ver.1.0

6 g

I t performs oin operation on t-o or more inputs tothe stage

!his is similar to s2l oin.

It provi'es

Inner

ull $uter

5eft $uter

(igh $uter

6oin Stage


120/216


1-

120Ver.1.0

6 g

6oin Stage%hoosethe "eyfor oin


121/216


1-

121Ver.1.0

6 gfor oin

6oin Stage%hoosethe oin

type


122/216


1-

122Ver.1.0

6 gtype

6oin Stage


123/216


1-

12"Ver.1.0

6 g

Merge Stage


124/216


1-

12#Ver.1.0

g g

Merge stage is processing stage It can have

More than one input lin"

Single $utput lin"

Same num*er of reect lin" as that of up'ate lin"s.

Merge Stage


125/216


1-

12$Ver.1.0

g g

Merge Stage

%hoosethe

merge


126/216


1-

12%Ver.1.0

g g"ey

Merge

>eep or'rop


127/216


1-

12&Ver.1.0

Merge

Merge Stage


128/216


1-

12'Ver.1.0

Co!parisonMerge Join Lookup


129/216


1-

12,Ver.1.01

Merge Join Lookup

Stream Input 2 to N 2 To N 1

Reference Input NA NA 1-N

Output

Merged data

Master Update Type SQL-type o!ned data

If no dup"!cates !n t#e "oo$up data e%pected

t#en one for e&ery !nput stream record

'"se If one reference stream pro&!des

"eg!t!mate dup"!cates( t#en mu"t!p"e ro)s for

t#ose records

Sort!ng re*u!rements A"" !nput A"" Input Stream Input On"y

+up"!cates

Not a""o)ed e%cept !n

"ast update "!n$ A""o)ed

A""o)ed !n Stream Input

Upto 1 reference "!n$ can #and"edup"!cates, In ot#ers( s!ng"ef!rst. &a"ue

returned,

/art!t!on Merge 0ey o!n 0ey Usua""y set 'nt!re3 for "oo$up data

Unmatc#ed Ro)s

Master - drop4$eep(

)arn!ng4no )arn!ng

Update - drop4reect

+epends on o!n type

NULL &a"ues on outer

o!n Unmatc#ed stream - reect4$eep

Memory

5ery fe) ro)s !n memory

as data !s sorted 6 no

dup"!cates are e%pected

7e) ro)s as data !s

sorted,8!g#er se*uent!a"(

opt!m!9ed. I4O for #!g#-

speed sort on !nput 6

reference data sets

Loo$up data !n memory- may page for

"arge &o"umes, Not su!ta:"e for "argereference data,

;#en "oo$!ng up aga!nst a data:ase( t#e

+< stage can :e set to pro&!de sparse "oo$-

up support

Use ;#en Larger sorted data Large data Sma"" reference data "oo$-up,

4unnel Stage


130/216


1-

1"0Ver.1.0

It com*ines multiple input to single output !he stage can have any num*er of input lin"s *ut a

single $utput 5in".

!he meta'ata of all the inputs has to *e i'entical

unnel Stage $perates in ; mo'es%ontinuous funnel

Sort funnel

Se2uence funnel

4unnel Stage


131/216


1-

1"1Ver.1.0

4unnel Stage


132/216


1-

1"2Ver.1.0

Choose thefunnel type

4unnel stage


133/216


1-

1""Ver.1.0

ypes o unnel


134/216


1-

1"#Ver.1.0

%ontinuous unnel

%ontinuous funnel com*ines recor's of the input 'ata in nogurantee' or'er

It ta"es one recor' from each input lin" in turn.

If 'ata is not availa*le on an input lin", the stage s"ips tothe net lin" rather than -aiting.

Sort unnelSort unnel com*ines the input recor's in the or'er 'e4ne'

*y the value9s: of one or more "ey columns, an' the or'erof the output recor's is 'etermine' *y these sorting "eys.

Se2uence unnel

Se2uence copies all recor's from the 4rst input 'ata set tothe output 'ata set, then all the recor's from the secon'input 'ata set, an' so on.

+evelop!ent and +e5ug Stages


135/216


1-

1"$Ver.1.0

#ead Stage "ail Stage

$ee% Stage

Column &enerator Stage

'ow &enerator Stage (rite 'ange )ap Stage

3ead Stage


136/216


1-

1"%Ver.1.0

It can have a single input lin" an' a single output lin"

Select 4rst C ro-s from each partition of an input 'ataset an' copies selecte' ro-s to output 'ata set.

!his is use' to De*ug large Data Sets

&roperty settings inclu'es the follo-ing

Cum*er of recor's to copy

&artition from -hich recor's are copie'

5ocation

Stage Page ; 3ead Stage


137/216


1-

1"&Ver.1.0

#eneral #eneral &roperties can *e provi'e' here

&roperties

&roperties -hich inclu'es num*er of ro-s per

partition, all ro-s or s"ip ro-s A'vance'

Di3erent eecution mo'es an' com*ina*ilitymo'e

Input Page ; 3ead Stage


138/216


1-

1"'Ver.1.0

#eneral &artitioning

%olumn

A'vance'

*utput Page ; 3ead Stage


139/216


1-

1",Ver.1.0

#eneral Mapping

%olumn

A'vance'

ail Stage


140/216


1-

1#0Ver.1.0

It can have single input lin" an' single output lin" It selects last C recor's from each partition an' copies

it to output 'ata set

Pee/ Stage


141/216


1-

1#1Ver.1.0

It has single input lin" an' any num*er of output lin"s

It let to print the recor' column values either in the)o5 log or separate output lin" as it copies recor'sfrom input to output.

It is helpful in monitoring the progress of theapplication or 'iagnose the *ug in the application.

Sa!ple Stage


142/216


1-

1#2Ver.1.0

It has single input lin" an' any num*er of output lin"s

Samples an input 'ataset

&ercent mo'e+It etracts ro-s *y selecting them *ymeans of a ran'om num*er generator an' -rites apercentage to output 'ata set.

Colu!n enerator Stage


143/216


1-

1#"Ver.1.0

It can have a single input lin" an' a single output lin".

%olumn #enerator a''s columns to the incoming Dataan' generates moc" 'ata for these columns for eachro- processe'.

!he ne- 'ata set is the output.

(o enerator Stage


144/216


1-

1##Ver.1.0

(o- generator has no input lin" an' a single outputlin"

(o- generator pro'uces moc" 'ata 4tting the givenmeta'ata.

It is use' to test -hen there is no 'ata availa*le

It has Stage page an' $utput &age

rite (ange Map Stage


145/216


1-

1#$Ver.1.0

!he /rite (ange Map stage ta"es an input 'ata setpro'uce' *y sampling an' sorting a 'ata set an'-rites it to a 4le in a form usa*le *y the rangepartitioning metho'.

A typical use for the /rite (ange Map stage -oul' *ein a o* -hich use' the Sample stage to sample a 'ataset, the Sort stage to sort it an' the /rite (ange Mapstage to -rite the resulting 'ata set to a 4le.

*+BC Stages


146/216


1-

1#%Ver.1.0

$DB% stage is use' to etract, -rite or aggregate'ata.

)ach $DB% stage can have any num*er of input lin"sor output lin"s.

Specify the input lin" using the follo-ing metho's

An SQ5 statement

A user 'e4ne' SQ5 2uery

A store' proce'ure

I!port ; *+BC ta5le de:nition


147/216


1-

1#&Ver.1.0

*+BC stage


148/216


1-

1#'Ver.1.0

Choose the ta5les to i!port


149/216


1-

1#,Ver.1.0

*+BC SA


150/216


1-

1$0Ver.1.0

*+BC stage


151/216


1-

1$1Ver.1.0

*utput Mapping


152/216


1-

1$2Ver.1.0

*uput !apping


153/216


1-

1$"Ver.1.0

*CI stage ; I!port Plugin !etadade:nition


154/216


1-

1$#Ver.1.0

*CI Stage


155/216


1-

1$$Ver.1.0

+ata source na!e and user detailsData*as


156/216


1-

1$%Ver.1.0

Data*ase name


157/216

Choose the ta5les or i!porting


158/216


1-

1$'Ver.1.0

*CI Stage and transor!er stage


159/216


1-

1$,Ver.1.0


160/216

Choose the ta5le


161/216


1-

1%1Ver.1.0

Choose the colu!ns


162/216


1-

1%2Ver.1.0

Surrogate ?ey generator stage


163/216


1-

1%"Ver.1.0

Surrogate ?ey enerator


164/216


1-

1%#Ver.1.0

Sort and 4ilter


165/216


1-

1%$Ver.1.0

Sort stage Properties%hoose


166/216


1-

1%%Ver.1.0

the "ey

*utput Mapping


167/216


1-

1%&Ver.1.0

4ilter Stageilter

con'ition


168/216


1-

1%'Ver.1.0

con'ition

ransor!er stage


169/216


1-

1%,Ver.1.0


170/216

ransor!er StageStage

varia*les

%onstraints


171/216


1-

1&1Ver.1.0

ransor!er stage


172/216


1-1&2Ver.1.0

ransor!er


173/216


1-1&"Ver.1.0

ransor!er


174/216


1-1Ver.1.0

ransor!er conditions


175/216


1-1&$Ver.1.0

Scenario&ro'uct 4le has pco'e an' pro'uct colour

&ro'ucts -ith yello- colour are move' to one 4le,*lue are move' to one 4le an' rest are move' toother

!his tas" is 'one using !ransformer stageconstraints

ransor!er stage ith " outputlin/s


176/216


1-1&%Ver.1.0

ransor!er stage constraints


177/216


1-1&&Ver.1.0

Change +ata capture stage


178/216


1-1&'Ver.1.0

%ompares t-o 'ata sets an' recor's the 'i3erences*et-een them.


179/216


180/216

+e:ning Para!eter in 6o5 Propertiesindo


181/216


1-1'1Ver.1.0

&aramete

rs option

@sing 6o5 Para!eters


182/216


1-1'2Ver.1.0

!o 7se the parameters in the o* useLparameternameL


183/216

Containers


184/216


1-1'#Ver.1.0

Container is a group of stages and lin%s. It is used to modulari*e Server !ob designs using Container Stage

DataStage provides + types of containers

ocal Container

Shared Container

ypes o Containers


185/216


1-1'$Ver.1.0

ocal Container"hese are created within a !ob and are accessible only within a

!ob.

Shared Containers

"hese are created and stored separately in repository as !obs.

"here are + types of Shared Containers

Server Shared Containers

Server shared containers can be included in the parallel

!obs

$arallel Shared Containers

Creating


186/216


1-1'%Ver.1.0

If a o* is comple group stages an' lin" through acontainer to save an eisting stages an' lin"s in localcontainer

Select the Stages

)'it+%onstruct %ontainer+5ocal

!o insert an empty containerclic" %ontainer

Dou*le clic" the stage an' a'' stages an' lin"s

s

+econstructing local Container


187/216


1-1'&Ver.1.0

!o %onvert container in to group of 'iscrete stages an'lin"s in the o*.

Select the %ontainer Stage an' choose Deconstructfrom shortcut menu

diting local containers


188/216


1-1''Ver.1.0

%hoose the container an' clic" )'it++

&roperties

Shared Container


189/216


1-1',Ver.1.0

!o store the eisting stages an' lin"s in the share'container

%hoose the stages an' lin"s

%hoose )'it%ontainer+Share'

&arameters to the components are copie' to share'

container as %ontainer &arameters Saving it is same as saving a o*.

6o5 Se8uences


190/216


1-1,0Ver.1.0

Speci4es a se2uence of o*s to run. Se2uence can contain control information

ie, It is possi*le specify 'i3erent course of action to*e ta"en 'epen'ing on -hether a o* succee's orfails.


191/216


1-1,1Ver.1.0


192/216

(estarta5le se8uence


193/216


1-1,"Ver.1.0


194/216


1-1,#Ver.1.0

ile ++Ce-+++


195/216


1-1,$Ver.1.0

&alette

(epository

Activity Stages


196/216


1-1,%Ver.1.0


197/216


1-1,&Ver.1.0

Ceste' con'itions

Allo-s you to further *ranch the eecution of ase2uence 'epen'ing on a con'ition.

Se2uencer

Allo-s you to synchronize the control 6o- of

multiple activities in a o* se2uence. Start an' en' loop

!ogether these t-o stages allo- you to implement aor...Cet or or...)ach loop -ithin your se2uence

!erminatorAllo-s you to specify that, if certain situations occur,

the o*s a se2uence is running shut 'o-n cleanly

Activity stages


198/216


1-1,'Ver.1.0

7ser Varia*le

Allo-s you to 'e4ne varia*les -ithin a se2uence.!hese varia*les can then *e use' later on in these2uence, for eample to set o* parameters.

)ceptional han'ler

It is eecute' if a o* in the se2uence fails to run9other eceptions are han'le' *y triggers: or if a

o* a*orts an' the Automatically han'le activitiesthat fail option is set for the se2uence.

$nly one )ception han'ler for a se2uence.

riggers


199/216


1-1,,Ver.1.0

!riggers provi'e control information to the StageActivities

Speci4es 'i3erent courses of action to *e ta"en *ase'on o*s status.

!rigger names must *e uni2ue

!ypes of !riggers%on'itional

7ncon'itional

$ther-ise

6o5 Se8uence Properties

Speci4esparameters

Displays co'e


200/216


1-200Ver.1.0

Scenario or )o5 Se8uence

N i 4l il *l i f l' i h h


201/216


1-201Ver.1.0

N input 4les are availa*le in a fol'er -ith the same

layout

Single Server


202/216


1-202Ver.1.0

/aits fortrigger 4le toappear

/hen any failureoccurs, control istransferre' here

)ecutes Sort


203/216


1-20"Ver.1.0

ppear option / does not deletethe file

after finding it

Current value of counter stored in

stage_label.2Counter

Progra!!ing in +ataStage

& i t


204/216


1-20#Ver.1.0

&rogramming components

(outines

!ransforms

unctions

)pressions

Su*routines

Macros

&rece'ence rules

(outines

( ti t ' i th ( ti f l' * ' f lt


205/216


1-20$Ver.1.0

(outines are store' in the (outines fol'er *y 'efault.

!he follo-ing components are classi4e' as routines

!ransform functions

BeforeAfter Su*routines

/hile 'esigning a o* it is possi*le to specify

%ustom 7niverse functions

ActiveP functions

7ecuting )o5s ro! co!!and line

d j b [ d [ NO!# $ %&%' $ (#)*'%


206/216


1-20%Ver.1.0

dsjob -run [ -mode [ NO!# $ %&%' $ (#)*'%+ + [ -param name,value + [ -arn n + [ -rosn + [ -ait + [ -stop + [ -jobstatus+ [-userstatus+ [-local+ [-opmetadata ['.% $#&%++ [-disableprjhandler+ [-

disablejobhandler+ [useid+ project job$job0id

Co!!ands

'sa'min comman'


207/216


1-20&Ver.1.0

'sa'min comman'

DSPImport Service comman'

Sync&roect comman'

Peror!ance tuning in +S

)nsure proper in'ees are create'


208/216


1-20'Ver.1.0

)nsure proper in'ees are create'.

&artition the ta*le -hereever re2uire'.

7se multiple no'es.

7se A&!D7M&S%$()

!ry to use or'er *y than sort.

Scenarios

Scenario 1


209/216


1-20,Ver.1.0

Scenario 1

If -e have ; o*s in se2uencer -hile running if o* 1is faile' then ho- to run other @ o*s

&roperties++trigger++++uncon'itional

Scenario @

!ry 5eft outer oin using 5oo"up stage

Server 6o5 Stages in '.1.2

%omple lat 4le Stage


210/216


1-210Ver.1.0

%omple lat 4le Stage

ol'er Stage

8ashe' 4le Stage

Se2uential 4le stage

Aggregator Stage

%omman' Stage

Interprocess Stage

!& plugin stage

5in" %ollecte' stage

5in" &artitioner stage

Server )o5 stages

Merge Stage


211/216


1-211Ver.1.0

Merge Stage

&ivot Stage

(o- merger Stage

(o- Splitter Stage

Sort Stage

!ransformer Stage


212/216

Parallel )o5 Stages

&rocessing stages


213/216


1-21"Ver.1.0

&rocessing stages

!ransformer Stage

Basic !ransformer Stage

Aggregator Stage


214/216


1-21#Ver.1.0

%ompress Stage

)pan' stage

%opy Stage

Mo'ify Stage

ilter Stage

)ternal 4lter Stage

%hange capture stage

%hange apply Stage

Di3erence Stage

%ompare Stage

Parallel )o5 Stages

)nco'e stage


215/216


1-21$Ver.1.0

)nco'e stage

Deco'e Stage

S-itch Stage

!& )nterprise stage

#eneric stage

Surrogate "ey generator stage

Slo-ly %hanging 'imension Stage

&ivot )nterprise Stage

%hec"sum stage

(estructure stage

%olumn Import Stage


216/216

%olumn Import Stage

%olumn )port Stage

Ma"e Su*recor' stage

Split Su*recor' stage

%om*ine recor' stage

&romote su*recor' stage

Ma"e Vector Stage

Split Vector Stage

Data Stage PPT Materials

Documents