Informatica PowerMart / PowerCenter 6.x Upgrade Features Ted Williams Informatica Corporation, 2002. All rights reserved.
Dec 15, 2015
Informatica PowerMart / PowerCenter 6.x
Upgrade Features
Ted Williams
Informatica Corporation, 2002. All rights reserved.
2
Informatica PowerCenter 6 Architecture
Not Shown: Client ODBC Connections for Source and Target metadata
HeterogeneousTargets
HeterogeneousSources
Repository
Targets
RepositoryServer
RepositoryAgent
Sources
Repository Designer Workflow Workflow Manager Monitor Manager
3
Repository Server
Each repository has an independent architecture for the management of the physical repository tables
Components: one Repository Server and a Repository Agent for each repository
Repository
RepositoryRepositoryServerServer
RepositoryRepositoryAgent(s)Agent(s) Repository Repository
Manager Manager
ServerServer
Client overhead for repository management is greatly reduced by the Repository Server
4
Repository Server
The Repository Server manages connections to the repository from client applications
The Repository Server can manage multiple repositories on different machines on a network
For each repository it manages, the Repository Server uses one process, called the Repository Agent, to insert, update, and fetch objects from the repository database tables
The Repository Server maintains object consistency by controlling object locking
The Repository Server runs on the same system running the Repository Agent
5
Repository Server Console
6
Designer Enhancements
Heterogeneous Targets ReplaceChr and ReplaceStr Copy and Paste (Designer) Sorter Object Transaction Control Object Cascading code changes More robust migration via XML import/export (Replace option) Compare Objects Compare Mappings Highlight Path Links Removed restrictions on Mapplets Dynamic Lookup Cache Update support Expand Mapplets for visualization in Mappings True Flat-file target objects Metadata Extensions for better reporting
7
Session Manager to Workflow Manager Change
• Represents the largest change to date
• Similar in structure to Designer
• New objects/tasks
• Concept of Links
• Better monitoring capabilities
8
Workflow ManagerTasksTasks
LinksLinks
Variables/Variables/EventsEvents
Work Work FoldersFolders
9
New Connection Features
Connection Dialogs
Replace Connections
Register Server
Assign to Workflows
10
Connection DialogsType FilterType Filter
Copy AsCopy As
Environmental Environmental OptionsOptions
- Type filters allow for easy - Type filters allow for easy viewing of Database specific viewing of Database specific connectionsconnections
- Copy As feature allow for - Copy As feature allow for quick propagation of main quick propagation of main connection info (maybe only connection info (maybe only user and password is user and password is different)different)
- Most significant change is - Most significant change is the ability to assign a the ability to assign a specific rollback segment specific rollback segment and/or prepare database and/or prepare database with environment SQLwith environment SQL
11
Replace Connections
Replace Replace ConnectionsConnections
With folder closed, click on the Replace Connections button.
This brings up the sub-dialog that allows a From-To operation to allow the global replace of one connection with another in the sessions.
12
Register Server
Similar to Relational Connection dialog
Same parameters apply with the exception of the new variables for Workflow logs.
13
Assign to Workflows
While folders are closed it is possible to assign choose sessions to a particular server
This dialog allows for individual or globally select sessions to be assigned to run on a particular server
14
New Objects (Tasks)
Local Tasks – Sessions
Commands
Decision
Assignment
Timer
Control
Event Raise
Event Wait
Global Tasks – Sessions
Commands
Tasks are now the default units of work for building the workflow. Global tasks are reusable across workflows. Local tasks are independent and self-contained within workflows.
15
Sessions (Updates)
GeneralGeneral
First page allows for options to treat conditional links attached to the object as AND/OR functionality. Also control option to fail the parent (container) if task fails or does not run. Disabling a task in a workflow allows the task to be skipped instead of having to remove it.
Updated Updated parametersparameters
16
Sessions (Updates)
PropertiesProperties
Addition to this page is the option to choose a different connection to resolve $Source and $Target used in lookups.
**Limitation - You can use $DBConnectionVariable but it will only resolve if the parameter is evaluated with the appropriate Source or Target Connection
If used when sourcing and targeting flat-files the parameter will not resolve.
Updated Updated connection connection resolutionresolution
17
Sessions (Updates)
Config ObjectConfig Object
This area is where typical allocation of memory occurs, log file options and error handling This area is where typical allocation of memory occurs, log file options and error handling strategies. The upgrade here is with the concept of maintaining common setting in a config strategies. The upgrade here is with the concept of maintaining common setting in a config object which is setup separately and chosen from a drop-down list.object which is setup separately and chosen from a drop-down list.
Drop-down for Drop-down for common common settingssettings
18
Config Object
Config ObjectConfig Object
By creating a session configuration object you can cut down on repetitively assigning memory By creating a session configuration object you can cut down on repetitively assigning memory and log options. When migrating sessions you can optionally migrate configurations or maintain and log options. When migrating sessions you can optionally migrate configurations or maintain environment specific settings to be reassigned.environment specific settings to be reassigned.
Setup for reusable Setup for reusable Configuration objectConfiguration object
19
Sessions (Updates)
SourcesSources
This area has primarily just been subdivided into sub tabs. The main addition is the filter drop-This area has primarily just been subdivided into sub tabs. The main addition is the filter drop-down that allows for either viewing all source object settings or by individual source object. down that allows for either viewing all source object settings or by individual source object. Instead of ‘filelist’ option it is either direct or indirect (filelist)Instead of ‘filelist’ option it is either direct or indirect (filelist)
FilterFilter
20
Sessions (Updates)
TargetsTargets
Same subdivisions as sources. However in the Properties sub tab is where the ‘Target Options’ Same subdivisions as sources. However in the Properties sub tab is where the ‘Target Options’ are defined. The writers sub tab preserves the legacy operation of writing to a flat-file while are defined. The writers sub tab preserves the legacy operation of writing to a flat-file while using a database specific target (as opposed to the new flat-file target). using a database specific target (as opposed to the new flat-file target).
FilterFilter
21
Source Target Connections
ConnectionsConnections
The drop-down list allows to select any available connection as well as specifically define the use The drop-down list allows to select any available connection as well as specifically define the use of a Connection Variable of a Connection Variable
Connection Connection dialogdialog
22
Sessions (Updates)
ComponentsComponents
The area where commands or email unique to this object can be defined. You can alternately The area where commands or email unique to this object can be defined. You can alternately select a reusable task to use as well. select a reusable task to use as well.
Choice of Choice of reusable or reusable or
local commandlocal command
23
Non-Reusable Commands
ComponentsComponents
Regardless of reusable or non-reusable it is necessary to name the object since there is Regardless of reusable or non-reusable it is necessary to name the object since there is potential to promote it.potential to promote it.
Option for local Option for local or reusableor reusable
Name of Name of command command
objectobject
24
Non-Reusable Commands
ComponentsComponents
The properties tab allows for error control for commands/tasksThe properties tab allows for error control for commands/tasks
Error Control for Error Control for multiple multiple
commands/taskscommands/tasks
25
Non-Reusable Commands
ComponentsComponents
The Command tab allows for editing of commands (obviously)The Command tab allows for editing of commands (obviously)
Command editorCommand editor
26
Reusable Commands
ComponentsComponents
A non-reusable command can be promoted to reusable or else you can select one from available A non-reusable command can be promoted to reusable or else you can select one from available tasks.tasks.
Reusable Reusable Command Command
Selection DialogSelection Dialog
27
Reusable Commands
ComponentsComponents
Limitation – a Limitation – a non-reusablenon-reusable and and reusablereusable command command cancan resolve Server Variables but only from resolve Server Variables but only from within the Components tab of the session. A command can within the Components tab of the session. A command can NOTNOT resolve Server Variables if resolve Server Variables if used in an external used in an external Command TaskCommand Task. Understand before promoting.. Understand before promoting.
LimitationLimitation
28
Sessions (Updates)
TransformationsTransformations
The transformations area preserves all the previous capabilities. The update is the addition of The transformations area preserves all the previous capabilities. The update is the addition of the main instance filter and another filter button that only displays transformation where overrides the main instance filter and another filter button that only displays transformation where overrides can be performed. Also this is where you can input the NEW pre/post SQL commands.can be performed. Also this is where you can input the NEW pre/post SQL commands.
Filter for modifiable Filter for modifiable transformationstransformations
Instance FilterInstance Filter
29
Sessions (Updates)
PartitionsPartitions
New partitioning scheme allows for repartitioning after Source Qualifier at almost any other New partitioning scheme allows for repartitioning after Source Qualifier at almost any other transformation object in the mapping. There are four main partition types Pass Through, Round transformation object in the mapping. There are four main partition types Pass Through, Round Robin, Hash Auto Keys, Hash User Keys.Robin, Hash Auto Keys, Hash User Keys.
Add Partition pointsAdd Partition points
Change Partition TypeChange Partition Type
30
Sessions (Updates)
Metadata ExtensionsMetadata Extensions
Metadata extensions are available to individual object/tasks or can be made reusable and global. Metadata extensions are available to individual object/tasks or can be made reusable and global. These columns can not be auto-populated with functions but they do allow the extendibility for These columns can not be auto-populated with functions but they do allow the extendibility for more reporting options (e.g., Developer Name, Version, Creation Date, etc.)more reporting options (e.g., Developer Name, Version, Creation Date, etc.)
Metadata columnMetadata column
Promote to Promote to reusablereusable
31
Command Tasks
CommandCommand
The command object can be created globally under the Task Developer. It can also be The command object can be created globally under the Task Developer. It can also be promoted here from within a mapping. promoted here from within a mapping.
Created in Task Created in Task DeveloperDeveloper
32
Command Tasks
CommandCommand
The properties section homes the ability to either run all commands regardless or run them if The properties section homes the ability to either run all commands regardless or run them if each previous command completes. Commands tab is where the actual commands are created. each previous command completes. Commands tab is where the actual commands are created. One command per line. One command per line.
Process Control Process Control for multiple for multiple commandscommands
33
Email Tasks
EmailEmail
Email task is very similar to the command task since it can be either created in the Task Email task is very similar to the command task since it can be either created in the Task Developer or promoted from a mapping. The properties tab allows for an expression editor for Developer or promoted from a mapping. The properties tab allows for an expression editor for text creation utilizing the built-in variables.text creation utilizing the built-in variables.
Email text Email text creation dialogcreation dialog
Built-in Built-in VariablesVariables
34
Links and Conditions
DefiinitionDefiinition
Links and their underlying conditions are what provide process control to the workflow. When an Links and their underlying conditions are what provide process control to the workflow. When an attached link condition resolves to TRUE then the attached object may begin processing. There attached link condition resolves to TRUE then the attached object may begin processing. There can be no looping and links can only execute once per workflow. However more complex can be no looping and links can only execute once per workflow. However more complex branching and decisions can be made by combining multiple links to a single object or branching branching and decisions can be made by combining multiple links to a single object or branching into decision type paths. Each link has its own expression editor and can utilize upstream into decision type paths. Each link has its own expression editor and can utilize upstream resolved object variables or user-defined variables for its own evaluation.resolved object variables or user-defined variables for its own evaluation.
Link conditionLink condition
35
Links and Conditions
Object VariablesObject Variables
The default set of object variables from a session can provide more information than just a status The default set of object variables from a session can provide more information than just a status of ‘Completed’. More complex evaluation can be done for ErrorCode, StartTime, of ‘Completed’. More complex evaluation can be done for ErrorCode, StartTime, SrcSuccessRows, etc. In addition to the default object variables, User Defined variables can be SrcSuccessRows, etc. In addition to the default object variables, User Defined variables can be created and populated via parameter files or changed in the workflow via Assignment tasks. created and populated via parameter files or changed in the workflow via Assignment tasks. Also any upstream task that has completed can have it’s variables utilized in downstream link Also any upstream task that has completed can have it’s variables utilized in downstream link conditions.conditions.
Object Object VariablesVariables
36
Workflow Variables
Pre-defined VariablesPre-defined Variables
This is the list of all pre-defined task level variables available to evaluate uponThis is the list of all pre-defined task level variables available to evaluate upon
Variable Task Type Datatype ** Supported Status Returns
ABORTED
DISABLED
FAILED
NOTSTARTED
STARTED
STOPPED
SUCCEEDED
Condition Decision Task Integer
EndTime All tasks Date/time
ErrorCode All tasks Integer
ErrorMsg All tasks Nstring*
FirstErrorCode Session task Integer
FirstErrorMsg Session task Nstring*
PrevTaskStatus All tasks Integer
SrcFailedRows Session task Integer
SrcSuccessRows Session task Integer
StartTime All tasks Date/time
Status** All tasks Integer
TgtFailedRows Session tasks Integer
TgtSuccessRows Sessions Integer
TotalTransErrors Sessions Integer
* Variables of type Nstring can have a maximum length of 600 characters.
37
Workflow Variables
User-defined VariablesUser-defined Variables
Variables are created at the container level much like the mappings. (Workflows=Mappings, Variables are created at the container level much like the mappings. (Workflows=Mappings, Worklets=Mapplets). Once created values can be passed to objects within the same container Worklets=Mapplets). Once created values can be passed to objects within the same container for evaluation. (Assignment Task can modify/calculate variables)for evaluation. (Assignment Task can modify/calculate variables)
Edit VariablesEdit Variables
38
Workflow Variables
User-defined VariablesUser-defined Variables
A user-defined variable can assist in more complex evaluations. In the above example, an A user-defined variable can assist in more complex evaluations. In the above example, an external parameter file contains the number of expected rows. This in turn is evaluated against external parameter file contains the number of expected rows. This in turn is evaluated against the actual rows successfully read from an upstream session. $ signifies and is reserved for pre-the actual rows successfully read from an upstream session. $ signifies and is reserved for pre-defined variables. User defined variables should maintain $$ naming.defined variables. User defined variables should maintain $$ naming.
User Defined User Defined VariablesVariables
Pre-Defined Pre-Defined VariableVariable
39
Parameter File
FormatFormat
The format is slightly different going forward for declaring variables but legacy naming is The format is slightly different going forward for declaring variables but legacy naming is preserved with the foldername.sessionpreserved with the foldername.session
•Workflow variables: [folder name.WF:workflow name]
•Worklet variables: [folder name.WF:workflow name.WT:worklet name]
•Worklet variables in nested worklets: [folder name.WF:workflow name.WT:worklet name.WT:worklet name...]
•Session parameters, plus mapping parameters and variables:[folder name.WF:workflow name.ST:session name]
or [folder name.session name]
or [session name]
40
Assignment Task
UsageUsage
The assignment task allows for workflow variables to be manipulated, assigned or calculated and The assignment task allows for workflow variables to be manipulated, assigned or calculated and stored (either for the session run or as persistent) for downstream evaluation and condition stored (either for the session run or as persistent) for downstream evaluation and condition resolution. The Expression Section allows for the selection of the appropriate variable to resolution. The Expression Section allows for the selection of the appropriate variable to manipulate along with the assignment through an Expression Editor. In order to use it is manipulate along with the assignment through an Expression Editor. In order to use it is necessary to first create a variable through the Edit Variables dialognecessary to first create a variable through the Edit Variables dialog
Edit VariablesEdit Variables
41
Event Task
UsageUsage
Events are similar to variables. They are repository tags that are a little more flexible (cleaner) Events are similar to variables. They are repository tags that are a little more flexible (cleaner) than dealing with indicator files (although event watch can be used with indicator files). Before than dealing with indicator files (although event watch can be used with indicator files). Before utilizing the new functionality it is necessary to create these Event tags within the container that utilizing the new functionality it is necessary to create these Event tags within the container that they will be used. Events can not be monitored across workflows or worklets (even if a worklet is they will be used. Events can not be monitored across workflows or worklets (even if a worklet is part of the workflow).part of the workflow).
Edit EventsEdit Events
42
Event Task
UsageUsage
If using Event tags then an Event Raise is used in conjunction with an Event Wait. In the above If using Event tags then an Event Raise is used in conjunction with an Event Wait. In the above example two branches are executed in parallel. The second session of the lower branch will example two branches are executed in parallel. The second session of the lower branch will remain in stasis until the upper branch completes triggering the event. The lower branches remain in stasis until the upper branch completes triggering the event. The lower branches event wait task recognizes the event and allows for the second session to start.event wait task recognizes the event and allows for the second session to start.
Event RaiseEvent Raise
Event WaitEvent Wait
43
Event Raise
UsageUsage
To configure the Event Raise task the drop-down box allows for selection of the appropriate To configure the Event Raise task the drop-down box allows for selection of the appropriate user-defined Event tag. This will create an entry in the repository for a matching event wait to user-defined Event tag. This will create an entry in the repository for a matching event wait to look for.look for.
44
Event Wait
UsageUsage
The event wait allows for configuration for an Event Raise (user-defined event) or existence The event wait allows for configuration for an Event Raise (user-defined event) or existence check for an indicator file.check for an indicator file.
User Defined User Defined EventEvent
Indicator FileIndicator File
45
Event Wait
UsageUsage
The properties section of the Event Wait task allows for further definition of behavior. If your The properties section of the Event Wait task allows for further definition of behavior. If your workflow has failed/suspended after Event Raise but before the Event Wait has resolved, then workflow has failed/suspended after Event Raise but before the Event Wait has resolved, then the Enable Past Events is able to recognize that the Event has happened already. If working the Enable Past Events is able to recognize that the Event has happened already. If working with indicator files you have to ability to either delete the file or allow it to stay in case some with indicator files you have to ability to either delete the file or allow it to stay in case some downstream Event Waits are also keying on that file.downstream Event Waits are also keying on that file.
Resume/Restart Resume/Restart SupportSupport
Flat-file CleanupFlat-file Cleanup
46
Decision Task
UsageUsage
The decision task allows for True/False based branching of process ordering. The Decision task The decision task allows for True/False based branching of process ordering. The Decision task can home multiple conditions and therefore downstream links can be evaluated simply upon the can home multiple conditions and therefore downstream links can be evaluated simply upon the Decision being True or False.Decision being True or False.
**Note it is possible to have the decision based on SUCCEEDED or FAILED of previous task, **Note it is possible to have the decision based on SUCCEEDED or FAILED of previous task, however if workflow is set to suspend on error than that branch is suspended and the decision however if workflow is set to suspend on error than that branch is suspended and the decision won’t trigger on a FAILED conditionwon’t trigger on a FAILED condition
47
Control Task
UsageUsage
The control task is utilized in a branching manner to present a level of stoppage during the The control task is utilized in a branching manner to present a level of stoppage during the workflow. Consider if too many sessions have too many failed rows. The options allow for workflow. Consider if too many sessions have too many failed rows. The options allow for different levels such as failing at the object level to Aborting the whole workflow.different levels such as failing at the object level to Aborting the whole workflow.
48
Timer Task
UsageUsage
The timer task has two main ways to be utilized. The first way is by absolute time that is time The timer task has two main ways to be utilized. The first way is by absolute time that is time evaluated by server time or a user-defined variable (that contains the date/time stamp to start).evaluated by server time or a user-defined variable (that contains the date/time stamp to start).
49
Timer Task
UsageUsage
The second usage is by Relative time that offers options of time calculated from when the The second usage is by Relative time that offers options of time calculated from when the process reached this (Timer) task, from the start of the container this task, or from the start of the process reached this (Timer) task, from the start of the container this task, or from the start of the absolute top-level workflow.absolute top-level workflow.
50
Practical
Business CaseBusiness Case
Need for three sessions to wait for Need for three sessions to wait for indicator file(s) to start each one. indicator file(s) to start each one. Window of opportunity is only between Window of opportunity is only between 10AM and 2AM (next morning). If any 10AM and 2AM (next morning). If any one runs an extra session must run. At one runs an extra session must run. At cutoff time there is need to stop the cutoff time there is need to stop the process (polling - not existing runs) so process (polling - not existing runs) so that new activity does not continue that new activity does not continue between 2AM and 10AM. Workflow is between 2AM and 10AM. Workflow is scheduled to run everyday at 10AMscheduled to run everyday at 10AM
Objects Used: •Assignment Task – Assigns the appropriate cutoff time for logic•File Wait Tasks – Polls for the appropriate Indicator files•Timer Task – Assigned to start based on the variable assigned by the Assignment task•Command Tasks – After cutoff time the commands will put an indicator file to release the polling
Link Logic – The remainder of the logic is contained within the links themselves. The main sessions evaluate end time of file wait tasks to the cutoff time. If within cutoff then sessions will run. If over cutoff sessions will not run. The cutoff branch also evaluates to see if file wait tasks are running over. If they are still running then the command tasks will fire.
51
Practical_Descriptive
52
Workflow Monitor
53
Workflow Monitor Task View