DataStage Admin Activities March 2010 Wipro Technologies @ Office Depot © Wipro Ltd - Confidential June 6, 2022 1
Oct 15, 2014
DataStage Admin Activities
March 2010
Wipro Technologies@ Office Depot
© Wipro Ltd - ConfidentialApril 7, 2023 1
Confidential © Copyright 2007 Wipro Ltd
Content
April 7, 2023 © Wipro Ltd - Confidential 2
• Purpose & Scope• Start/Stop DS Engine• To Check the status of DS Engine• To Release Job locks• To Delete Jobs• To Delete Datasets• To Restore a Dataset• System Monitoring Tools
– iostat– vmstat
• To View Who Last Modified a Job• DataStage Housekeeping • Add Environment Variables• Suggested Environment Variables For Reporting • Import DataStage Jobs From Command Prompt• Export DataStage Jobs From Command Prompt• Compiling Jobs• Message Handler• To Configure/Test DataBase connectivity
– Oracle– Other Databases
Confidential © Copyright 2007 Wipro Ltd
Purpose & Scope
April 7, 2023 © Wipro Ltd - Confidential 3
• Purpose
Purpose of this Presentation is to give some knowledge on DataStage Admin Activities performed on Server side and DataStage Client Side.
• Scope
This document will educate you about DataStage repository database (UV). This presentation doesn’t cover all the properties of UV Database. This document will educate you about Configuring and trouble shooting
the database connectivity for DataStage.
Confidential © Copyright 2007 Wipro Ltd
Start/Stop DS Engine
April 7, 2023 © Wipro Ltd - Confidential 4
From time to time you may need to stop or restart the DataStage engine, for example, when you want to shut down the server.
A script “uv” is provided for these purposes.
su to dsadm (su – dsadm) and thenIssue appropriate stop or start command below.
To stop the server engine, use:
# dshome/bin/uv -admin –stop
To restart the server engine, use:
# dshome/bin/uv -admin –start
Confidential © Copyright 2007 Wipro Ltd
To Check the status of DS Engine
April 7, 2023 © Wipro Ltd - Confidential 5
The following command is used to check the status of the DS Engine. $ uv -admin –info
The following data will be displayed
$ uv -admin -infoDetails for DataStage Engine release 7.5.2.1 instance "ade"================================================================Install history : Installed by root (admin:dsadm) on: Feb 03 2005 14:18:21 : Upgraded by root (admin:dsadm) on: Sep 21 2006 17:20:25 : Upgraded by root (admin:dsadm) on: Sep 21 2006 19:53:45 : Upgraded by root (admin:dsadm) on: Sep 21 2006 20:57:20 : Upgraded by root (admin:dsadm) on: Sep 22 2006 10:17:17 : Upgraded by root (admin:dsadm) on: Sep 22 2006 11:53:46 : Upgraded by root (admin:dsadm) on: Sep 22 2006 14:32:55Instance tag : adeEngine status : Running w/active nlsEngine location : /ascential/Ascential/DataStage/DSEngineBinary location : /ascential/Ascential/DataStage/DSEngine/binImpersonation : EnabledAdministrator : dsadmAutostart mode : disabledAutostart link : noneStartup script : /ascential/Ascential/DataStage/DSEngine/sample/ds.rcCache Segments : 0 activeUser Segments
Confidential © Copyright 2007 Wipro Ltd
To Release Job locks
April 7, 2023 © Wipro Ltd - Confidential 6
There are Three methods to unlock the DataStage jobs
– Using DataStage Administrator Tool.– Using UV Utility – Using DataStage Director
Confidential © Copyright 2007 Wipro Ltd
Unlock jobs -Using DataStage Administrator Tool
April 7, 2023 © Wipro Ltd - Confidential 7
• Log into Administrator (as dsadm)• Open the Command line for the project• Execute “LIST.READU EVERY”• Identify the values for INODE and USER columns for the job for which the locks
need to be released.• Execute UNLOCK INODE <inodenumber> ALL• UNLOCK USER <user number> ALL
Confidential © Copyright 2007 Wipro Ltd
Unlock jobs -Using UV Utility
April 7, 2023 © Wipro Ltd - Confidential 8
• Logon to Unix• Su to dsadm• Go to a corresponding project• Type “UV”• Type “DS.TOOLS”• Select the option “4” (4. Administer processes/locks >>)• Again Select the option “4” (4. List all locks)• Identify the values for PID columns for the job for which the locks need to be
released.• Then select the Option “7” (7. Clear locks held by a process)• Give the PID value here(Enter pid#=) and press Enter.
Confidential © Copyright 2007 Wipro Ltd
Unlock jobs -Using DataStage Director
April 7, 2023 © Wipro Ltd - Confidential 9
– Open DataStage Director– Go to the Job – Cleanup Resources– In the Job Resources window,– Select “Show All” (Processes)– Find your User Name and click on “Logout”
Confidential © Copyright 2007 Wipro Ltd
To Delete Jobs
April 7, 2023 © Wipro Ltd - Confidential 10
The DataStage jobs can be deleted either via DS Administrator or the TCL prompt.
1. In Administrator select the project and run this command:DELETE DS_JOBS <job name>
2. From the TCL prompt you will need to do this:$DSHOME/. ./dsenv./bin/uvshLOGTO <project name>DELETE DS_JOBS <job name>
Using TCL, you could automate this via script. Also developers can delete the jobs through Designer Client.
Note: The deleted jobs can be restored only with the disk backup
Confidential © Copyright 2007 Wipro Ltd
To Delete Datasets
April 7, 2023 © Wipro Ltd - Confidential 11
The Datasets in DataStage can be deleted using following two methods,
1. Using orchadmin2. Using DataSet Management
Using orchadmin
The orchadmin command is used to delete the datasets$APT_ORCHHOME/bin/orchadmin delete <dataset name>For Example orchadmin delete
/detld1/etl/Ascential//DataStage/Projects/Staging/EDPRO_20100225_129.ds
Using DataSet Management
The DataSet Management utility is available in DataStage Manager and Director. Using this we can delete the datasets.
Confidential © Copyright 2007 Wipro Ltd
To Restore a Dataset
April 7, 2023 © Wipro Ltd - Confidential 12
The dataset can be restored from a disk backup
• Restore the header of the dataset, which will be found in the directory where
specified in the job.
• Restore the detail records of the dataset, Which will be found in the resource
directory specified in the config file used to run the job.
Confidential © Copyright 2007 Wipro Ltd
System Monitoring Tools
April 7, 2023 © Wipro Ltd - Confidential 13
iostat
iostat is useful for examining the throughput of various disk resources. If one or more disks have high throughput, understanding where that throughput is coming from is vital. iostat can also help a user to determine if there is excessive IO for a specific job.
The specifics of iostat output vary slightly from system to system. Here is an example from a Linux machine which shows a relatively light load:
(The first set of output is cumulative data since the machine was booted)
$ iostat 10Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtndev8-0 13.50 144.09 122.33 346233038 293951288
… every N seconds (10 in the command line example) iostat outputs:Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtndev8-0 4.00 0.00 96.00 0 96
Confidential © Copyright 2007 Wipro Ltd
System Monitoring Tools
April 7, 2023 © Wipro Ltd - Confidential 14
vmstat vmstat is useful for examining system paging. Ideally, a EE flow, once it begins
running, should never be paging to disk (si and so should be zero). Paging suggests EE is consuming too much total memory.
$vmstat 1 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 10692 24648 51872 228836 0 0 0 1 2 2 1 1 0
vmstat produces the following every N seconds: 0 0 0 10692 24648 51872 228836 0 0 0 0 328 41 1 0 99
mpstat will produce a similar report based on each processor of an SMP
Confidential © Copyright 2007 Wipro Ltd
To View Who Last Modified a Job
April 7, 2023 © Wipro Ltd - Confidential 15
• Logon to Unix• Su to dsadm• Go to a project folder• Type “UV”• Issue the following SQL
SELECT INSTANCE, DTM, MODIFIER FROM DS_AUDIT WHERE CLASS = '2' AND INSTANCE LIKE '%MyJob';
Confidential © Copyright 2007 Wipro Ltd
DataStage Housekeeping
April 7, 2023 © Wipro Ltd - Confidential 16
There is very little housekeeping that can be done from the OS level.
– Clearing /tmp folder – Clearing the &PH& directories– Delete the temporary files, Data Sets or File Sets. It is advised to
do this in the application side itself.
Confidential © Copyright 2007 Wipro Ltd
DataStage Housekeeping
April 7, 2023 © Wipro Ltd - Confidential 17
Before clearing the &PH& directories make sure ever one logged out of the datastage.
– To check the user connections use the command “uv –admin –info” .– From any project directory– . $DSHOME/dsenv– $DSHOME/bin/uv– LOGTO UV– RUN ADMIN_BP REMOVE.SHM– Or $DSHOME/bin/dsdlockd -p
Run the following command for each project.
– LOGTO (Project Name)– CLEAR.FILE &PH&
Confidential © Copyright 2007 Wipro Ltd
Add Environment Variables
April 7, 2023 © Wipro Ltd - Confidential 18
The following are the steps to add the environment variables in DataStage
– Logon to DataStage Administrator – Choose the Project– Go to Environment– Select User Defined tab to add the new used defined variables.– Select the other options to change the values for the existing
variables.
Confidential © Copyright 2007 Wipro Ltd
Suggested Environment Variables for Reporting
April 7, 2023 © Wipro Ltd - Confidential 19
The following environment variables are concerned with various aspects of WebSphere® DataStage® jobs reporting their progress.
– APT_DUMP_SCORE– APT_ERROR_CONFIGURATION– APT_MSG_FILELINE– APT_PM_PLAYER_MEMORY– APT_PM_PLAYER_TIMING– APT_RECORD_COUNTS– OSH_DUMP– OSH_ECHO– OSH_EXPLAIN– OSH_PRINT_SCHEMAS
Confidential © Copyright 2007 Wipro Ltd
Import DataStage Jobs from Command Prompt
April 7, 2023 © Wipro Ltd - Confidential 20
Using DS_IMPORTDSX Command
This command runs within the DS engine using dssh. It canimport any job executables found within specified DSX files. Run it asfollowing,
1. CD to the project directory on the server:2. Run the dssh shell: ..\..\engine\bin\dssh3. At the dssh prompt enter the DS_IMPORTDSX command
The DSXimport command is as follows,
DS_IMPORTDSX filename [[-OVERWRITE] -JOB[S] * | jobname …] | [-LIST]
Confidential © Copyright 2007 Wipro Ltd
Import DataStage Jobs from Command Prompt
April 7, 2023 © Wipro Ltd - Confidential 21
The following are the arguments,
<ilename> The name of the DSX file containing the components to import.
<OVERWRITE> Specify this to overwrite any existing executables of the same name.
<JOB[S]> Specify one or more job executables to import. <LIST> Specify this to list the executables in a .DSX file rather than import them.
If you don’t specify a JOB or a LIST argument, the default is to import all job executables in the specified file.
Confidential © Copyright 2007 Wipro Ltd
Export DataStage Jobs from Command Prompt
April 7, 2023 © Wipro Ltd - Confidential 22
Using dscmdexport command
The dscmdexport command used to export the datastage jobs from Command prompt,
dscmdexport /H hostname /U username /P password /O omitflag project pathname /V
The arguments are as follows: <hostname> Specifies the DataStage Server from which the file will be
exported. <username> The user name to use for connecting to the DataStage Server. <password> The user’s password.
Confidential © Copyright 2007 Wipro Ltd
Export DataStage Jobs from Command Prompt
April 7, 2023 © Wipro Ltd - Confidential 23
<omitflag> Set this to 1 to omit the username and password (onlypossible if you are connected to the DataStage Server via LANManager).<project> Specify the project to export the components from.<pathname> The file to which to export.<V> Use this flag to switch the verbose option on.For example, the following command exports the project dstage2from the R101 to the file dstage2.dsxdscmdexport /H R101 /O 1 dstage2 C:/scratch/dstage2.dsxMessages from the export are sent to the console by default, but canbe redirected to a file using '>', For example:dscmdexport /H R101 /O 1 dstage99 c:/scratch/project99.dsx /V >
c:/scratch/exportlog
You can simply type dscmdexport at the command prompt to get help on the command options
Confidential © Copyright 2007 Wipro Ltd
Compiling Jobs
April 7, 2023 © Wipro Ltd - Confidential 24
There is a program on the Client side called dscc.exe. This resides in the Clients\Classic folder where the Designer and Director reside. This is used to Compile the jobs.
C:\IS81CL\Clients\Classic>dscc /?/? Show program usage/bo (value) BuildOp to compile, * = All BuildOps, \folder\* = folder/d (value) ASB Domain Name/f Use the Force Compile/h (value) Name of Host to attach to/j (value) Job to compile, * = All Jobs, \folder\* = folder/jt (value) Job Type To Compile -1 (Default) = All Types 0 = Server 1 = Mainframe 2 = Sequence 3 = Parallel/mfcgb (value) Mainframe Code Gen base directory location/mful (value) Mainframe Job Upload Profile/ouc Only Compile Uncompiled Objects/p (value) Passwordproject Project To Attach to/qs (value) QS objects to provision * = All QS objects, \folder\* = folder/qspa Provision all dependant QS objects/r (value) Routine to compile, * = All Routines, \folder\* = folder/rd (value) Report Directory and Name/rt (value) Type of report to produce : X = xml, T = txt - default/u (value) User Name
Also the jobs can be compiled from DataStage Designer Client or from Manager through “Run Multiple Job Compile” Option.
Confidential © Copyright 2007 Wipro Ltd
Message Handler
April 7, 2023 © Wipro Ltd - Confidential 25
The Message Handler is used to suppress the Warnings.
• Log into the Director• Select the job• And then view the logs• Select the warning• Right click on the Warning message• A popup will display• Select “ Add Rule to the Message Handler”• Now choose the “Add rule to chosen message handler”• And click on the Button “ Add Rule”
Confidential © Copyright 2007 Wipro Ltd
To Configure/Test DataBase Connectivity
April 7, 2023 © Wipro Ltd - Confidential 26
For Oracle
Configure Tnsnames.ora file, Which defines databases addresses for establishing connections. This file normally resides in the ORACLE HOME\NETWORK\ADMIN directory
Add the following entry in your TNSNAMES.ORA file and change the values shown in brackets to suit your environment:
<addressname> = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(Host = <hostname>)(Port = <port>)) ) (CONNECT_DATA = (SERVICE_NAME = <sid>) ) )
This can also be done through LDAP
Confidential © Copyright 2007 Wipro Ltd
To Configure/Test DataBase Connectivity
April 7, 2023 © Wipro Ltd - Confidential 27
Tnsping
TNSPING is a utility in the ORACLE HOME/bin directory used to test if a SQL*Net connect string can connect to a remote listener (check if the socket is reachable).
Note: This utility only tests if the listener is available. It cannot tell if the databases behind the listener is up or not
For Example
$ tnsping myDB
TNS Ping Utility for Linux: Version 10.2.0.1.0 - Production on 24-MAY-2007 08:55:13Copyright (c) 1997, 2005, Oracle. All rights reserved.Used parameter files:/app/oracle/product/10.2.0/db_1/network/admin/sqlnet.ora
Used TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521)) (CONNECT_DATA = (SERVER =
DEDICATED) (SERVICE_NAME = MyDB)))OK (10 msec)
Confidential © Copyright 2007 Wipro Ltd
TO Configure/Test DataBase Connectivity
April 7, 2023 © Wipro Ltd - Confidential 28
When using an LDAP (OID) server - NAMES.DIRECTORY_PATH=(LDAP) in sqlnet.ora:
$ tnsping myDB
TNS Ping Utility for Solaris: Version 9.2.0.7.0 - Production on 10-DEC-2007 15:05:50Copyright (c) 1997 Oracle Corporation. All rights reserved.Used parameter files:/app/oracle/product/10.2.0/db_1/network/admin/sqlnet.ora
Used LDAP adapter to resolve the aliasAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST =
localhost)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = MyDB)))
OK (300 msec)Retrieved from "http://www.orafaq.com/wiki/Tnsping"Categories: Network services | Utilities
Confidential © Copyright 2007 Wipro Ltd
To Configure/Test DataBase Connectivity
April 7, 2023 © Wipro Ltd - Confidential 29
The SQLPLUS is used to check the databases behind the listener is up or not
The following is the syntax,
sqlplus username/password@servicename
Confidential © Copyright 2007 Wipro Ltd
To Configure/Test DataBase Connectivity
April 7, 2023 © Wipro Ltd - Confidential 30
Other Databases
• Add an entry in odbc.ini file.• Every DSN to which the application connects must have an entry in this file.
The following example shows the configuration for a DSN called Informix 9
[Informix9] Driver=/work/informix/lib/cli/iclis09b.so Description=Informix 9.x ODBC Driver LogonID=user1 pwd=tigress4 Database=odbc_demo ServerName=my_server
Confidential © Copyright 2007 Wipro Ltd
TO Configure/Test DataBase Connectivity
April 7, 2023 © Wipro Ltd - Confidential 31
The datadirect example program is used to check where the database is in relation to the DS Server.
The following are the steps used to check this,
$cd /export/home/752NLS/Ascential/DataStage/DSEngine$. ./dsenvThen cd to the branded_odbc/example directory $cd /export/home/752NLS/Ascential/DataStage/branded_odbc/exampleand run the example program $./example
When prompted put in the odbc entry name and the credential similar to this below sample:
$ ./example./example DataDirect Technologies, Inc. ODBC Example Application.Enter the data source name : <yourdatasourcename from .odbc.ini>Enter the user name : <yourusername>Enter the password : <youruserpassword>
And then you should see the below (or otherwise some type of error)
Enter SQL statements (Press ENTER to QUIT)SQL>Exiting from the Example ODBC program
Thank You
April 7, 2023 © Wipro Ltd - Confidential 32