Autosys

I N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Concept of AutoSys - Apr 2010

Syed Mohani

2

Agenda / Topics to be Covered I N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Topics:-

----------------------------------

--- What is autosys how it works

--- How to check Autosys is running.

--- Jil (Job Information Language)

--- How to setup a new/delete/upgrade Jobs

--- Autosys vs Cron jobs

--- Common commands used (Operate Prospective)

--- Automatic conversion from Cron jobs to Autosys jobs

--- Changes from Solaris to Linux Environment AutoSys jobs

--- Automation on Monitoring

--- Demo

--- Q & A

3

What is AutosysI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

AutoSys is an automated job management system that lets operate schedule, monitor and report.

This presentation is intended for Operate who will be responsible for

defining, monitoring and managing jobs run by AutoSys.

Using Autosys utilities, we can define a job by assigning it a name and specifying the arrributes

that describe its associated behavior.

--- The AutoSys job information Language (JIL)

4

Software ComponentsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

From a software perspective, the AutoSys environment is composed these main components:-

--- Event Server (Autosys Database)

--- Event Processor

--- Remote Agent

5

Jobs TypesI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

--- Command Jobs

--- File Watcher Jobs

--- Box Jobs

6

Jobs TypesI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

--- Box Jobs

A Box is a container of jobs with like starting conditions, either Date/time conditions

or job dependency conditions.

7

AutoSys Commands and MonitorsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

There are several commands unique to AutoSys that can be used to control, configure, and

report on the system behavior.

---- chk_auto_up (to check connectivity with the database)

---- autoping ( -x returns ver info -D check the database info

---- autorep (to check the job status)

---- autostatus (Status of the job)

8

AutoSys Job DefinitionI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

There are four major JIL subcommands.

--- insert_job Allows to add a new job to AutoSys

--- update_job Allows to change an existing job

--- delete_job Allows to remove an existing job

--- delete_box Removes a box job and all of its subordinate jobs

9

AutoSys Job DefinitionI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

A job performs a particular action based on a set of instructions called a job definition.

Starting Conditions:

--- Data and time condition

--- days_of_week

--- start_times

--- run_window

--- Job dependency conditions

--- success(job_name)

--- failure(job_name)

--- notrunning(job_name)

--- done(job_name)

10

Job StatusI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

At every point in processing, a job has what is called its current status or “state”.

--- STARTING ST (The job is in the process of ‘coming up”)

--- RUNNING RU (The job is running)

--- INACTIVE IN (Job has not been processed, has never been run)

--- SUCCESS SU (Exit code equal)

--- FAILURE FA (Job exited with an exit code)

--- TERMINATED TE (Job terminated while in the RUNNING state)

--- RESTARTED RE (Job was unable to start (or failed) and schedule to restart)

--- ACTIVATED AC (Job is itself has not yet started.)

--- ON_ICE OI (Job is removed from all conditions and logic, but is still defined.

--- ON_HOLD OH (Job is on hold and will not be run until it

receives the JOB_OFF_HOLD.

The difference between "on hold" and "on ice" is that when an "on hold" job is taken off hold, if its

starting conditions are already satisfied, it will be scheduled to run, and it will run. On the other

hand, if an "on ice" job is taken "off ice," it will not start, even if its starting conditions are already

satisfied. This job will not run until its starting conditions reoccur.

11

Common CommandsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

# autorep -j %lmc%

Job Name Last Start Last End ST Run Pri/Xit

uat_lmc_live_rate_Dly 04/14/2010 14:26:03 ----- RU 1868/1

Syntax autorep {-J job_name -M machine_name -G global_name} [-s -d -q -o over_num] [-r run_num]

12

Common CommandsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

# autostatus -j <job-name> i.e test-for-autosys-job-creation

SUCCESS

Checking on Jobs in the STARTING state.:-

# chase -E -A

Display all jobs scheduled to run between these two dates: job_depends -t -J ALL -F "04/01/2010 00:00" -T "12/26/2010 00:00"

Get version information:- # autoflags -a

INC2 LINUX ANY 11.0 SP2 46a989d1 snjcqacriba01

To Check AutoSys is up and running:-

# chk_auto_up

Primary Scheduler is RUNNING on machine: <Machine-name>

# ps –ef |grep autosys # autoping -m <Machine-name>

13

Common CommandsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Checking system Status

--- autoflags -a (info about autosys and system configuration)

--- autoping -m ALL -D (Verifies versions AutoSys comm are correctly)

--- autosyslog -e (Display the Event Processor log files)

--- chase –A –E (Verifies autosys database are running)

--- chk_auto_up (Inspects the DB /environment variables)

Definuing Calendars

--- autocal

--- autocal_asc (Adds, delete, and print custom calendar)

Maintaining Databases

--- archive_events (Removes old info from autosys database)

--- clean_files –d days (Removed Remote Agent Log Files)

--- dbstatistics

Managing Security

--- autosys_secure (Maintain Autosys Edit and Exec super user owner)

14

Common CommandsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Reporting Job Dependencies and Conditions

--- job_depends

Reporting Job status

--- autorep

--- autostatus –j <job-name> (Reports current status of the job)

AutoSys job status send

--- sendevent

Backup of All jobs

--- autorep –J ALL –q > dump_file

15

Common CommandsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

To check the jobs schedule between two Dates:-

# job_depends -t -j ALL -F "04/01/2010 00:00" -T "04/30/2010 00:00“

Job Forecast Report

From: 04/01/2010 00:00:00 To: 04/30/2010 00:00:00

Job Name Next Start Atomic Start Conditions

______________________________ _______________

qa_cri_batch02_queue_Health_c_min 04/21/2010 18:00:00 -------

qa_cri_batch03_queue_Health_c_min 04/21/2010 18:00:00 -------

16

Common CommandsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

To specify that the jobs can be run on any day except those days specified in the <name-holidays> calendar.

exclude_calendar : <name-hoilidays>

Sendevent:-

sendevent -J <job-name> -E CHANGE_STATUS -s TERMINATED

sendevent -J <job-name> -E STARTJOB

sendevent -J <job-name> -E FORCE_STARTJOB

sendevent -J <job-name> -E JOB_ON_ICE

sendevent -E DELETEJOB -J <job-name>

17

Common CommandsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Logs:-

# autosyslog -e -J <job-name>

18

SendEvents OptionsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

To start or force start a job manually using sendevent :

sendevent –E FORCE_STARTJOB -J <job-name>

sendevent -E STARTJOB -J <job-name>

To put jobs on OFF ICE or ON ICE :

sendevent -E OFF_ICE -J <job-name>

sendevent -E ON_ICE -J <job-name>

19

SendEvents OptionsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Sendevents to AutoSys for a variety of purposes, including starting or stopping AutoSys jobs,

stopping the Event processor, and putting a job on hold. This command is also used to set AutoSys

global variables or cancel a scheduled event. sendevent is normally used with "-E" & -J option

-J job_name : Specifies the name of the job to which the specified event should be sent.

-E event :Specifies the event to be sent. This option is required. Any one of the following events

may be specified as:

STARTJOB KILLJOB DELETEJOB FORCE_STARTJOB JOB_ON_ICE JOB_OFF_ICE JOB_ON_HOLD JOB_OFF_HOLD CHANGE_STATUS STOP_DEMON CHANGE_PRIORITY COMMENT ALARM SET_GLOBAL SEND_SIGNAL

20

Command AttributesI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

AUTOREP

USAGE: autorep [-J JobName] [-d | -s | -q | -o OverRide # | -a | -w]

[-B GroupName] [-I ApplicationName] [-t]

[-X ExternalInstanceName] [-Y JobType]

[-R Run_num] [-L PrintLevel] [-N Retry] [-x]

[-z GlobalBlobName] [-f BlobDownloadDirectory]

[-G GlobalName] [-M MachineName] [-D DataSource] [-?]

21

Job Physical loactionI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

# pwd

/cri_staging/syed/jil/lmc

*** 2193 smccuatcriwa06 citwl:ugcrmcit /cri_staging/syed/jil/lmc

# ls -altr

total 20

drwxr-xr-x 3 citwl ugcrmcit 4096 Apr 13 17:27 ..

-rwxr-xr-x 1 citwl ugcrmcit 423 Apr 13 18:31 uat_lmc_live_rate_Dly.bak

-rwxr-xr-x 1 citwl ugcrmcit 423 Apr 14 11:42 uat_lmc_live_rate_Dly.jil.20100414

-rwxrwxrwx 1 citwl ugcrmcit 395 Apr 14 12:07 uat_lmc_live_rate_Dly.jil

22

Jobs StreamsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

A job stream (or Flow) refers to the logic of how related AutoSys jobs are run.

---- Date and Time conditions

---- Job dependency conditions

lob_backup Command job has date and time condition (Runs every day of the week at 01:01 am

Job_clean Command job has a job dependency condition specified for its starting parameters.

It will run only on the successful completions of job_backup.

23

Jobs StreamsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

File Watcher Jobs:-

A file watcher job is used to watch for an incoming file. Job has date and time conditions specified

For its starting condition. Example job is to be run every day of the week at 2:00 pm. It watches

For a file arriving. The job_Fwatch job will check every 15 second for the creation of the incoming

File. When condition is met, job_Fwatch will notify the server and complete with a SUCCESS status.

24

Defining and Running jobs using Job Information Language (JIL)

I N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

We can use AutoSys proprietary scripting language --- JIL to create jobs definitions. This job definition must be submitted to the AutoSys database before the job can be run.

JIL Sub-Commands:-

--- insert_job

--- update_job

--- delete_job

Jil Attributes Types:-

--- Job Name (insert_job: <job-Name>)

--- Job Type (c f and b) (job_type: c)

--- Machine Name (machine: <machine-name>)

--- Starting Conditions (days_of_week: mo, we, fr)

(condition: success(my_job)

--- File to Watch For (watch_file: /tmp/<file-name>

25

JIL SyntaxI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Writing a JIL script, follow the syntax rules:-

1 – sub_commnad: job_name (insert_job: <user-define-job-name>

2 – attribute_keyword: value

Example:-

Creating a job:vi <test-for-autosys-job-creation.jil>

# JIL Script

# This script will re-direct a message to the file /tmp/test-output.out

update_job: test-for-autosys-job-creation job_type: c

machine: cri_batch_job_queue

permission: gx,wx,mx

command: /bin/echo "AutoSys KT demo" > /cri_staging/syed/jil/demo/test-output.out

std_out_file: /cri_staging/syed/jil/demo/test-output.out

Saving a job:

Jil < test-for-autosys-job-creation.jil

26

Jobs DetailsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

# autorep -j uat_lmc_live_rate_Dly -q

/* ----------------- uat_lmc_live_rate_Dly ----------------- */

insert_job: uat_lmc_live_rate_Dly job_type: c

command: /apps/ss/userprojects/lmc/live_rate.sh

machine: cri_web_job_queue

owner: citwl@smccuatcriwa06

permission: gx,wx,mx

date_conditions: 1

days_of_week: mo,tu,we,th,fr

start_times: "22:00"

description: "Live Rate Job"

std_out_file: /local/ss/cit/userprojects/log/lmc/lmc_liverate.stdout

std_err_file: /local/ss/cit/userprojects/log/lmc/lmc_liverate.stderr

alarm_if_fail: 1

27

JIL SyntaxI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Delete a Job:

At the unix Command Prompt

# jil <enter-key>

At the jil prompt

jil>>1> delete_job: <job-name>

exit

28

Job Delete thru jil:-I N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Delete the Job:-Step 1 --- Vi the jil file and replace command with delete_job:

/* ----------------- prd_cit_cap_CPPmonthRun ----------------- */delete_job: prd_cit_cap_CPPmonthRun job_type: ccommand: `cd /home/capbg/scripts; ./CPP_monthly.sh`machine: snjprdcria1.jpmchase.com#owner: [email protected]: gx,ge,wxcondition: success(prd_cit_cap_CPPfile_fwatch)std_out_file: /home/capbg/scripts/CPP_month.stdoutstd_err_file: /home/capbg/scripts/CPP_month.stderralarm_if_fail: 1

Step 2 --- Save the jil file into Database.

# jil < [Jil-file-name]

Step 3 --- received the outpuy from save

Attempting to Delete Job: prd_cit_cap_CPPfile_fwatch

Jobs which are dependent on the

DELETED Job: prd_cit_cap_CPPfile_fwatch

prd_cit_cap_CPPmonthRun

Delete was successful!

Exit Code = 0

29

Auto Transfer Cronjobs into Autosys jobsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Commands:-

cron2jil - Translates crontab files into JIL format

cron2jil -f crontab_file [-d output_directory] [-i include_file] [-m machine] [-p prefix]

---- input as crontab file

---- # cron2jil –f file-name

---- convert files names i.e. cp_[123]

run_calendar: <callendar-name>

autocal_asc -s test_demo -E test_export.txt

Export the calendar entries into file

autocal_asc -lc

List the cycle calendar

Note:

cron2jil does not generate JIL files for jobs that are defined in crontab to start every minute

After the *.jil or *.cal files are generated, you must submit them to the database using the

jil and the autocal_asc commands, respectively.

30

Auto Transfer Cronjobs into Autosys jobsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Combing Calendars:-

Calendars can be combined in a number of ways. For expample, we can create a calendar

that includes all the dates that are in either one calendar or another.

We can import calendar text files. Calendars contained in ASCII test files can be imported

Into AutoSys database. These test files may contain multiple calendars, each of which must

be delimited with the calendar: Calendar_name attributes.

calendar: Q1paydays

01/01/2010

01/15/2010

02/01/2010

02/15/2010

03/01/2010

06/15/2010

calendar: Q1holidays

01/01/2010

31

Auto Transfer Cronjobs into Autosys jobsI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Date Range:-

The following resource sets the number of years in the data range of the calendar, as a

default at start up. This can be overridden manually by way of the Date Range option from

the options menu.

! 1 = prior year plus current year.

! 2 = thru next year, 3 = thru third year,

! 4 = thru forth year, 5 = thru fifth year,

! 10 = thru tenth year

Autocal.dateRange:

32

New Setup for Linux EnvironmentI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Reporting Job Dependencies and Conditions

Virtual Machine Physical Machine

crrt_job_queue citcrrtapp[12]-qa.us.jpmchase.net

crm_crws_job_queue citportalapp[123]-qa.us.jpmchase.net

crm_rad_job_queue citportalapp[12]-qa.us.jpmchase.net

cri_batch_job_queue citssbatchapp[1234]-qa.us.jpmchase.net

cri_web_job_queue citsswebapp[1234]-qa.us.jpmchase.net

machine: cri_web_job_queue    <------ Virtual Machine Name

permission: gx,wx,mx                <------ Job Permissions add “mx”

To convert existing Jobs definitions

Jil out the current job definition from existing environment into a text file Update machine name with virtual machine name (refer next slide )update existing permission to add additional permission “mx”Jil in the job definition to new environment.

33

AutoSys Jobs naming conventionI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

Get the AutoSys jobs status thru web tool.

While apply the naming convention we can write a cgi script and use apache to get

status on web.

Env-App-JobName

XX-XXXXX-XXXXXXXXXXXXXXXXXXX

XX =                       [DR/ST/QA/UT/PR]

XXXXX =               [CMART/CAPLO/BASEL/IBFTP/……]

XXXXXXXXXXXXXXXXXXXX =  JOB Name

34

AutoSys vs CronJobI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y

max_run_alarm: <mins>

Specifies the max run time that a job should require to finish normally. This test

can catch an error. Such as the applications stuck in a loop.

n_retrys: 5

To set the job to be automatically restarted up to five times after an application

Failure.

Priority: 1

Set the job priority for run.

run_calender: <calendar-name>

Running job on the last business day of the every month.

Run_window: <time-time>

Before starting the job autosys will check the window time.

watch_file: <path>

Attributes used as Watch file min file size and watch interval

35

Q&AI N

V E

S T

M E

N T

B

A N

K

T E

C H

N O

L O

G Y