Application Engine ETL

Application Engine ETL – Direct access to external databases from PeopleSoft made easy

Session #08714Ocotber 6, 2011

Your Presenters David Sexton PeopleSoft ver. 5.01 Senior Systems Analyst “Application Engine ETL – Direct access to external databases from

PeopleSoft made easy”, presentation at HEUG Alliance ’11, March, 2011.

“Justifying a Data Governance Initiative” presentation at HEUG Alliance ’11, with Lumen Data, March, 2011.

“Justifying a Data Governance Initiative” presentation at Oracle Open World ’10, with Lumen Data, September, 2010.

“Putting the Service into SOA – Consuming WSDLs in PeopleSoft”, presentation at HEUG Alliance ‘09, Anaheim, March, 2009.

“Web Services for PSQuery”, presentation at Oracle Open World, September, 2008.

“Web Services for PSQuery”, presented at HEUG Alliance ‘08, March, 2008.

“Extending Applications with Java”, presented at HEUG Alliance ‘07, March, 2007.

“Real World Web Services Integration from PeopleSoft”, presented at HEUG Alliance ‘07, March, 2007.

University of Utah

28,000 students 18,000 employees Joining PAC 12 2002 Olympic Opening and

Closing Ceremonies Tier 1 Research Instition Medical School and Hospital 30 minutes from….

+ =

PeopleSoft Timeline Sep 97 : SA Campus Community / Admissions / Recruitment

Jul 98 : FS Purchasing / Accounts Payable / General Ledger

Oct 98 : Human Resources

Apr 99 : Go live HE Upgrade 7.0 to 7.5 / Time & Attendance (KRONOS)

May 99 : HR Base Benefits / Payroll

Apr 00 : HE HR/SA Upgrade 7.5 to 7.6 / SA Financial Aid

May 00 : SA Student Financials

Feb 01 : FS Upgrade 6.0 to 7.02

Oct 02 : HE HR/SA Upgrade 7.6 to 8.0 / Enterprise Portal 8.4

Dec 03 : Upgrade 8.4 to 8.8 Enterprise Portal

July 04 : FS Upgrade 7.02 to 8.4

Mar 06 : HE HR/SA Upgrade 8.9

Feb 10 : FS Upgrade 8.4 to 9.0

Oct 10 : HR/SA Upgrade 9.0

PeopleTools 8.50.12

We have been working with PeopleSoft for awhile

!!!!!! Disclaimer !!!!!! This presentation is based on our

experience This presentation contains techniques

only endorsed by us, and not necessarily by our employers

Hopefully this presentation encourages you to share your experience at next year’s Openworld conference.

If you have questions, please ask!

Overview

What is App Eng ETL ? Why we developed it

Why not SOA/ Web Services Advanced Use Case What is required DEMO Where you can get it

App Eng ETL – what is it

We create an application engine program that leverages the PeopleSoft record object to dynamically create insert statements. These statements are executed against the external database through a  JDBC connection which is instantiated in PeopleCode.

App Eng ETL – what is it

Transform the data in the PeopleSoft record/view

Extract the data from PeopleSoft using the SQL and Record objects

Load the data using a JDBC connection to the external databse

App Eng ETL Why ?

Started MDM/Data governance/ Integration project Nov 2009

Session Justifying a Data Governance Initiative: Get Approval the First Time #29312 Monday Mar 28 4:30 – 5:30

People want Data ! The more the better

We wanted something that was Easy to maintain/reusable Easy to consume

App Eng ETL Why not Web Services?

SOA requires public services, data sets your organization has declared public

https://wiki.doit.wisc.edu/confluence/display/EBOS/Repository+Home

We’re not there yet Each data request must be approved individually

Each integration is point to point

Consuming web services (wsdl) is not a common skill set (on our campus)

People are excited about it, but can they use it today?

Thank You to Jim Phelps at Wisconsin

App Eng ETL Why not Web Services?

Web Services are best suited where the data request is high frequency

And the response data is low bandwidth Good use of web services

App Eng ETL Why ?

• Build an SQR• Creates a file• Get permissions on the destination server• SFTP the File

Destination does

We do

Custom process to load the file into their database

We do

• Create View• Configure – 1 page• Run a process

Destination doesNothing – data shows up in their data base

App Eng ETL Configure

App Eng ETL Run

App Eng ETL Positives

All integration objects are managed/show on compare reports SQL is in a view – provides a layer or abstration, should

not be impacted by upgrades All integrations are easily visible Low cost per additional integration

No additional coding, configuration only Less moving parts

Customers LOVE it !

App Eng ETL what is required ?

1) Server name where external DB is, DB sid

Jeff.Robbins.awesome testschema

2) Username/Password for the external DB

3) /PSHOME/Appserv/Classes directory contains

driver manager (“java.sql.DriverManager”) The MySql driver (“com.mysql.jdbc.Driver”)

App Eng ETL what is required ?

4) Record/View in Peoplesoft containing the data to be integrated, all fields

will be sent

5) A record on the external database with the same structure as the PeopleSoft record/view

field names, field types, field lengths of the source and target table must match EXACTLY Generate the DDL from the PeopleSoft

record/view, send to external database

Demo configure and run

WHAT ABOUT SECURITY?Gee Dave, this sounds great, but is the

transmission of the data secure? Muliple options to secure the tranmission

Oracle Wallet Talk to your Sales Rep Encyrption options at the JDBC level Source and destination must have Oracle Wallet

Other Vendors supply JDBC drivers that have encrytpion capabilities

Customs options we have investigated Custom Java using ports and SSL SSH tunneling with JDBC*

http://www.ibm.com/developerworks/data/library/techarticle/dm-0312lurie/index.html

SSH Tunnel

Key Code PiecesSSH script

# Referenced in UU_DATA_PUSH application engine. Used for # integrating PeopleSoft data to remote systems.  This script # sets up a secure tunnel to a remote host to connect to a # remote database securely.   local_port=$1 remote_host=$2 remote_database_host=$3 remote_database_port=$4   ssh -f -g -L $local_port:$remote_database_host:

$remote_database_port $remote_host sleep 60

Key Code Pieces Call SSH script from PeopleCode

&script_path = &path | "ssh_tunneling.sh"; &local_port = &properties.get(UU_DP_RQST_AET.URL | ".local_port"); &remote_host = &properties.get(UU_DP_RQST_AET.URL | ".remote_host"); &remote_database_host = &properties.get(UU_DP_RQST_AET.URL |

".remote_database_host"); &remote_database_port = &properties.get(UU_DP_RQST_AET.URL |

".remote_database_port"); &shell_command = "/usr/bin/ksh -c """ | &script_path | " " | &local_port | " "

| &remote_host | " " | &remote_database_host | " " | &remote_database_port | """";

&processId = Exec(&shell_command, %Exec_Asynchronous + %FilePath_Absolute);

Questions?

Contacts David Sexton

Senior Systems Analyst

University Information Technology

University of Utah

[email protected]