Using WSO2 ESB with SAP ERP (Retail) by Harsha Senanayake Head of Enterprise Solutions John Keells Holdings PLC
Jan 20, 2015
Using WSO2 ESB with SAP ERP (Retail)
by
Harsha SenanayakeHead of Enterprise Solutions
John Keells Holdings PLC
Agenda• John Keells Group / SAP ERP
• JKG : Enterprise Applications & Technology landscape / Vision
• KeellsSuper Business requirements
• Solutions Evaluated
• SAP Integration technologies
• Solutions Evaluated - Recommendation
• Solution Architecture Implemented
• Message flow and Enterprise Integration Patterns used
• Project team & plan
• Deployment architecture
• Lessons Learnt
• Q&A
John Keells Group
• Founded in 1870; Sri Lanka’s largest capitalized (1.6 billion USD ) company with shares listed internationally, with Global Depository Receipts (GDRs) listed on the Luxembourg Stock Exchange with a AAA+ credit rating
• Diversified into key sectors of the economy - Hospitality, Transportation and Logistics, Consumer Foods, Retail, Financial services, IT, Real estate and Property development and Plantation services
• 10,000+ employees
• Ranked among the “200 Best under a Billion” in Asia Pacific by Forbes
• Ranked the #1 Corporate in Sri Lanka by the Lanka Monthly Digest (LMD) 8 times in the recent past
• Signatory to the United Nations Global Compact
SAP ERP
• SAP (NYSE: SAP) is the market leader in enterprise application software with over 170,000 customers
• The company's best known products are its enterprise resource planning application (SAP ERP), SAP BusinessObjects software, and most recently and Sybase mobile products
• John Keells Holdings implemented SAP across its 70 subsidiaries in 2005 – the largest SAP implementation in Sri Lanka
JKG: Enterprise Applications & Technology
landscape
DW
Group ERP
ES
B
WCM ESS BI dashboards Composite apps Collab
Messaging
Standard contents
Search
Por
lets
(IT
S,
BS
P, W
eb
dyn
rpo
)N
ativ
e R
FC
(Ja
va E
SS
ap
ps)
cal
ls
DirectoryServices
Infrastructure – Servers, Storage, Network, Data Center services
CPM
ABAP - RFC data sourcesSAP Query
Industry specific LOB solutions
Infrastructure
ECM
Solution Lifecycle Management
MDM
Identity Management
& SSO
Business logic layer : ABAP, J2EE , .NET
Customer/Partners
EDI /ExtranetPortal
EDI / Web-services / WSRP
BE
x /
OD
BO
(O
LE D
B f
or
OL
AP
) M
DX
Fle
x /
IGS
/ X
ML
A (
MD
X)
SO
A g
over
nanc
e
Presentation layer :Enterprise Portal
Sync/Async Web ServicesXML/ IDOCS
Por
tlets
SAP GUI Rich /Web clients
Integration vision
[Gartner 2004 ] “An Enterprise Service Bus (ESB) is a new architecture that exploits
Web services, Message-oriented-middleware (MOM) Intelligent routing and transformation.
ESBs act as a light weight, ubiquitous integration backbone through which software services and application components flow”
Peace of mind : Once data is reliably delivered to the ESB, it is considered "safe" and out of the hands of the client
Enterprise service bus By David A. Chappell
Some of the Integration scenarios @ JKH
• Async Web services postings via SAP PI (EAI)
• AR/AP/GL DFinance_Document_POST -> XI IDOC->R/3
• Status retrieval by Polling with message GUID
• Sync web-service interfaces via SAP PI (EAI)
• Credit limit check WS -> BAPI -> R/3
• Banking Interfaces
• Custom ABAP programs
• Payment files – MT100, Encrypted propriety forms upload to bank sites
• Bank reconciliation - SWIFT MT940/ MultiCash
• Direct posting / reading of data via RPCs and WebAS Webservices
• RFC/BAPIs using .Net / DCOM / Jco connectors
• Companies – JMSL-POS, LMS-Sales app, BPOMate
• Direct upload of CSV files via FTP to SAP R/3 – AR/AP/GL Upload program ZF247
• ABAP BDC uploads
• Hotels
• PMS integration with Call accounting systems/ PBX, Door locking system, Minibar systems, In room internet systems, Voice
• CRS integration with GDS and OTAs
• Retail
• POS integration
• 3rd party settlement integration
• Transport and logistics
• EDI – CUSDEC
• SAP R/3 and RedPrairie WMS integration (XML<-> IDOC)
• Stock brokers
• Direct Market access via FIX
Group – SAP related Industry specific
KeellsSuper
• KeellsSuper is one of the leading supermarket chains in the country which has been in operation for the last 20 years
• KeellsSuper was Instrumental in popularising Modern trade in Sri Lanka
• Currently at 42 outlets and expanding
First Retailer to implement a leading ERP in Sri Lanka
Introduced the first Loyalty program in Retail with Integrated CRM and BI in Sri Lanka
Only online supermarket in Sri Lanka -KeellsSuper.com
Real-time stock taking using handheld scanners
Business requirement
• KeellsSuper has been running SAP ERP for 5 years and decided to migrate to SAP’s ERP Retail industry solution to benefit from the rich industry specific specific functionally available in SAP Retail ERP solution
Business Requirement
• Streamline the SAP and POS integration - Eliminate the performance issues and failures faced with the current SAP and POS integration
• Seamless integration with the online store – keellssuper.com.com
Business Requirements Required Integration scenarios
Direction Description
Article masters - new and changes
Condition masters - price changes
Vendor masters - new and changes
Customer master (credit customers)
Bonus Buy Conditions / Promotions
Download phys. inv. docs, upload phys. inv. count data
Credit l imits
Stock balances
Gift vouchers master data
Application acknowledgements
Financial transcations
Upload day-end closing POS
Upload sales data (compressed)
Inventory counts
Gift vouchers - issues
Inbound(POS - > Head offi ce)
Outbound(Head offi ce -> POS)
Business Requirements
• Guaranteed delivery - message queuing and guaranteed delivery semantics to ensure "unavailable" applications will get their data queued and delivered at a later time
• Unreliable and slow networks (specially in outstations), POS Server outages
• Quality of service (QOS) requirements
• Exactly Once In Order – E.g. Price changes
• Exactly Once – E.g. Sales data
• Prioritization of Messages
• Price changes Vs Article description change
• Better performance without impacting the ERP system
Solutions evaluated
1. Adopt the existing synchronous RFC based Point-to-Point integration solution developed using SAP DCOM / .Net Connector / Web services
2. Develop a Point-to-point integration solution based on SAP WS-RM enabled asynchronous Web services
3. Develop an IDOC based Point-to-point integration solution using SAP .Net connector
4. Implement an IDOC based ESB integration solution using SAP Netweaver PI
5. Implement an IDOC based ESB integration solution using WSO2 ESB
SAP Integration Technologies
RDBMS
NCo(.net)
RFC
BAPI
IDOC
SAP ERPBusiness logic
(ABAP)
Application
JCo(Java)
IDOC EDIIDOC XML
RPC
SOAP / WS-RM
SAP Application Server
JDBC/ODBC/ADO
SOAP Processing
Synchronous Vs Asynchronous services
• With synchronous services , clients invoke a request on a service and then suspend their processing while they wait for a response from the provider. The client cannot perform another task until a response or confirmation is received from the provider.
• With asynchronous services , clients initiate a request to a service and then resume their processing without waiting for a response. The service handles the client request in the provider system and returns a response at some later point, at which time the client can retrieve the response and proceed with its processing.
SAP Integration Technologies RFC/BAPIs
• An RFC is a synchronous interface method that calls and executes predefined functions
• A BAPI is a a RFC-enabled function that has been developed by SAP E.g :
• BAPI_PO_CREATE
• BAPI_PO_GETDETAIL
• Synchronous RFC/BAPI calls consume SAP ERP dialog processes and slow network connections can really hog ERP system resources
SAP Integration Technologies Web services
• The ABAP Web Application Server (WebAS 6.20+) can provide existing functions (BAPIs, RFCs) as Web services
• Since SAP NetWeaver 7.0, SP 14, Web Services Reliable Messaging (WS-RM) has been introduced as part of the ABAP stack. WS-RM provides reliable delivery of asynchronous messages using the SOAP protocol
• SAP was an early adopter of SOA for its business applications
• http://esworkplace.sap.com : The ES Workplace is the central place to view consolidated information about all available Enterprise Services delivered by SAP.
SAP Integration TechnologiesIDOCs
• SAP Intermediate Documents (IDOCS) are EDI like documents that are asynchronous in nature.
• The actual TRFC call to submit the IDOC to SAP is performed synchronously and very quickly, but the actual business processing can happen at some later time defined in the SAP system
• Schedule Report RBDAPP01 as a Background Job to process the IDocs (see SAP Note 399271).
• The outbound result (E.g. sales order confirmation) can also happen at some later time – Message type ALE AUDIT
• IDOCS offer additional queuing and retry capabilities
• IDOCs can be triggered using posting routines and Change pointers
• E.g At the time of Creating Purchase Order using ME21N
IDOCs available for the required integration scenarios Direction Message Type Interface description
ARTMAS Article masters - new and changes
COND_A Condition masters - price changes
CREMAS Vendor masters - new and changes
DEBMAS Customer master (credit customers)
WPDBBY Bonus Buy Conditions / Promotions
WVINVE Download phys. inv. docs, upload phys. inv. count data
ZFI_CRED Credit l imits
ZMM_STOCK Stock balances
ZSDGV Gift vouchers master data
ALEAUD Application acknowledgements
WPUFIB Financial transcations
WPUTAB Upload day-end closing POS
WPUUMS Upload sales data (compressed)
WVINVE Inventory counts
ZSDGVRE Gift vouchers - issues
Inbound(POS - > Head offi ce)
Outbound(Head offi ce -> POS)
Receiving and Sending IDOCs using .NET and Java
1.package com.sap.conn.idoc.examples;3.import com.sap.conn.jco.*;4.import com.sap.conn.idoc.jco.*;5.import com.sap.conn.idoc.*;6.import java.io.*;8.public class IDocClientExample {
10.public static void main(String[] args) {12.try38.// see configuration file BCE.jcoDestination provided in the installation directory.39.JCoDestination destination=JCoDestinationManager.getDestination("BCE"); 40.IDocRepository iDocRepository = JCoIDoc.getIDocRepository(destination);41.String tid = destination.createTID();42.IDocFactory iDocFactory = JCoIDoc.getIDocFactory();44.// a) create new idoc45.IDocDocument doc = iDocFactory.createIDocDocument(iDocRepository, "MATMAS02");46.IDocSegment segment = doc.getRootSegment();47.segment = segment.addChild("E1MARAM");48.// and so on. See IDoc Specification .....49.JCoIDoc.send(doc, IDocFactory.IDOC_VERSION_DEFAULT, destination, tid);56.destination.confirmTID(tid);58.}59.catch(Exception e)60.{61.e.printStackTrace();62.}63.System.out.print("program end");64.}65.}
Java IDOC Client (Sender)
.NET IDOC Server (Receiver)
Why IDOCs based integration was better suited for our requirement
• Synchronous BAPIs/RFC/WS calls are resource intensive
• All required WS-RM asynchronous outbound services are not available
• Tried and tested technology used by almost all SAP ERP Retail customers
• Only the changed data can be transferred from SAP
• ‘POS Interface Monitor’ functionality available to monitor IDOCs
Solutions evaluated – Recommendation
1. Adopt the existing RFC based Point-to-Point integration solution developed using SAP DCOM / .Net Connector / Web services
2. Develop a Point-to-point integration solution based on WS-RM enabled asynchronous Web services
3. Develop an IDOC based Point-to-point integration solution using SAP .Net connector
4. Implement an IDOC based ESB integration solution using SAP Netweaver PI
5. Implement an IDOC based ESB integration solution using WSO2 ESB
Synchronous RFC/BAPI/Web service calls consume SAP ERP dialog processes and slow network connections can really hog resources
Point-to-point spaghetti!
All required WS-RM asynchronous outbound services were not available Not tried and tested by SAP Retail customers, Point-to-point spaghetti!
Made sense but writing code for IDOC parsing , transformations and handling reliable messaging would have been LOT of effort!
And why reinvent the wheel? Separately licensed - SAP provides POS integration content out of
the box Dual stack (Java+ABAP) architecture makes PI very complex
and heavy - Java only ESB was introduced with NW 7.3 Light weight, Open source, Met our key requirements, Local
support, SAP adapter was being developed, Joint go-to-market strategy
WSO2 team claimed Ebay processes 1 billion transactions a day…
Implemented Solution architecture
Asanka Abeysinghe
SAPReceiver
ESB POSSender
SQL
ç (1) Query Sales line-items by Polling• Update on message successfully
accepted by the ESB - (Update TRANFFERED_TO_ESB =“TRUE”)
ç (3) Post IDOC ‘WPUUMS01’
è (4) Technical ack - IDOC created successfully /failedmessages if reprocessed in SAP.
(2) Transform/Map to IDOC structure- WPUUMS01 has limit on max line items
per IDOC therefore will need to be split into multiple IDOCs
è (7) Update on Application ack - UPDATE record:
• ALEAUD message status and update field TRANSFFERED_SAP _APPLICATION_ACK = “TRUE” and update field TRANSFFERED
è (5) Update on Technical ack - UPDATE record :
• TRANFFERED_TO_SAP_TEC_ACK=“TRUE”
Polling ConsumerSpilter Message translator
(XSLT)
è (6) Application ack - (ALEAUD)(status, message) – Multiple ALEAUD messages if reprocessed in SAP.
Store and Forward
Content-Based Router
Content-Based Router
Message flow and Enterprise Integration Patterns usedExample Scenario : POS Sales Data interface to SAP
Notations from the book Enterprise Integration Patterns - Gregor Hohpe
Project team
Head of PMO
SAP Functional
team
SD-POS Consultant
FICO Consultant
MM Consultant
SAP Technical
Team
ABAP Consultant with IDOC
skills
POS Team
.NET and SQL Developers
WSO2 Team
Architect
Project Manager
ESB Consultants
DS Consultants
QA Team
Core Team
Process leads
Admin / Infra Team
Project Plan
Project PlanJun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11
Project Kick off 1st JuneERP
Blueprint Document POS integration scenariosFinalise IDOCs and fields to required
RealisationConfigure standard IDOCsEnhance/develop IDOCs
WMSBlueprint Realisation
POS DevelopmentPOS Integration
Solution architectureWSO2 SAP Adapter developmentSizing and Deployment architectureIntegration scenarios / use casesField mapping and specificationsTrainingDevelop Transformation scriptsQA / Stress test
GoLive and Post Golive phaseGolive preparationGoLive 29th FebPost Golive support
Deployment architecture - Sizing
• 500k transactions on an average month
• 1 million on seasonal months
• Avg Size of a message - 100kb – 5mb
Direction Message Type # of IDOCs on May 2011
ALEAUD 4,569
ARTMAS 75,189
COND_A 325,517
CREMAS 214
DEBMAS 19
WPDBBY 646
WVINVE 4,110
ZFI_CRED 301
ZMM_STOCK 11,592
ZSDGV 491
WPUFIB 5,058
WPUTAB 1,339
WPUUMS 8,565
WVINVE 33,866
ZSDGVRE 4,140
Outbound(Head offi ce -> POS)
Inbound(POS - > Head offi ce)
Deployment architecture
Production Server Configuration DR
Server 1:-CPU:- 4 Cores
RAM:- 8 GB
Storage :- 50 GB
Server 2: CPU:- 2 Cores
RAM:- 4 GB
Storage :- 140 GB
Development Server Configuration
CPU:- 4 Cores
RAM:- 8 GB
Storage :- 105 GB
Lessons learnt• Avoid this:
• Always revalidate requirements to avoid over engineering – E.g. Do you really need EOIO for all integration scenarios?
Lessons learnt
• What worked in the Lab with sophisticated load test tools can break in the real environment
• When you have multiple components (SAP ERP, POS, 50+ end points over unreliable networks) it’s difficult to simulate a real production environment
• Strike a balance between offshore and onsite model
• Have frequent project / steering committee meeting to ensure EVERYONE is on the same page to avoid surprises
Q&A