June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 1 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA ODTUG Kaleidoscope 2009 Lucas Jellema SOA
May 06, 2015
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 1
The Story of How an Oracle Classic Stronghold Successfully
Embraced SOA
ODTUG Kaleidoscope 2009Lucas Jellema
SOA
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 2
Overview• What is an Oracle stronghold?• Triggers to start moving towards Services• Levels of embracing Services and SOA
– Objectives, benefits, costs & challenges– Demonstration
• Pitfalls, Lessons Learned & Best Practices• Summary
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 3
What is an Oracle stronghold?
• The typical Oracle stronghold– Using Oracle RDBMS & Oracle Development tools
• Lot of SQL and PL/SQL• Probably Oracle Forms and maybe APEX as well• Possibly Oracle Designer, tools for BI & Reporting
– Several databases with many years of essential corporate data
– IT staff has Oracle veterans – 5-15 years or more– Internet development may have taken place
largely separate from the Oracle technology stack
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 4
My central case…• Oracle Forms application tightly integrated
with an Oracle RDBMS– The database is known by the name of the app– Containing jobs, workers, timesheets, payments,..– Used in hundreds of branches as well as the
central (back)office• Main business driver that required attention
– Business partners requested a programmatic interface to load multiple timesheets – to save on time and hassle (time again)
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 5
Similar cases …• Oracle Forms application for assigning homes
offered SaaS-style– SaaS Customers want the details on houses and
their availability published on a website– .NET applications need access to data in the
Oracle Database– Customers want their local applications to
interface (programmatically) with the SaaS application
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 6
Then wat happened… (2)• Car Lease company has various custom
applications and databases per department– Business requires IT to support processes that go
across those applications and databases• through a single, unified User Interface• involving a legacy database and a 3rd party ERP system• with eventually some self service web modules
• Insurance company sells policies through agents using a Forms application– New direct channel: On-line policy selling
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 7
Then wat happened… (3)• Agricultural company supports ‘cow
insemination’ process with Forms application– Farmers and inspectors need to be able to record
data anytime and anywhere through PDAs (that run a .Net application)
• Mid-sized chemical pharmaceutical company uses BoB stand-alone systems and databases– To allow for faster (near real-time) responses to
customer demands and logistical challenges, tighter integration between the systems is needed
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 8
Common Characteristics• Cross Boundaries
– Cross Technology - .Net, Uniface, Java, Tibco– Cross Channel – Back Office, Web, PDA, API– Cross User Group – Internal, Agents, Self Service– Cross Domain – Multiple departments & systems– Cross Enterprise – Interact with external partners
• Data synchronization or consolidation• Management whim Vision• From Database angle: providing services
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 9
Objectives
• Business Agility– (Faster) responses to changing demands
• Or at least an urgent, currently pending demand
– Creating new business from existing resources• IT Flexibility
– Optimize locally without impact ‘globally’– Prepare for future developments
• Lower costs– Through reuse, better integration, decoupling
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 10
Objectives
• (longer term) Higher Quality and Faster Process execution– Automated data exchange cross boundaries– Workflow and task orientation
• integrated, cross department, in a controlled way based on sound understanding of the business processes
– Business Event driven interaction• Manage risks and fear
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 11
SOA = BAD
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 12
SOA =BusinessAgility through
Decoupling
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 13
Decoupling≈
Managing Dependencies
minimize impact of change while maximizing reusability
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 14
Types of decoupling• Functional
– Interface - Encapsulation of implementation• Design by Contract, Implement by Design
• Technical– No proprietary technology, protocol, message format– Standards based (XML, HTTP, RSS, WSDL…)
• Temporal– Asynchronous communication (separate response)
• Development– Separate teams working in parallel based on mutually
agreed interface definitions
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 15
Decoupling Applications & Data
Data
Application(User Interface)
Application(User Interface)
Data
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 16
Decoupling Applications & Data
Application(User Interface)
Data
Application(User Interface)
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 17
Decoupling Applications & Data
(User Interface)
Data
Application
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 18
Decoupling Applications & Data(User Interface) Application
WorkflowEngine
EmailIM
Fax
CMS
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 19
Data Ownership• Data no longer exclusively owned by a single
application• Data (query and manipulation) available via APIs,
(web)services and open standards– For example based on XML, XSD, WSDL, SOAP, HTTP
• Data Hubs are formalized, structured approach where data is completely separated from applications– All access is through services– No data duplication at all in the enterprise– Data ownership is separate process
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 20
Decoupling from Table to ESB+
http
WEBDAVFTP
http
WS/SOAP
WS*
WS*
WS*
WSRP
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 21
Tables in Database• SQL for retrieval and manipulation• Data Model in plain “view”• Decoupling between DML and Retrieval• Retrieve data:
– select e.empno, e.ename, d.dname from emp join dept using (deptno)
• Manipulate data:– insert into dept;insert into emp;
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 22
View in Database• Hide (encapsulate) Data Model
– Manage access privileges• SQL for retrieval and manipulation• Instead Of trigger decouples DML operations• select id, name, department from emp_vw
• Insert into emp_vw• Use case: new UI on top of ‘legacy’ data model
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 23
Package in Database• Hide (encapsulate) SQL• Procedure calls for retrieval and manipulation
– Potentially complex data structures using Object Types and (nested) Cursors
• HRM_MGR.get_emp( id) return emp_t• HRM_MGR.create_emp( emp_t);
• Use case: tailor made business services to support clients that understand types– Or core service implementation around which a
non-type wrapper is applied
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 24
Database Object Types
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 25
Package in Database (2)• Hide (encapsulate) SQL and Oracle
and user defined Types • Procedure calls for retrieval and manipulation
– Input and output parameters standard types only (string and number)
– Complex datastructures: XML passed as string• HRM_MGR.get_emp( id) return string• HRM_MGR.create_emp( string);• Use case: packaged business services to
support any client (that can access the DB)
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 26
Handling XML• Structured, multi-level data in a string: XML• Oracle Database has XMLType
– Can be created from a String, will parse XML• Can validate against a schema definition (the XML data
design)
– Support XPath operations to retrieve specific bits and pieces from the XML document
– Can do XSLT transformations of the incoming or outgoing XML
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 27
XML based PL/SQL interface
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 28
AQ for Asynchronous• Decouple consumer and provider
in time – asynchronuous processing• Consumer is registered on the AQ
– Usually a package that processes User Defined Type that is sent as payload in the AQ message
• Use case: asynchronously processed one-way (fire-n-forget) requests– Potentially lenghty requests– High volume of requests
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 29
Hiding the database
http
WEBDAVFTP
http
WS/SOAP
WS*
WS*
WS*
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 30
Publish package as http-based API using dbms_epg
• Hide database protocol– Not its physical location nor the schema, and user
authentication• HTTP communication is truly cross technology
– Browser, Java, .Net, JavaScript & RIA clients, …– Approximation of RESTful services (very du jour)
• Can publish in various formats– Text, HTML, CSV, JSON, XML, RSS
• Use case:cross-technology, internal no WS*/ESB
http
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 31
HTTP API – directly on top of the RDBMS
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 32
Publish static resources on various protocols with XMLDB
• Hide database protocol– support FTP, WEBDAV, HTTP(S)
• Run scheduled batch jobs (PL/SQL) to periodically create & expose resources– Can also consume and process resources
• Use case: cross technology need for retrieving slowly changing resources (CSV, XML)– Possibly uploading resources for batch processing
• Use case:
WEBDAV
FTP
http
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 33
Publishing RDBMS resources in a decoupled fashion
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 34
SOAP based WebServices
http
WEBDAVFTP
http
WS/SOAP
WS*
WS*
WS*
WSRP
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 35
SOAP WebServices• All messages (input and output) are XML• The message consists of two parts inside an
envelope (a SOAP XML wrapper)– The header with meta-data– The body with the contents to be handled by or
returned by the service• The WebService Definition Language (WSDL)
document describes the service• An XML Schema Document (XSD) describes
the structure of the XML messages– XSD is like an ERD or Table Design
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 36
Oracle RDBMS 11g - Native Database WebServices
• Schema can be published throughnative database web services– Each package corresponds with a WSDL– Every program unit with an operation– WSDL and XSD are dynamically generated
WS/SOAP
– http and https is supported– Limited control over WSDL & XSD
– Use case: internal, cross technology, WS enabled client, no ESB or Application Server available
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 37
WebService in App Server based on PL/SQL package
• Hide database– Protocol, location, authentication:
everything handled by the application server• Use JPublisher (embedded in JDeveloper) to
publish a PL/SQL package as WebService– JPublisher creates JAX-WS-annotated class, utility
classes and possibly helper types in the database• Alternatively: create ADF BC Application
Module & publish it as a WebService (Bulldog)– With support for SDO (Service Data Objects)
• Use case: WS (SDO!) enabled client, no ESB
WS*
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 38
Enterprise Service Bus
Data
Service Service Service
App 2
Data
Service
External Partner
Service
App 1
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 39
The Enterprise Service Bus• Should first of all be considered ‘a pattern’• Virtualizes services – hides the real service
from consumers– Deals with the physical location of the services
• Adapts synchronous to a-synchronous and vv.• Can use multiple real services to offer one
virtual (composite) service• Allows callers to use a generic, canonical
message structure that it will transform to the service contract– It may even allow callers to use their own “lingo”
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 40
Canonical Data Model• Common Business Language
– Esperanto for service invokers
DataData
ServiceService
ClientsCustomers
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 41
The Enterprise Service Bus (2)• Handles various QoS & SLA aspects
– sometimes in concert with tools like OWSM– Encryption, signing, authentication– Retry and fallback– “Throttle” (prevent peak loads)
• Does monitoring, tracking & auditing, reporting, notification and escalation
• Works with Adapters to access technologies– like RDBMS (SQL, PL/SQL), AQ and JMS, File
System, FTP, Java, E-Business Suite
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 42
Publish PL/SQL Package through ESB
• Use Database Adapter to createService, combine with ESB Routing Service
• Use case: – external access to services– virtualize location of service –
• route to service based on content of the request– virtualize part of contract of service
• Package-derived XSD not suitable for consumers– handle peak loads– monitor service levels and trace service access
WS*
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 43
Enterprise Service Busin action
• One team responsible for exposing services to external consumers– Working with those customers for establishing the
contract and testing across the firewall– This team built from internally provided services
• Team two worked inside database – providing package based API for granular services
• Team three created ESB level database adapter and routing/mapping services
WS*
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 44
Evaluation – end of phase 1• Done well
– Working “application” in production!– Worked together (across teams and departments)
on the Service definition and the ‘XSD’• To determine the scope and granularity of services and
operations for optimal reuse• To define
– Management sponsorship (though IT mainly)• To be improved
– More involvement from the business for defining both services and especially the canonical model
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 45
Evaluation – end of phase 1• Done well
– Achieved good way of working with database team developing packages underneath services
– Developed a feel for using services rather than immediate database access
– Built up XML and ESB skills• To be improved
– Involvement of administrators– Setting up Dev-Test-Acceptance-Production
• Managing Service End Points
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 46
Evaluation – end of phase 1• Done well
– Ambitions, drive - However: technology driven• To be improved
– Define the Enterprise Architecture/BluePrint and work within its context
• “Think globally, act locally”• Set up CEA – mandated by business and IT
– Reduce number of service calls • Externally exposed services do not need to be 1:1 with
database packages!• Granularity and Composite Services
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 47
Evaluation – end of phase 1• To be improved
– Share success with business and other IT teams– Establish a mechanism for sharing, exposing,
finding, adopting SOA artefacts• Services, Canonical Data Model, Policies/SLAs, …
– And also for ‘governing’ the life cycle: how to change services already being used?
– (automated) Testing– Monitoring service traffic (some focus on SLA)
• Peak load, response time/overhead, availability
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 48
End of Phase 1• Working service implementation and
infrastructure – across departments– End-to-end from backoffice database to customer
• “Foundation for innovation”– Lost the fears, ready for the next step– Basis for reuse and widening the scope– Shimmering light at the end of the tunnel for the
Forms application
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 49
Improve & Broaden the scope of the Canonical Data Model
• Involve business representatives for all of business• Add business terminology and (some) business rules
that help describe the model– Build up translations of domain values and business object
identies across ‘systems’
• More accessible and consistent structure– using namespaces (domains), naming conventions and
guidelines (element vs. attribute, data types)– Annotate the model and provide examples
• Eridicate database model legacy or technology specific elements
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 50
Pending “Research”• How to make use of cache infrastructure to
prevent services being called unnecessarily– Yet guaranteeing non-stale data
• How do we process binary attachments through the layers of our architecture
• Do we have some low-hanging-fruit to pick in order to gain some additional benefits– Design patterns, best practices, short-cuts, …
• Security and Policies – what levels, mechanism
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 51
Phase 2• A new user interface (internet website) on top
of the back office database– Partially covered by existing services and an
existing Web Application and CMS• Business is also looking for Web 2.0,
Social/Community-style interaction in Portal– Running process flows that potentially go from
external workforce through branch to (multiple entities within the) back office
• How to move from peak load batch processing
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 52
Introducing BPEL for Service• BPEL adds to service
– Long running (stateful) ‘service instances’– Composite services that include
• Multiple service calls (including asynchronous)• Exception handling including retry and compensation• Human Task for manual steps & integration Rule Engine• Process flow logic
• Use case: – data request must be fulfilled by various services;– DML impacts several systems and/or requires
human approval
WS*
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 53
Publish Business Events• Extremely Decoupled Architecture • Any system – including database – reports
events that may be interesting to other parties• The Event backbone (could be the ESB)
– Defines Event Types (name, structure of payload)– Registers Event Listeners (“please call me when
the event occurs and send the details”)– Receives events – instances of the predefined
event type with payload and timestamp• Propagate events to all registered listeners• Without blocking the event producer
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 54
Event Driven Architecture (EDA)
Data
Service Service Service
App 2
Data
Service
External Partner
Service
App 1
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 55
Database publishing events• Table Trigger intercepts DML
– Checks for Business Events such as new employee– Sends them to package EVENT_PRODUCER
• Package EVENT_PRODUCER sends events– Via UTL_HTTP to a WebService– Via AQ to a listener (ESB AQ Adapter)
WS*
EventProducerEMP
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 56
Publishing Service with UI• Instead of only publishing a
programmatic service interface– A service can be published with a User Interface;
the service-with-UI is called: Portlet• The standard approach:
– Portlet Container in Application Server exposes WSRP services for the portlets
– The Portlet produces (X)HTML and handles HTTP requests
– A Portal consumes the WSRP Portlets in a web page
WSRP
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 57
Decoupling from Table to ESB+
http
WEBDAVFTP
http
WS/SOAP
WS*
WS*
WS*
WSRP
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 58
Increasingly decoupled• More hiding of the implementation• More Formal Interface Contract• Less (proprietary) technology & more
standards for interacting• Less exposure of (legacy) data model• More support for asynchronous interaction• More reuse potential• Pervasive throughout enterprise• More suitable for external consumption
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 59
Comes at a cost…• More run time overhead
– Additional tiers– XML serialization and deserialization
• More infrastructure– Burden of Administration– License Costs– Hardware
• Broader skills palette – more stuff to master• Harder to get started
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 60
Pitfalls• Do SOA from the IT/technology side only• Inconsistent, illegible, unstructured
namespaces and (XML) canonical model– Having the database (table and column names)
shine through in the canonical data model – And forgetting the database design wisdom
• Reusable SOA artefacts are not found, understood nor trusted
• Lack of Balance between reusability and usefulness (Fine grained vs. coarse grained)
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 61
Pitfalls• The greedy clutches of enterprise architects
– Think globally, talk, (high level) design, draw & write, present, think, talk, …. (no real action)
• … versus the technology driven, ’think and act locally’ approach from the ‘developer squad
• Introducing new unmanaged dependencies– Hard coded endpoints (service URLs)– Calling external services without proper SLA or
fallback option– Using complex technology without proper skills
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 62
Pitfalls• No DTAP process & environment
– And/or completely manual• Involving DBAs/Administrators way too late• Inappropriate use of the SOA infrastructure
– Web applications retrieving each individual record (or even field) through a separate service call
• Running Forms on top of the ESB!– Sending debug and trace messages via the ESB– “Package calls other package via ESB”
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 63
Useful • Use Mock Service implementations during
development and test– Developers that depend on services not yet
available easily get stuck• Automated Functional and Performance Test
of individual Services• Automatic Service ‘ping’ utility
– Early detection of service unavailability
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 64
Summary• Objective: agility through decoupling
– Managing dependencies– Crossing boundaries – functional, technology, time
• Just do it! (well, “think big, do (small)”)– Get started – at the right level for your situation
• Do not go off and buy BPEL just like that
– Even though it won’t be perfect the first time round – you will learn (only) through experience
• Do it explicitly, visibly and with all involved
SOA
June 2009 The Story of How an Oracle Classic Stronghold Successfully Embraced SOA 65
Q &A SOA