SOA Data Integration - The Unsolved, Unspoken Problem.
Post on 18-Dec-2015
216 Views
Preview:
Transcript
SOA Data Integration - The Unsolved, Unspoken Problem
David Webber
SOA Architect
Chair OASIS CAM TC
November, 2007
drrwebber@acm.org
Metatopia Conference 2007 - Washington DC 3
Agenda
SOA information integration needs
Selection of illustrative field examples
Solution and technology approach
Demonstration
Summary / Next Steps
Metatopia Conference 2007 - Washington DC 4
Agenda
SOA information integration needs
Selection of illustrative field examples
Solution and technology approach
Demonstration
Summary / Next Steps
Metatopia Conference 2007 - Washington DC 5
CEO - Enterprise Strategic Vision
Improve Program Integrity
Improve Products &
Services
Reduce Costs
ImproveHumanCapital
Management
Modernize& IntegrateSystems
- Reconcile current business demands, economics and future business strategies with present state of legacy systems, technology and human resource skills
-Strengthen financial management and internal controls
-Provide effective oversight
-Ensure the accuracy of all data
-Implement an integrated set of information systems
- Enable cost reduction and re-engineered business processes
- Establish metrics to measure performance and productivity trends across the enterprise over time
-Ensure enterprise has the right people in the right jobs with the right skill sets
-Create and foster an environment for collaborative problem solving and decision making
-Listen to the customer to understand and prioritize needs and/or desired products and solutions
-Inform business solutions by incorporating industry best practices
Metatopia Conference 2007 - Washington DC 6
Some SOA claims and justifications
Service Oriented Architecture means:
• Reusable components exposed through
services shared with business partners
and enterprise systems enabling low-
cost, low-impact incremental and organic
adaptation
Metatopia Conference 2007 - Washington DC 7
SOA Characteristics
7
ModularApproach
Questions?
SupportsStrategicInitiativesand Goals
SwappableAgile
Components
Open Standards
- SOA implements the functionality of the system using formal open standards for information exchange, transport and delivery.
- SOA is an attempt to modularize large complex systems in such a way that they are composed of independent software components that offer services to one another through well-defined interfaces.
- This supports the notion that any of the components could be ‘swapped’ for a better one, when it becomes available.
- Reconcile current business demands, economics and future business strategies with present state of legacy systems, technology and human resource skills
- How can we swap things if the information interfaces are not clearly defined and easily verifiable?
- How can we leverage inconsistent legacy interfaces?
Metatopia Conference 2007 - Washington DC 8
What are SOA Services?
• The ability to orchestrate service logic with business processes
• Not just static point information service sinks or sources• Ability to associate behavior with information exchanges
– Support for context and role– Support for business process actions and signals– Support for access profiles and partner agreements– Support for dynamic structure queries and responses
• Use of open public standards to ensure predictable access to services
• Independent of platform deployment and legacy systems constraints
Metatopia Conference 2007 - Washington DC 9
What makes something Agile?
• Re-usable methods that can be applied to many areas– Based on open standards and approach; not proprietary
• Context and role driven and aware– This allows tailoring to specific profiles and use pattern templates dynamically
• Self-adaptive– When requirements change can be adjusted on-the-fly in real time
• Fault tolerant and not brittle– Ability to ignore non-critical interchange items and especially not to fail for trivial
reasons or slight version nuances
– Leveraging XML capabilities to make self-describing transactions possible rather
than static fixed legacy exchanges
• Able to support new uses without extensive re-programming– Usage patterns set via external configuration allowing broad but controlled uses
Metatopia Conference 2007 - Washington DC 10
How to Support Discovery?
• As an organization develops extended numbers of services the need to catalogue and share these collaborative increases.
• Services need to:– Describe their means of connection and interfacing– Provide a reference to the domain of use– Be associated with a catalogue and classification system
• The most powerful discovery mechanism is not simple text search but context and profile agreement parameters linked to templates
• Associate with a domain vocabulary to facilitate shared meaning and core components
• Align to interoperability of information exchanges and agile concepts• Bottom line – answer the questions:
– Can I use this? (context)– How can I set it up? (templates) – Will it work & connect with my existing services
(behaviors)?
Metatopia Conference 2007 - Washington DC 11
Defining Information Services
Create Samples
RulesEditor
XML
AnalystTemplates
Samples
Develop
Verify
StructureRulesContextVocabulary
3
2
Rules
html
XML
Resultshtml
Publish
1
Detail Use Rules
Test Rules on Samples
4
Share Results
EnablingAgile Information Exchanges
Structure + Vocabulary
Rules + Context Templates
Verify Template Outcomes
Metatopia Conference 2007 - Washington DC 12
Partner Conformance
Test andCertification
Validate
Templates
Report
PartnerCreatesXMLResults
Test
Pass / Fail
Templates
XMLhtml
StructureRulesContext
4
5
6
Publish
Metatopia Conference 2007 - Washington DC 13
Facilitation Needs
• For business communities:– Ability to create sharable templates for communities
of practice that need consistent XML transaction handling definitions that are open and public.
• For business data analysts:– Printable rule documentation support and features. – Example domain templates within communities of
practice – Genericode codelists implementation.
• For programmers:– XML content manipulation support and rules ( xslt,
XPath …)– Web services and SOA support
Metatopia Conference 2007 - Washington DC 14
Select your technology set
• Many technologies are available, including application servers, distributed objects, and integration servers. Agile use of XML a prerequisite.
• The choice of technology will likely be a mix of components that together meet your SOA needs.
– Avoid vendor lock-in
– Prefer Open Standards and Open Source solutions
– Ensure proper use of layers to separate and manage complexity
– Select simple components that integrate easily
Metatopia Conference 2007 - Washington DC 15
Basic WSDL Web Service
TransportTransport
SecuritySecurity
Internet Infrastructure
DeliveryDelivery
Data Services
Data ServicesProc
ess
Proc
ess 33
22
4455
Requirements
UML diagrams
Application Logic
Packaging / Encoding
XML
Validation
XSD
Query / Response
WSDL
11
SOAP / http / https
Metatopia Conference 2007 - Washington DC 16
Extended Data Services
TransportTransport
SecuritySecurity
Internet Infrastructure
DeliveryDelivery
Proc
ess
Proc
ess 33
22
55
Query / Response
WSDL
11
SOAP / http / https
Data Services
Data Services
44
Vocabulary / Semantics
Packaging / Encoding
XML / edi
Validation / Assembly
Mapping / Transform
XSD, CAM, Schematron
Business Model
ServiceAgreement
Context/ Roles
BusinessProcesses
Description
Msg ExchangeProfile
State & Context
Addressing / Envelope
Transactions
Coordination
Push / Pull
Metatopia Conference 2007 - Washington DC 17
SOA Orchestration
TransportTransport
SecuritySecurity
Internet Infrastructure
DeliveryDelivery
Data Services
Data ServicesProc
ess
Proc
ess
RegistryRegistry
33
22
4455
66
Business Model
ServiceAgreement
Context/ Roles
BusinessProcesses
Description
Msg ExchangeProfile
Search / Classification
Ownership Profiles
Requests to Consume
Service Definitions
Management / Versioning
Vocabulary / Semantics
Discovery
Vocabulary / Semantics
Packaging / Encoding
XML / edi
Validation / Assembly
Mapping / Transform
XSD, CAM, Schematron
State & Context
Addressing / Envelope
Transactions
Coordination
Push / Pull
Other protocols
Other services
Security Credentials
Quality of Service
Reliable Messaging
REST
WSDL
B2B
WSI
11
SOAP / http / https
Metatopia Conference 2007 - Washington DC 18
Agenda
SOA information integration needs
Selection of illustrative field examples
Solution and technology approach
Demonstration
Summary / Next Steps
Metatopia Conference 2007 - Washington DC 19
Domain Examples
DOJNIEM/LEX
Fannie MaeEDI-esque
XML
PESCDictionary
andschema
UBL schema
and CCTS
-Universal Business Language (UBL)
-OASIS standard
-Uses UML models and CCTS approach
-Derived from xCBL and simplEDI approach
-Supply chain schema with joint initiative EU / Asia / US
-Developed by GTRI for DOJ community
-Vocabulary based
-LEXS schema built using NIEM vocabulary
-Law enforcement and court applications
-Developed for Education Department
-form-style XML
-Older XSD techniques for simple flat schema model
-No re-use of common structures
-Student loans and transcripts
-Developed by BAH and NG for form-based application submissions
-Translate previous paper form into multiple schema sections
-Massive schema with extended namespaces
-Set of functional transactions and configuration templates for managing and processing elections
-Common set of components and vocabulary
-Widely differing use patterns and items by country localization
OASISEML
Grants.govform based
schema
-Transactions replacing EDI
-Overloaded reuse of structure components
-Financial reporting data
-Financial transaction data
-Extended use of code sets to label transaction content / purpose
Metatopia Conference 2007 - Washington DC 20
Challenges for Users and Implementers
DOJNIEM/LEX
Fannie MaeEDI-esque
XML
PESCDictionary
andschema
UBL schema
and CCTS
-Complex schemas that meet extended use cases
-Less obvious how to create sparse subset for typical everyday uses (in State / in Country)
-Verbose syntax and structures from CCTS models
-Complex schema with use model that does not necessarily map to domain information stores and search mechanisms
-Platform transport formats determine transaction packaging
- Limited interoperability of search methods
-Implementers have created baseline samples for typical student use cases
-Backend applications developed by solution providers
-Interoperability verified between providers solutions
-Extended form field edits and code lists
-Large structure and verbose syntax
-Excessive use of namespaces
-Extensive cross-field and form relations
-Related content edits (PDF documents)
-Documenting widely differing use patterns and items by country localization
-Understanding how to use flexible structure constructs for their own applications
-Verification and certification of exact usage and computations
OASISEML
Grants.govform based
schema
-Simple structures with need to show transaction use patterns
-Extensive use of codes
-Calculations and numbers hard to validate using schema alone
-Extensive cross-field relations
Metatopia Conference 2007 - Washington DC 21
Solution Metrics
• Can I create a standard simple open format to
describe my message structures and data content
rules?
• Can my partners validate their transactions in test
BEFORE they send them?
• How do people know what I will send them?
• I want something that’s simple and standards based –
leverages existing XML components
• Can I generate HTML documentation that is readable
by business analysts?
Metatopia Conference 2007 - Washington DC 22
Why not use XSD?
• Today’s XML schemas have complex structures with no context
awareness + no cross field association rules + no dynamic
lookups
• The XSD provides a model of ALL possible structure instances –
not the particular instance
• Excessive use of namespaces make for fragile XML transaction
handling
• Generating valid sparse transaction layout is tough
• Documentation diagrams hard to read
• How to create simple re-usable templates?
Metatopia Conference 2007 - Washington DC 23
XSD – PESC “GPA” model example
EVERYTHINGis optional! So what do I REALLY sendto you?
Metatopia Conference 2007 - Washington DC 24
Conceptual View of PESC transactions
Transaction
Templates
StructureRulesContext3
2
Use Rules
html
XML
Reportshtml
Publish
1
Common Details
Rules
4
Share Details
Address
Course
Contact
Student
Loan
Sponsor
Transcript
Content Rules
Lookup Values
Context
Versioning
Metatopia Conference 2007 - Washington DC 25
What about these needs?
• Versioning• Content consistency• Use rules• Codelists• Associations (what uses which?)• Guidelines• Providing local contextual validation services
Metatopia Conference 2007 - Washington DC 26
Versioning Challenges
• If the schema version changes – how to ensure it does not break our in place validations?
• How to rapidly adapt to rule changes in a production environment?
• How to develop user context driven deep version control and re-use of sub-components?
• Enhance and automate Test release cycle by improving transparency for bug fix process and expose change deltas to speed testing process? Support for regression testing?
Metatopia Conference 2007 - Washington DC 27
Agenda
SOA information integration needs
Selection of illustrative field examples
Solution and technology approach
Demonstration
Summary / Next Steps
Metatopia Conference 2007 - Washington DC 28
History and Status of CAM work
• OASIS technical committee• Five years of combined development in UN/CEFACT
and OASIS• OASIS v1.1 public standard and specification • jCAM open source implementation in Java• Creating simple XML-scripted open standard
mechanisms for XML transaction assembly and processing
• Re-use: leverages XPath and xslt, and extends schematron approach
• Developing templates for common industry formats
Metatopia Conference 2007 - Washington DC 29
CAM Process Architecture
CAMCAMCAMCAM
XML ParserXML Parser/ DOM/ DOM
XML ParserXML Parser/ DOM/ DOM
Built-in Built-in FunctionsFunctionsBuilt-in Built-in
FunctionsFunctions
XPathXPathhandlerhandlerXPathXPath
handlerhandler
SQL SQL persistencepersistence
SQL SQL persistencepersistence TermsTerms
RegistryRegistryTermsTerms
RegistryRegistryRule EngineRule EngineRule EngineRule Engine Post-Post-ProcessingProcessing
/ Errors/ Errors
Post-Post-ProcessingProcessing
/ Errors/ Errors
XML-awareXML-awareXML-awareXML-aware
EXTENSIONS
Metatopia Conference 2007 - Washington DC 30
jCAM Functional Components
CAMCAMCAMCAM
XML ParserXML Parser/ DOM/ DOM
XML ParserXML Parser/ DOM/ DOM
Built-in Built-in FunctionsFunctionsBuilt-in Built-in
FunctionsFunctions
XPathXPathhandlerhandlerXPathXPath
handlerhandler
SQL SQL persistencepersistence
SQL SQL persistencepersistence TermsTerms
RegistryRegistryTermsTerms
RegistryRegistryRule EngineRule EngineRule EngineRule Engine
EXTENSIONS
jaxenxerces
e.g. JRules,Others…
Saxon - xslt
planned using AJAX / ebXML
Post-Post-ProcessingProcessing
/ Errors/ Errors
Post-Post-ProcessingProcessing
/ Errors/ Errors
XML Data Mapping
XML-awareXML-awareXML-awareXML-aware
cam
XXXX
Metatopia Conference 2007 - Washington DC 31
Deployment Options
B2B Web services
jCAMenginejCAM
engine
XML
Templates
XML
MessageSystemXMLJava
APIReceive
Process
Validate Server
TemplatesRequest
XML
htmlResponseJavaAPIProcess
StructureRulesContext
1
2
3
1
2
3
Standalone
EDITOR
Template
XMLhtml
Report
Metatopia Conference 2007 - Washington DC 32
How does CAM work?
• CAM uses WYSIWYG approach to XML
• Starting with your XML sample – creates structure template from
that + default data content model
• Next – add your structure use rules – optional / repeatable, date
fields, allowed values, lookups
• Then make context business rules – cross field use rules, exclude,
include, variables
• Save template – run against samples
• Eclipse editor tool makes this all easy to do!
• Deploy to production using jCAM processor
Metatopia Conference 2007 - Washington DC 33
Eclipse CAM Editor
1
2
3
4
5
Available structures
Structure Rule Viewer
Rule Details
Results Viewer
ValidationProcess
Metatopia Conference 2007 - Washington DC 34
Agenda
SOA information integration needs
Selection of illustrative field examples
Solution and technology approach
Demonstration
Summary / Next Steps
Metatopia Conference 2007 - Washington DC 35
Using jCAM :
• Start with sample XML
• Use Eclipse template editor
• Load XML, generate CAM
• Enhance base template
• Test, refine and deploy
• Generate documentation
• Deliver business solution
Build SimpleBuild SimpleTemplateTemplateBuild SimpleBuild SimpleTemplateTemplate
ExtendingExtendingTemplateTemplateExtendingExtendingTemplateTemplate
Verify ResultsVerify ResultsVerify ResultsVerify Results
SimpleSimpleXML instanceXML instance
SimpleSimpleXML instanceXML instance
Document Document Rule DetailsRule DetailsDocument Document Rule DetailsRule Details
1
2
3
4
5
Metatopia Conference 2007 - Washington DC 36
Summary of CAM template features
• Ability to use <as:include> to modularize complex content
• Use of XPath references enables agile rule handling
• Full contextual support including variables, rules and context
sections
• Code lists can be externally built and then referenced
• Extensions section provides customization options
• Integration with xslt provides extended results handling and
error reporting capabilities
• Template representation model and XML enables extended
post-processing and documentation options
Metatopia Conference 2007 - Washington DC 37
CAM Functions Summary
excludeAttribute() startBlock() useElementByID() useAttribute()
excludeElement() endBlock() useTreeByID() useChoice()
excludeTree() checkCondition() setDefault() useElement()
makeOptional() makeRecursive() restrictValuesByUID() useTree()
makeMandatory() setUID() orderChildren() useAttributeByID()
makeRepeatable() restrictValues() setRequired() useChoiceByID()
setChoice()
setId()
setLength()
setNumberRange()
setLimit()
setValue()
setDateMask()
setStringMask()
setNumberMask()
allowNulls()
datatype() or setDataType()
<as:BusinessUseContext> <as:Rules> <as:default> <as:context> <as:constraint action="makeRepeatable(//Items/Item)"/> <as:constraint action="makeOptional(//Item/comment)"/> <as:constraint action="setLength(//shipTo/state,2)"/> <as:constraint action="setDateMask(//PurchaseOrder/shipDate,YYYY-MM-DD)"/> <as:constraint action="makeOptional(//PurchaseOrder/comment)"/> <as:constraint action="restrictValues(//shipTo/@type,'US'| 'CA'| 'MX', 'US')"/> <as:constraint action="setDateMask(//PurchaseOrder/@orderDate,YYYY-MM-DD)"/> <as:constraint action="setNumberMask(//Item/@pno,###-###)"/> <as:constraint action="setNumberMask(//Item/quantity,###)"/> <as:constraint action="setNumberMask(//Item/price,####.##)"/> <as:constraint condition="//Item/@pno = 123-678“ action="restrictValues(//shipTo/state,'WA')"><as:annotation> <as:documentation type="documentation">Can only ship item 123-678 to Washington State </as:documentation></as:annotation> </as:constraint> <as:constraint condition="$QuickBooks = true“ action="excludeElement(//Item/comment)" /> </as:context> </as:default> </as:Rules> </as:BusinessUseContext>
Example of Rules Use
Metatopia Conference 2007 - Washington DC 38
Interactive Documentation (iDoc)
CAM Template
XSLT
iDocwiki
HTML
Metatopia Conference 2007 - Washington DC 39
Agenda
SOA information integration needs
Selection of illustrative field examples
Solution and technology approach
Demonstration
Summary / Next Steps
Metatopia Conference 2007 - Washington DC 40
Value Proposition
• Making XML transaction handling simpler and predictable
• Extends and clarifies your existing XSD schema structures
• Quick and easy rule building from sample XML transaction
• Enabling more robust fault tolerant processing + versioning
• Providing open sharable templates and documentation
• Re-use easier through support for includable components
• Ability to integrate to business processes and context
• Open source, open public standard toolkit – editor + engine
Metatopia Conference 2007 - Washington DC 41
What’s Next / Call to Action
• Develop template sets for your business domain
• Integrate into your messaging exchanges
– OrionSMG supports jCAM
– Web service test-bed
• Publish to registry to facilitate adoption
– Create document templates to generate registry
content / guidelines
• Use Wiki / SourceForge to facilitate sharing
Metatopia Conference 2007 - Washington DC 42
Questions?
Metatopia Conference 2007 - Washington DC 43
www.jcam.org.uk
camprocessor.sourceforge.net
www.drools.org (JRules)
www.oasis-open.org/committees/cam
docs.oasis-open.org/cam
Resources:
Metatopia Conference 2007 - Washington DC 44
A special mention for our contributors to the CAM and jCAM work:
UK - Martin Roberts and team from BTplcUS - Sidhartha Nagolu from AC-Tech / NIH
Credits:
top related