Application IntegrationApplication Integration””The devil is in the details”The devil is in the details”
ContentContent
• Why is integration an issue?• How do I recognize a decent integration architecture
when I see one?• Could there possibly be something called an
Integration Model?• Problems you might run into as an application
integrator!• Saved by an Application Integration methodology!• Corus Quicklink – What sort of hack is that?
What the heck does it do?
Why is integration an issue?Why is integration an issue?
• Demand of new and improved business processes
• Time To Market • Protection of IT investments• The dream of the Connected Economy
Why doesn’t standards solve the Why doesn’t standards solve the integration problem?integration problem?
• Having many standards is like having none• Standards does not protect you from
semantical differences• Standards are not the solution to the
system legacy
• They are however a goal to aim for.
Why are there so many Why are there so many buzzwords in this area?buzzwords in this area?
• Application integration has become HOT• No generally accepted problem definitions• Complete mixup in terminology
– Standards– Infrastructures– Architectures– APIs– Protocols– Products
What is application integration?What is application integration?
• Seemless interaction between applications• Doing the unintended• Old thing, new requirements• Both but independently information and
access
Real life integration scenarioReal life integration scenario
X
CallCenter
Oracle Applications
Pris
AbalonCRM
ColumbineBroadcast
IntranetExtranet
WebbportalLuvitLMS
AR(Kundreskontra)
AP(Leverantörs-
reskontra)
Personal
SökinterfaceContent
Projects
006BroadcastContent
012Medlemsinfo
och kursregistrering
023Debiteringsunderlag
kursavgifter
011Debiteringsunderlag
reklamProvisionsunderlag
027Debiteringsunderlag
timmar
004Inköps-
information
026Prissatta
debiteringar
021Debiterings-
underlagorder
007Webb-
Content
1999-12-08, lax
028Försäljningsprovision
underlag
CMCASH HR
(HumanResources)
GL(Huvudbok)
FAAnläggnings-
register
005Likviditetsinfo
Systemsamband ÖversiktK-World Prio 2 Snart
SubscriberAuthorization
System(Nätoperatörer)016
KundinfoAuktorisationer
Avauktorisationer
015Betalningsstatus
LedningsinfoDataware-
house
1/831/10-99/00
11/10-99/00
2000-03
2000-03
99/00
1/91/101/9
008Tablå i SI-form
043Innehållsinfo
?
022Kundinfo
020Aktörer och
kontaktpersoner
013Validerad
medlemsinfokursdeltagare
017Studie-resultat
utvärderingar
Mindport(Canal+)
001Playlist
025Tittar-
aktiviteter
010Besöks-
aktiviteter
009Leverantörs-
info
030Asrun
029 Bokföring löner
031Tittarkunder
Kortnr
034Aktuellprislista
032PAP
033Produktinfo
Kunskaps-marknadenEasy-
update
003Tx Förbrukning
035Kursutbud
Web
014Studenter
och aktörerlearning
PARBGC, PG
UC
036Medlems- och kundposterAdresser och kreditfakta
038Valuta till Columbine
Inventory
040Organisations-
info
E-commerce
Press-Master
041Säljare
042Kursutbud
LMS044
BasinfoKodtabeller
How do I recognize a decent integration How do I recognize a decent integration architecture when I see one?architecture when I see one?
Rocky road to integration...Rocky road to integration...
• Complexity increases heavily with number Complexity increases heavily with number of integrated systems of integrated systems
• Heavy impact on the integrated systemsHeavy impact on the integrated systems• System dependencies reduces availability System dependencies reduces availability
for integrated systemsfor integrated systems• Hard to support and change existing Hard to support and change existing
integration solutionsintegration solutions• Integration crosses organization Integration crosses organization
boundaries with “political” discussions as a boundaries with “political” discussions as a resultresult
Central Knowledge BaseCentral Knowledge Base
System B
System C
System A
System E
System D
Knowledgebase
Integration model with common viewIntegration model with common view
System B
System C
System A
System E
System D
Could there possibly be something Could there possibly be something called an Integration Model?called an Integration Model?
Modelling realityModelling reality
Picture of a Forest
Model of a Forest
Trees AnimalsArea
Litter
Organisms
1..1
0..*
1..1
0..*
1..1
Forest
Sun
Lumberharvest
Use of a modelUse of a model
• Declared semantics of information objects• Declared syntax of information objects
• But then there is operational semantics which is uncaptured by the model because it occurs in the eyes of the beholder
Operational SemanticsOperational Semantics
Trees Animals
Area
Litter
Organisms
1..1
0..*
1..1
0..*
1..1
Forest
0..*
Program------------------------------------------------
Integration ModelIntegration Model
• Contains all application views of information objects + common reference view
• Describes business rules in the form of– Integrity checks– Transformations– Filters– ....
Integrity checksIntegrity checks
• Can be local to an information object IF carType = ”SPORTSCAR” THEN horsePower > 200
• Can also be referential i.e. depending on other information objects Car.driver EXISTS IN DriverLicenseRegister.driver
TransformationsTransformations
• Vital to a integration model because they describe how to go from one information view to another
• Note, transformation can (and often do) ”fork out” or ”fork in” i.e. one information element in one view correlate to several in another information view.
Syntax transformationsSyntax transformations
Project management viewCustomer:•Name 20 chars•Phone number is optional•Identified by a number
Sales Support viewCustomer:•Name 30 chars•Phone number is mandatory•Identified by a string
Finance department viewCustomer:•Name 30 chars•Identified by Phone number
Common viewCustomer:•Name 30 chars•Identified by internal id•Phone number optional•Any other attribute
Semantical transformationsSemantical transformations
Project management viewCustomer:•Created at project start•Deleted at project end
Sales Support viewCustomer:•Created at first contact•Never deleted
Finance department viewCustomer:•Created at first invoice•Deleted 10 years after last invoice
Common viewCustomer:•Common lifecycle for all views which implies new state attributes
FiltersFilters
• Filters are criterias describing when information in different views are eligible for delivery to applications.
Integration ModelIntegration Model
applicationA
IntObject: X
IntObject:U
Y_to_Z
Z_to_X
X_to_Y
Y_to_U
W_to_Y_Z
sem_BU
BU_filter
sem_BW
BW_filter
sem_AX
AX_filter
publishingfrom
application B
publishingfrom
application Bsubscription toapplication A
publishingfrom
application A
subscription toapplication B
subscription toapplication B
1
0..n
1
0..n
IntObject:Z
Proxy modelfor
application A
Commonreference
model
Proxy modelfor
application B
IntObject:Y
IntObject:W
Use-casesUse-cases
• Use-cases are good practice in modelling a integration scenario
• Use-cases translates to an chain of operations in the integration model
application AIntObject: X
IntObject: U
Y_to_Z
Z_to_X
X_to_Y
Y_to_U
W_to_Y_Z
sem_BU
BU_filter
sem_BW
BW_filter
sem_AX
AX_filter
publishingfrom
application A
subscription toapplication B
1
0..n
1
0..n
IntObject: Z
Synonymmodel for
application A
Commonglobalmodel
Synonymmodel for
application B
IntObject: Y
IntObject: W
Error handlingError handling
• Since there are no good receiver of errors in an integration scenario. Error situations and their counter actions must be modelled
• It is common that the error situation use-cases outnumbers the normal operation use-cases (if not, you might have neglected some possible error situations)
Problems you might run into as an Problems you might run into as an application integrator!application integrator!
Some important problem casesSome important problem cases
• Multiple master synchronisation• Asynchronous information correlation• Message operation calculations
Multiple master synchronisationMultiple master synchronisation
System BSystem A
Customer CustomerIntegration
environment
System BSystem A
Customer CustomerIntegration
environment
Loops
Differentlatencytimes
Once per hour
Once per second
Asynch. information correlationAsynch. information correlation
System B
System A
Customer
Order withvalid
customer
Integrationenvironment
System C
Order
Once per hour
Once per second
Message operation calculationsMessage operation calculations
System BSystem A
Customer CustomerIntegration
environment
System BSystem A
Customer CustomerIntegration
environment
Good update
Bad create
Good create
Bad create
Good remove
”Do nothing”
Problem summaryProblem summary
• Technical by nature• Asynchronous communication gives extra
complexity (Asynch comm is however necessary to have loose integration)
• The solutions of these problems involves the integration model
IS-supported business processesIS-supported business processes
Businessplan
Placeorder
PayFind
productUse
product
Selling
Production
Marketing
Deliver
Invoicing
Production
Sales
Customer
Finance
Information systemsplan
Webportal PDM
OACRM
order
Productdef.
customer
Payingcustomer
Productdef.
Top-down approachTop-down approach
From process-analysis to mapping bit and bytes,preferably with the same tool and methodology!
Order_header
order_id number(20)customer varchar(30)status varchar(1)
Order
Order varchar(10)member varchar(30)
rules
and m
appin
g
Utilities and common resources
Methodology anatomyMethodology anatomy
Integration process
ImplementationBusinessanalysis
feedback
Anatomy; continuedAnatomy; continued
Integration process
Overall control of an integration project. Assures that relevant benefit is delivered to customer
Business analysis
Assure that the integration will be relevant according to business goals. The business analysis is independent on the integration tool used.
Implementation Defines an efficient way to implement the ”whats” of the business analysis
Data access Subpart of implementation, dealing with connectivity between applications. - How can applications be accessed?
Information content
Subpart of the implementation, dealing with transformation and interpretation of information - How should information be handled when transferred between applications.
A closer look at business analysisA closer look at business analysis
Utilities and common resources
Integration process
ImplementationBusinessanalysis
feedback
• Goal model• Process / activity model• Conceptual model• Application-Activity-Object model• Relatively small part of projects and when it is necessary!
Business analysisBusiness analysis
Workshops
1. Current state
2. Future state
3. Investigate the need for integration
4. Evaluate for solution
Goal modelGoal model
«sub-goal»Supply chainmanagement
«mean»Business
automation
«goal»Business
continuance
«mean»Focus on core
business
«quantitative goal»Increased profit
prerequisite forfacilitatesfacilitates
«problem»No XML order/invoice capabilities in financial
application
«quantitative goal» Minimize costs
«mean»1) Obtain Web-system that
facilitates XML-order/invoice 2) integration of order/invoice
between Web and financial appl.
Mitigates problem
enables
contradictory
Is part of
To create a relationship between organization intentions and the
objectives for integration
Process/activity modelProcess/activity model
PayBuyUse
product
Sell
Produce
Marketing
Deliver
Invoicing
Production
Sales
Customer
Finance
Identify organization business processes and their interaction
Process/activity model; continuedProcess/activity model; continued
Identify:
• the activities within the processes
• the need for information
• where the information is available
• information produced
• where the created information is stored
Sell
register customer
place order
Identify prospects
Prepare prospects
Conceptual modelConceptual model
• Defining the participating business objects and their associations
• Optional• UML class diagram
ProductOrder
Customer
is placed by
contains
Organization
Person
Application-Activity-ObjectApplication-Activity-Object diagram diagram
OA
OA
OA
WEB
CRM
WEB
CRM
cust
om
er
PDM
PDM
WEB register customer
CRM register customer
place order
PDM
pro
du
ct
Ob
jec
t s
ou
rce
Ac
tiv
ity
p
erf
orm
ed
in
Ob
jec
t ta
rge
t
product development
pro
du
ct
cust
om
er
production
Pro
du
ct
ba
lan
ce
Pro
du
ctb
ala
nce
cust
om
er
ord
er
Business analysis; continuedBusiness analysis; continued
Strict integration focus:
• Eliminate activities where no information flow is prevalent
• Order between activities are not important• Where activities can be performed are important• (potential) source / target of information are important
Result of business analysisResult of business analysis
• Consistent view on integration need - or alternatives to integration!
• Integration goals and means mapped against organization intentions or visions
• Input to implementation resulting in transparency between analysis and implementationapplication-activity-object diagram application collaboration diagram
• Conceptual model - optional, used for consistency control
A closer look at implementationA closer look at implementation
Utilities and common resources
Web-site
Integration process
ImplementationBusinessanalysis
feedback
Implementation
Data access
Informationcontent
Application collaboration diagramApplication collaboration diagram
• Integrated applications described as subsystems• Only data flow (normally asynchronous)• Integration objects or their sub-sets• No attributes or similar details • Collaborations outside integration for visualization
billing info (p
ay date)
order
order (shipping date)
Customer, billin
g info
Storagesystem
Financesystem
WebPortal
Spedition
Integration syntax and semanticsIntegration syntax and semantics
Project management:
Customer is: • Anybody within an active project• 20 chars in name• phone number is a mandatory attribute • identified by a number
Sales department:
Customer is: • anyone which may be involved in business• 30 chars in name • phone number is a non-mandatory attribute • identified by a string
Finance department:
Customer is: • Legal entity which has been involved in a business transaction• 25 chars in name• identified by organisation number (mandatory)
Reference:
Customer is: • anyone which may be involved in business• 30 chars in name• identified by internal key (mapped against a string, phone number and a number)
Information contentInformation content
Sale support system
Projectmanagement
Finance system
Proxymodel
Integration model
Reference model
Businessrules
Use case diagrams Use case diagrams - - for requirement specificationsfor requirement specifications
Actors: integrated applications
Publish use case: responds to creation, update or deletion of integration objects in publishing application with C, U or D in ref. model
Subscribe use case: C, U or D of integration objects in subscribing client application as a result of C, U or D in ref. model
Application A«extend»
Application B
Publish use case
Subscribeuse case
Use cases; continuedUse cases; continued
Publish from WEB
Reference model
Use case Scenario (condition) Result
Customer is created - FIN: null
Order is created - LOG: Order is shipped
FIN: Billing info is created
Use case Scenario (condition) Result
A customer enters member information
- Customer is created
Order is entered Related to a customer Order is created
Not related to a customer Error case
Compilation of integration modelCompilation of integration model
• Proxy models and reference model defines static structure of information (relations between integration objects, i.e. referential integrity)
• Data flows in application collaboration diagram defines which transformations must be created
• Use cases describes the behavior of the transformations (and events for integration)
• Attribute mapping is a part of defining the transformations (may be separated later on)
Class diagrams for integration modelClass diagrams for integration model
-Organization no-Name
Customer *
Project *
1
0..*
Proxy model A
-Organization no-Name
Customer
Project
Reference model
-Customer-Contact person-Project no
Project
po6
po4
po3
p
Proxy model B
po5
po7
1
0..*
po1
po2
po3
po4
Integration model:- Proxy model- Referential integrity constraints- Reference model- Post-operations
Corus QuicklinkCorus QuicklinkWhat sort of hack is that ?What sort of hack is that ? What the heck does it do?What the heck does it do?
Where does it come from?Where does it come from?
• Based on experiences made from a number of integration projects
• Theoretical reasoning • Demands from customers, implementers and
maintenance staff
Corus ArchitectureCorus Architecture
i BUILDER• i CONTROLLER• i CONFIGURATOR• i USER
Generators
HTTP Server
Servlet Engine
Repository
i BUILDER server
ODS
HTTP Server
Servlet Engine
i ENGINE
Participant
UI layer
DB layer
App layer
iDM Participant
UI layer
DB layer
App layer
iDM
Integration object - Quicklink atomIntegration object - Quicklink atom
Table
Event log
History log
View Messagetrigger
Pre OP Post OP
System accessSystem access
Client application A
Integration Engine
Log fileINI file
DialogManager
Java VM
ODS
Client application B
DialogManager
Log fileINI file
Java VM
Plug-in Plug-in
AP
I
Genericqueuetable
Genericqueuetable
Publishingadaptor
Subscribingadaptor
Feature example - Pending eventsFeature example - Pending events
• Solves the problem with inconsistent messages– what can the publishing application do with an error?
• A pended message stays in the message table and post- operation is not performed.
• Can self-heal
Integration engine
Publishingsystem
Subscribingsystem
Subscribingsystem
Feature example - Consistency monitorFeature example - Consistency monitor
• Autonomous process checks consistency• Can release pending events
Integration engine
Publishingsystem
Subscribingsystem
Subscribingsystem
CM
The Corus Integration ProcessThe Corus Integration Process
Order process Production
Information Model
Data Model
Application 1 Application 2
Information Model
Data Model d i g i t a l d i g i t a l
d i g i t a l WANIS/IT
IA
RepositoryGenerator
// Kundinfoadaptor// for system sysA
procedure send_kundinfo{ for columns 1 to 3 do { store Kunder.col(i); } … //user part begin call datum_check //user part end}
Goal model for integration automationGoal model for integration automation
Contradictional to
Methodology
leveraged by
SimplicityFlexibility and
advanced functionality
Rapid implementation
Solving all types of integration
Competetive integration
PrerequisiteforFacilitates
Prerequisitefor
Facilitates
Basic principles of Corus methodologyBasic principles of Corus methodology
• Top-down approach– from process, all the way down to the bits and bytes
• Iterative development and successive implementation • Transparency
– no hand-over– single, minimum documentation– all deliverables persistent in one single repository
• Problem split – technology vs. business– decisions will be made by correct persons
• Plug-able. Can be adopted to RUP or PROPS• Middle of the road (solve 80 %)• Standards where possible
– UML for documentation and visualization