PeerDirect Distributed PeerDirect Distributed Enterprise Enterprise Reach Your Technical Peak The Final Ascent Session IV – Q4 2003
PeerDirect Distributed PeerDirect Distributed EnterpriseEnterprise
Reach Your Technical Peak
The Final Ascent
Session IV – Q4 2003
© 2003 Progress Software Corporation2Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
PeerDirect Distributed EnterprisePeerDirect Distributed Enterprise
■ Introduction to Database Replication
■ Architecture Overview
■ Configuration
■ Replication Rule Design
■ APIs and Event Callback
■ Administration
Agenda
© 2003 Progress Software Corporation3Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
PeerDirect Distributed EnterprisePeerDirect Distributed Enterprise
■ Introduction to Database Replication
■ Architecture Overview
■ Configuration
■ Replication Rule Design
■ APIs and Event Callback
■ Administration
Agenda
© 2003 Progress Software Corporation4Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Introduction to Database ReplicationIntroduction to Database Replication
■ Back-up■ Fail-over■ Latency avoidance■ Load balancing■ Reporting■ Distributed environments
Why Database Replication?
© 2003 Progress Software Corporation5Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Introduction to Database ReplicationIntroduction to Database Replication
Define Distributed?
Data Center
Remote Offices
Remote Users
DB
Apps
© 2003 Progress Software Corporation6Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Introduction to Database ReplicationIntroduction to Database Replication
Goals of a Distributed Enterprise
■ Latency avoidance
■ Risk avoidance
■ Application functionality
© 2003 Progress Software Corporation7Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Introduction to Database ReplicationIntroduction to Database Replication
DB
What if…
Data Center
Remote Offices
Remote Users
DB
Apps
Apps
Apps
Apps
DB
DB
© 2003 Progress Software Corporation8Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Introduction to Database ReplicationIntroduction to Database Replication
■ Breaks dependency on centralized application architectures
■ Radically improves employee productivity with decentralized applications– Disconnected use
■ Creates an inherently resilient application architecture
■ Eliminates the latency and availability problems caused by the centralized model
PeerDirect Distributed Enterprise
© 2003 Progress Software Corporation9Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Introduction to Database ReplicationIntroduction to Database Replication
■ Log based– Database activity logged for periodic ‘replay’ at all
locations■ Queue based
– Middleware intercepts application to database activity for periodic replay at all locations
■ Table based– Captures changes, queries source and duplicates
values at all locations
Approaches
© 2003 Progress Software Corporation10Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Introduction to Database ReplicationIntroduction to Database Replication
■ Log based– Log size– Can not remain dormant long– Periodically need to re-synch– Inefficient use of bandwidth– Typically only hub and spoke topology
■ Queue or Message based– All of the above– Application must be modified
■ Table based– Not real-time– Increases database size
Drawbacks of each approach
© 2003 Progress Software Corporation11Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
PeerDirect Distributed EnterprisePeerDirect Distributed Enterprise
■ Introduction to Database Replication
■ Architecture Overview
■ Configuration
■ Replication Rule Design
■ APIs and Event Callback
■ Administration
Agenda
© 2003 Progress Software Corporation12Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Architecture OverviewArchitecture OverviewThe Basics
■ Table based– Vs Log or Queue based
■ System tables for storing metadata■ Engine appears as another user■ Access via SQL and ODBC
© 2003 Progress Software Corporation13Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Architecture OverviewArchitecture OverviewFeatures and benefits
Feature Benefit■ Bi-directional update-
everywhere model■ Read-write data between
multiple databases
■ Replicate, synchronize, and distribute corporate data across multiple locations
© 2003 Progress Software Corporation14Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Architecture OverviewArchitecture OverviewFeatures and benefits
Feature Benefit■ Scheduled
synchronization■ Net change compression■ Strong encryption■ Subset data using
‘slices’
■ Maximize network efficiency, reduce costs
■ Security– Controlled access to
data
© 2003 Progress Software Corporation15Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Architecture OverviewArchitecture OverviewFeatures and benefits
Feature Benefit■ Replicates between
different database types– MS SQL Server and
Progress– Oracle and PostreSQL
■ Supported databases– Progress
– Oracle
– SQL Server
– DB2
■ Share data in mixed environments
■ Consolidated view of corporate data
© 2003 Progress Software Corporation16Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Architecture OverviewArchitecture OverviewFeatures and benefits
Feature Benefit■ Multiple topologies
supported– Peer-to-peer– Hub and spoke– Load balance clustering
■ Provide new options for building scalable systems
■ Flexible configurations
© 2003 Progress Software Corporation17Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Architecture OverviewArchitecture OverviewFeatures and benefits
Feature Benefit■ Auto-discovery of nodes
within the replication network
■ Improve quality of service and system availability
■ Improved system administration
■ Allows mobile workers who are disconnected or have low-bandwidth limitations access to enterprise applications
© 2003 Progress Software Corporation18Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Architecture OverviewArchitecture OverviewFeatures and benefits
Feature Benefit■ Database level
configuration– Adds needed tables to
replicated database to manage replication
– Does not modify user-defined tables
■ Existing application does not necessarily need to be altered
© 2003 Progress Software Corporation19Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Architecture OverviewArchitecture Overview
■ Dynamic Data Slicing Architecture (DDSA)– Table partitioning - schema– Work set partitioning - query– Column level partitioning - fragment– Dynamic data migration
■ Dynamic Bandwidth-Managed Partner Selection (DBP)– Auto-discovery, auto-balanced– Avoids overloading any one server– Backbone, server and workstation algorithms
■ Collision Avoidance Methodology (CAM)– Record fragment management – related columns– Consistent resolution contracts– Rich API for custom resolution
Patented replication technology
© 2003 Progress Software Corporation20Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
System Tables41 tables ofsetup data
Control Tables1 table per
database table
Architecture OverviewArchitecture Overview
Data Table “M” Rows
Database
Application
Control Table “M” Rows
PK
PK
Table-based
© 2003 Progress Software Corporation21Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Architecture OverviewArchitecture Overview
■ Principal components
PeerDirect Distributed Enterprise
ReplicationEngine(PDRE)
ReplicationDesigner(PDRD)
ReplicationAdministrator
(PDRA)
PeerDirect Distributed Enterprise(PDDE)
© 2003 Progress Software Corporation22Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
PeerDirect Distributed EnterprisePeerDirect Distributed Enterprise
■ Introduction to Database Replication
■ Architecture Overview
■ Configuration
■ Replication Rule Design
■ APIs and Event Callback
■ Administration
Agenda
© 2003 Progress Software Corporation23Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationConfiguration
■ Communicates via SQL through ODBC■ The closer the Replication Engine is to the
database, the better the performance– Supports Win32 and Linux
■ Configuration of InnerEdge and OuterEdge servers affect the way in which sites will be replicated
Replication Engine
© 2003 Progress Software Corporation24Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationConfigurationPDRE setup
Engine
ReplicateEngine
Replication
DB
DB
DB
DB
Engine
Replication
Engine
Replication
DB
© 2003 Progress Software Corporation25Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationConfiguration
■ Site Types – 3 Types
■ Complete ■ High Availability■ Low Availability
– Based upon■ Data set■ Connectivity
– Determines replication partner selection
Site Types
© 2003 Progress Software Corporation26Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationConfiguration
■ Considerations– A Replication Engine should be on either
end of any WAN connection– PDRE traffic is compressed and encrypted
whereas ODBC traffic is not– On a LAN, consider fault tolerance and
performance vs. cost of hardware– Engine may reside on a different platform
■ Win32 and Linux
Engine placement
© 2003 Progress Software Corporation27Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationsConfigurations
■ Dynamic partner selection algorithms allow for load balancing and fault tolerance
■ Partner selection is affected by a site’s type
■ Available site types– Low Availability– High Availability– Complete
Site types
© 2003 Progress Software Corporation28Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationsConfigurations
■ Low Availability– Sites that are mostly or occasionally
disconnected– Compact
■ High Availability– Sites that are mostly connected but do not
contain all data– ‘Spine sites’
Site types
© 2003 Progress Software Corporation29Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationConfigurationSimple Topologies – Inner Edge
– High Availability sites– All or most data slices– Reliable connectivity
Engine
ReplicateEngine
Replication
DB
Data Center
© 2003 Progress Software Corporation30Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationConfiguration
– High or Low availability sites– Pertinent data slices– Questionable connectivity
Engine
ReplicateEngine
Replication
DB
Remote Office
Simple Topologies – Outer Edge
© 2003 Progress Software Corporation31Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationConfiguration
Engine
ReplicateEngine
Replication
DB
Data Center
Engine
ReplicateEngine
Replication
DB
Engine
ReplicateEngine
Replication
DB
Remote Office
Engine
ReplicateEngine
Replication
DB
Remote Office
Sample PDDE Topology
© 2003 Progress Software Corporation32Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationConfiguration
■ Aircraft Manufacturer– Aircraft maintenance application– Maintenance records were handled either on paper or in different
centralized databases– Maintenance issues cause commercial aircraft delays and military
readiness issues■ Solution: One PeerDirect InnerEdge Server and multiple OuterEdge
Workstations– Remote capabilities, work sets, and occasionally connected users– Aircraft maintenance data captured while aircraft is being maintained
resulting in significant cost savings
InnerEdge
Distributed Enterprise
OuterEdge
OuterEdge
OuterEdge
OuterEdge
© 2003 Progress Software Corporation33Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
ConfigurationConfiguration
■ Medical Imaging– Patient test results contain images and complex data sets– Data only available at site where test performed– Collaboration with doctors at other locations nearly impossible
■ Solution: One PeerDirect InnerEdge Server and multiple OuterEdge Workstations– New test introduced into replication network– Can be viewed by physician at any location at anytime
InnerEdge
Distributed Enterprise
OuterEdge
OuterEdge
OuterEdge
OuterEdge
© 2003 Progress Software Corporation34Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
PeerDirect Distributed EnterprisePeerDirect Distributed Enterprise
■ Introduction to Database Replication
■ Architecture Overview
■ Configuration
■ Replication Rule Design
■ APIs and Event Callback
■ Administration
Agenda
© 2003 Progress Software Corporation35Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule Design
■ Specify the application database■ Select the tables to replicate■ Define fragments to minimize the number of data
collisions■ Subset data into work sets■ Arrange tables in transaction sets and/or groups■ Verify and save rules■ Prepare to activate replication-enabled database
Basic steps to defining rules
© 2003 Progress Software Corporation36Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule DesignAnalyze the schema Partial Sports2000
Schema
Order
PK,U1 Ordernum
FK1,FK2,FK3,U1 CustNumI1 OrderDate
ShipDatePromiseDateCarrierInstructionsPOTerms
I3 SalesRepFK3 BillToIDFK2 ShipToIDI2 OrderStatus
WarehouseNumCreditcard
RefCall
PK,U2,U1 CallNum
FK1,U1 CustNumCallDateSalesRep
U2 ParentI1 Txt
BillTo
PK,FK1 CustNumPK BillToID
NameAddressAddress2City
FK2 StatePostalCodeContactPhone
Salesrep
PK SalesRep
RepNameRegionMonthQuota
OrderLine
PK,FK1 OrdernumPK Linenum
FK2,I1 ItemnumPriceQtyDiscountExtendedPrice
I2 OrderLineStatus
ShipTo
PK,FK1 CustNumPK ShipToID
ContactAddressAddress2City
FK2 StatePostalCodePhoneCommentsName
Customer
PK CustNum
I3 NameAddressAddress2City
FK2 StateI2 Country
PhoneContact
FK1,I4 SalesRepI1 Comments
CreditLimitBalanceTermsDiscount
I2 PostalCodeFaxEmailAddress
Invoice
PK Invoicenum
FK2,I1 CustNumI2 InvoiceDate
AmountTotalPaidAdjustment
FK1,I3 OrderNumShipCharge
SalesRep Work Set
© 2003 Progress Software Corporation37Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule Design
■ Unit of replication is whole record■ Changes occur at multiple locations
– One user changes Addr– One user changes Limit
■ Can cause ‘False Collisions’
Generic replication
Name Addr City Zip Ctry Pymt Rating Limit
© 2003 Progress Software Corporation38Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule Design
■ Unit of replication is each field■ Changes occur at multiple locations
– One user changes Addr and Limit– One user changes Addr and Zip
■ Can cause ‘Silent Data Corruption’
Common solution to generic replication
Name Addr City Zip Ctry Pymt Rating LimitName Addr City Zip Ctry Pymt Rating Limit
© 2003 Progress Software Corporation39Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule Design
■ Unit of replication is a group of fields■ Changes occur at multiple locations
– One user changes Addr and Limit– One user changes Addr and Zip
■ Helps avoid ‘False Collisions’■ Optimizes the replication cycle
PeerDirect solution - fragments
Name Addr City Zip Ctry Pymt Rating Limit
© 2003 Progress Software Corporation40Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule Design
■ PeerDirect allows you to define the business rules for sub-setting around base tables
Introduction to work sets
Cust
Prod
Price
HistCreditAcct
ADetl Trans
TDetl
Ctry
Off
HistCreditAcct
ADetl Trans
TDetl
Off Prod
Price
© 2003 Progress Software Corporation41Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule Design
■ Define the ‘Ctry’ base table■ Subscribe the site to its country
Defining work sets
142 - Bouchard
217 - W. Gates
330 - A. Palmer
401 - L. Haney
. . .
84006239
93050403
93072677
96193414
97005567
1 - Canada
2 - USA
3 - Australia
. . .
Ctry
Cust
Acct12 - New York
29 - Toronto
37 - Montreal
44 - Sydney
. . .
Off
550 - N. Peart
98717696
99656643
. . .
99700174
© 2003 Progress Software Corporation42Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule Design
■ Each site stores "all and only" the data it needs
■ Tables belonging to work sets are replicated based on a request, called a subscription
■ Nesting work sets allows for further sub-setting of data
Work sets
© 2003 Progress Software Corporation43Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule Design
■ A set of insert, update or delete operations– A single unit of work– Must be completed as a whole
■ Offer the ability to enforce data integrity■ Two types of transactions supported
– Transaction sets– Transactions groups
Introduction to transactions
© 2003 Progress Software Corporation44Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
Replication Rule DesignReplication Rule Design
■ Maintains transaction integrity of dependant table during incomplete replication session
■ Updates all dependant records in a single unit of work
■ Must include a base table■ Creation process similar to work sets
Transaction sets
© 2003 Progress Software Corporation45Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
PeerDirect Distributed EnterprisePeerDirect Distributed Enterprise
■ Introduction to Database Replication
■ Architecture Overview
■ Configuration
■ Replication Rule Design
■ APIs and Event Callback
■ Administration
Agenda
© 2003 Progress Software Corporation46Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
APIs and Event CallbackAPIs and Event Callback
■ Available for use by all third-party developers
■ Available in both Unicode and ANSI■ 32-bit C-style function calls (standard
calling convention)
APIs
© 2003 Progress Software Corporation47Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
APIs and Event CallbackAPIs and Event Callback
■ Errors and Logging■ Control■ Command■ Configuration■ Database Record Information■ Event Handling■ Miscellaneous
API categories
© 2003 Progress Software Corporation48Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
APIs and Event CallbackAPIs and Event CallbackEvent handler callbacks
■ Allows you to modify some behavior■ Receive various kinds of event notifications■ PDRE events include
– Replication monitoring– Custom conflict resolution– Enhanced logging
■ Write and register custom DLLs■ Sample code
– C:\Program Files\PeerDirect\pdre\Samples\EvntClbck
© 2003 Progress Software Corporation49Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
PeerDirect Distributed EnterprisePeerDirect Distributed Enterprise
■ Introduction to Database Replication
■ Architecture Overview
■ Configuration
■ Replication Rule Design
■ APIs and Event Callback
■ Administration
Agenda
© 2003 Progress Software Corporation50Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
AdministrationAdministration
■ PDRE process listens for commands on port 2584– Network security infrastructure must be
configured to pass TCP data on this port■ ASCII protocol■ Provides interactive or programmatic
interface
Remote protocol
© 2003 Progress Software Corporation51Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
AdministrationAdministrationRemote protocol example
© 2003 Progress Software Corporation52Sim
plify
you
r bu
sine
ssS
impl
ify y
our
busi
ness
AdministrationAdministration
■ Administers the replication network– Maintenance of site information– Subscribing and unsubscribing to and
from slices– Setting up replication schedules
■ Allows for remote administration of all tasks
GUI Administrator