ASE121: Shared Disk Cluster Technical Review Linda LinSenior [email protected]
Ganesan Gopal Senior [email protected]
August 15-19, 2004
Sybase TechWave 2004
The Enterprise. Unwired.
Sybase TechWave 2004
The Enterprise. Unwired.
UnwirePeople
UnwireInformation
ManageInformationSybase WorkspaceIndustry and Cross Platform SolutionsAdaptive Server EnterpriseAdaptive Server AnywhereSybase IQDynamic ArchiveDynamic ODSReplication ServerOpenSwitchMirror ActivatorPowerDesignerConnectivity OptionsEAServerIndustry Warehouse StudioUnwired AcceleratorUnwired OrchestratorUnwired ToolkitEnterprise PortalReal Time Data ServicesSQL Anywhere StudioM-Business AnywherePylon Family (Mobile Email)Mobile SalesXcelleNet Frontline SolutionsPocketBuilderPowerBuilder FamilyAvantGo
Sybase TechWave 2004
AgendaASE HA ArchitectureShared Disk Cluster Feature OverviewShared Disk Cluster Architecture OverviewWhat Benefits The Customers Would See?Q&A
Sybase TechWave 2004
ASE HA ArchitectureCurrentStandby (active-passive): ASE on standby node is not started until primary node failure
Distributed workloads (active-active)12.0 implementation of companion server architecture
Work in progressConcurrent access (shared disk cluster)
Sybase TechWave 2004
Active-Passive
Disk StorageSANHA SystemASE DB QuorumTwo nodes failover cluster with single database store
ASE on primary node is active
ASE on standby node is inactive before primary failover
Sybase TechWave 2004
Active-Active Symmetric configurationSANHA SystemASE #1ASE #2ASE #1ASE #2Two node failover cluster with two database stores
Both ASE servers are active servicing disjoint applications
Both ASE servers are companions of each other for availability
Disk Storage
Sybase TechWave 2004
Motivations for Shared Disk ClusterTCOEmerging Low Cost Intel/Linux combo - 4-8 nodes with high speed interconnectAbility to consolidate multiple apps for effective utilization of unused H/W capacityEase of administration
Continuous AvailabilityUninterrupted operation in case a component fails.
High PerformanceIncrease the capacity by adding additional SMP Boxes with no restructuring of data.
Single System PresentationSingle Server appearance to the applicationApplications run at multiple nodes with Shared Data
Technology Trends
Sybase TechWave 2004
Shared Disk Cluster
Sybase TechWave 2004
Shared Disk Cluster - An instance of server failover
Sybase TechWave 2004
AgendaASE HA ArchitectureShared Disk Cluster Feature OverviewShared Disk Cluster Architecture OverviewWhat Benefits The Customers Would See?Q&A
Sybase TechWave 2004
Shared Disk Cluster HighlightsShared Disk Cluster (Concurrent Access)Single database storeSingle system presentationAll servers provide services at all timeAll servers can access all DBs directlyIndependent of platform-specific clustering serviceIncremental node growth and shrinkCluster-aware application partitioningWorkload based load balancingInstantaneous server failoverWorkload is re-distributed at a server failure
Sybase TechWave 2004
Shared Disk Cluster
Sybase TechWave 2004
Shared Disk Cluster Feature Logical ClustersSimilar to the Logical Process Management (LPM) feature thats currently available in ASERules can be specified at a instance level rather than at the engine level (LPM will be supported as well)
DBAs can setup application, users, etc binding to the various instances.
Sybase TechWave 2004
Logical Cluster with Application Partitioning
SANS1S2S3S4
Client connections are transparently redirected to a logical cluster
Cluster-aware application partitioning improves scalability
Easy to implement Service Level AgreementPrivate InterconnectQuorumSales ApplicationFinance ApplicationFinance ClusterSales ClusterS1S2S3S4Shared Disk StorageCluster DBFinance DBSales DBSharedFinance Logical ClusterSales Logical Cluster
Sybase TechWave 2004
Logical Clusters for Disjoint Applications
SAN
Single system presentation with transparent connection redirection to logical cluster
Cluster-aware application partitioning improves scalability
Frequently accessed data is available locally and infrequently accessed data is shared by all serversQuorumSales ApplicationFinance ApplicationPrivate InterconnectFinance Logical ClusterSales Logical ClusterS1Shared Disk StorageCluster DBFinance DBSales DBShared
Sybase TechWave 2004
Logical Cluster with Incremental Node Growth
SAN
Add more low cost SMP boxes to logical cluster as demands grow - TCO
New connections are directed to the least loaded server within a logical clusterQuorumSales ApplicationFinance ApplicationSales ApplicationPrivate InterconnectShared Disk StorageCluster DBFinance DBSales DBSharedFinance Logical ClusterSales Logical Cluster
Sybase TechWave 2004
Logical Cluster with Server Failover
SAN
Continuous Availability with Instantaneous Failover
Failover connections are redirected to another server in the same logical clusterPrivate InterconnectQuorumSales ApplicationFinance ApplicationSales ApplicationShared Disk StorageCluster DBFinance DBSales DBSharedSales Logical ClusterFinance Logical Cluster
Sybase TechWave 2004
Logical Cluster Failover
SAN
Continuous Availability with Instantaneous -Failover across logical cluster
When a logical cluster fails, failover connections are redirected to an alternate logical cluster
Databases are available as long as one server is available
Private InterconnectQuorumSales ApplicationFinance ApplicationSales ApplicationS3Sales Logical ClusterShared Disk StorageCluster DBFinance DBSales DBSharedFinance Logical Cluster
Sybase TechWave 2004
Logical Cluster Recovery
SAN
After a logical cluster recovers, connections are redirected to the logical cluster.
Private InterconnectQuorumSales ClusterSales ApplicationFinance ApplicationSales ApplicationS3Sales Logical ClusterShared Disk StorageCluster DBFinance DBSales DBSharedFinance Logical Cluster
Sybase TechWave 2004
Shared Disk Cluster Installation and Administration
Topics
Platform Recommendations
Installation
Configuration and Administration
Upgrade
Sybase TechWave 2004
Platform RecommendationsHomogeneous operating systems Linux or Solaris
Similar HW configuration
Gigabit Ethernet as private interconnects
Raw devices on SAN connected storage for globally accessible databases and quorum disk devicesOther options: clustered file systems or virtualized storage with volume managers
Single installation of $SYBASE on globally mounted file system
Sybase TechWave 2004
Cluster InstallationProcedure similar to that of a standalone SMP
Done from a single node in the cluster
Installation is on globally accessible raw devices and file systems
All instances in the cluster share single installation of databases,server binaries and scriptsconfiguration files.
Sybase TechWave 2004
Cluster Administration sybclusterClient side cluster administration tool Command line interfaceJava-based program supporting multiple platforms
Communicates with Unified Agent Framework (UAF) and ASEAgent on server side
UAF and ASEAgent are configured through UAF Web Console.
Sybase TechWave 2004
Cluster Administration sybcluster architecture sybclusterServer Node 1
Server Node 2
Sybase TechWave 2004
Cluster Administration sybcluster usageExample:sybcluster addcluster cluster ..sybcluster addserver server -cluster ..sybcluster startcluster cluster ..sybcluster startserver server cluster ..sybcluster stopserver server -cluster ..sybcluster stopcluster cluster ..sybcluster dropserver server -cluster sybcluster dropcluster cluster ..sybcluster serverstatus server -cluster ..sybcluster clusterstatus cluster ..
Command line input also takes user authentication and UAF agent discovery information
Sybase TechWave 2004
Cluster Administration Integration with third party cluster admin tool start, stop, monitor scripts binaryThird party cluster admin agentstart, stop, monitor scripts binaryServer Node 1
Server Node 2
Third party cluster admin toolThird party cluster admin agent
Sybase TechWave 2004
ASE Cluster Management Clusters are contained within the clusters folder.Operations are organized by task.Servers are color coded with status.
Sybase TechWave 2004
ASE Cluster Management Cluster Summary
Sybase TechWave 2004
Upgrade to Shared Disk Cluster
Upgrade SMP Installation to Shared Disk ClusterUpgrade existing installation as in SMP serverConfigure cluster and add additional servers to the cluster
Upgrade Active-Passive Installation to Shared Disk ClusterDisable and de-configure Active-Passive ModeUpgrade existing installation as in SMP serverConfigure cluster and add additional servers to the cluster
Upgrade Active-Active Installation to Shared Disk ClusterDisable and de-configure Active-Active CompanionshipUpgrade one installation as in SMP serverDump and load databases from the other installation (if needed)Configure cluster and add additional servers to the cluster
Sybase TechWave 2004
Shared Disk Cluster Feature Single System PresentationTopicsCluster Configuration FileInterfaces FileServer NameASE Configuration ParametersStored ProceduresCluster Database AdministrationDDL and DMLRPC and DTM HandlingError LogTempdbTrace Flag HandlingLoad BalancingConnection Failover
Sybase TechWave 2004
Single System Presentation Cluster configuration fileFormat
quorum Interfaces_file < interfaces file>config_file Traceflags < trace flags separated by space>interconnect primary server server server interconnect secondary server server server
Sybase TechWave 2004
Single System Presentation Cluster configuration file (cont.)Example
cluster1quorum /dev/rdsk/c1d2s4interfaces_file /opt/sybase/interfacesconfig_file /opt/sybase/server.cfgtraceflags 3623 3605interconnect primary udpserver 1 S1 node1 24040 24051server 2 S2 node2 24040 24051server 3 S3 node3 24000 24011interconnect secondary udpserver 1 S1 node1 24060 24071server 2 S2 node2 24060 24071server 3 S3 node3 24060 24071
Sybase TechWave 2004
Single System Presentation Interfaces fileLocation must be specified in the cluster configuration file
The -i option will not be allowed in a cluster environment
Each instance must have a server entry in the interfaces file
ASE instance on each node will use the server name as it does today in a SMP environment
Sybase TechWave 2004
Single System Presentation Interfaces file (cont.)The recommended way on the client side
specify a cluster name as server name entry in the interfaces file
add query lines for all the instances in a cluster
These provide transparency on the client side in terms of not having to know which instance it is communicating to
Continuing from the cluster configuration file example, the recommended way would be
Sybase TechWave 2004
Single System Presentation Interfaces file (cont.)# Will be used by server S1S1master tcp ether node1 4048query tcp ether node1 4048# Will be used by server S2S2master tcp ether node2 4048query tcp ether node2 4048# Will be used by server S3S3master tcp ether node3 4048query tcp ether node3 4048# Will be used by the clients when connecting to the clustercluster1query tcp ether node1 4048query tcp ether node2 4048 query tcp ether node3 4048
Sybase TechWave 2004
Single System Presentation Server name@@servername Cluster vs SMP
global variableSMPCluster@@servernameServer nameCluster name@@instancenameNAInstance name
Sybase TechWave 2004
Single System Presentation Config parametersAll the instances will use the same server config file
Location of the file must be specified in the cluster config file
The -c option will not be allowed
Most of the config parameters will apply to every instanceFor example, lets assume a 2 node cluster and the config parameter number of locks is set to 1000, then both the instances comes up with 1000 locks each
Sybase TechWave 2004
Single System Presentation Config parameters (cont.)Few config parameters will be instance specific
New config block will be defined in the config file to override the global values on a per-instance basis
Example: .tcp no delay = 1.max online engines = 10..[Instance 1]max online engines = 5[Instance 2]tcp no delay = 0
Sybase TechWave 2004
Single System Presentation Config parameters (cont.)Config parameters that are currently dynamic will continue to remain as is.
2 types of static config parameters One that requires just an instance rebootExample: max online engines
The other which requires a cluster rebootExample: enable java
Sybase TechWave 2004
Single System Presentation Database administration
Dump and Load
one backup server for the entire cluster
Space and Log Management
Single set of database devices and single log as in SMP server (minimal administration impact)
Sybase TechWave 2004
Single System Presentation
No change to existing DDL and DML
Some stored procedures can have a cluster-wide scopeExample: sp_who
New set option will be added to indicate the desired scope cluster wide or instance specificProvides ability to control the scope of the execution of the applicable stored procedures
Sybase TechWave 2004
Single System Presentation RPC and DTM handlingWithin a cluster, node-to-node RPCs will not be supported
Transparent handling of Distributed transactions and Replication
Sybase TechWave 2004
Single System Presentation Error log and tempdbEach instance will use an instance specific errorlog
Each instance can be configured to use an instance specific tempdb
Worktable and #temp tables will be created in this instance specific tempdb
The system wide tempdb will be shared across all the nodes
This will be used for persistent tempdb tables
Sybase TechWave 2004
Single System Presentation Tempdb (cont)Instance specific tempdb is not recovered on a failover
This will imply that #temp tables will be lost on a failover
Persistent temp tables will be logged and recovered in the event of a failover and will be available on all instances
The overall architecture would look like:
Sybase TechWave 2004
Single System Presentation Tempdb (cont)Instance 1 specific tempdbSystem tempdb (shared by instance 1 and instance 2)Instance 2 specific tempdb
Sybase TechWave 2004
Single System Presentation Trace flagsTraceflags that are needed across all the instances need to be specified in the cluster config file
The usage of -T option will only be applicable to the instance using it, i.e, this traceflag will not be set on other instances
dbcc traceon() will be instance specific
A new dbcc option will be introduced to propagate a traceflag among all the instances in a cluster
Sybase TechWave 2004
Single System Presentation Load balancingCritical aspects on load balancing:
Connection re-direction
Client side load balancing
Sybase TechWave 2004
Workload Management Connection re-directionBased on workload,Client connecting to an available instance be redirected to a different available instanceTransparent to the applicationNew OCS capability (enabled by default) for clients that link with the new OCS libraries
Sybase TechWave 2004
Workload Management Client-side load balancingClient randomly picks a query line from the list of query lines. Example:cluster1query tcp ether node1 4048query tcp ether node2 4048 query tcp ether node3 4048query tcp ether node4 4048
Follows Round Robin once a query line is picked Example: Once node 2 is picked, client will try node 2, followed by node 3 and node 4 wrapping up with node 1 Option will be turned off by defaultCan be enabled in the OCS config file without any application change
Sybase TechWave 2004
Single System Presentation - Connection failoverConnection failover
HA Aware Client applications will continue to get the same connection failover behavior
Sybase TechWave 2004
AgendaASE HA ArchitectureShared Disk Cluster Feature OverviewShared Disk Cluster Architecture OverviewWhat Benefits The Customers Would See?Q&A
Sybase TechWave 2004
Shared Disk Cluster ArchitectureAn Instance of Clustered ServerASE KernelData ServiceCluster Lock ManagementBuffer Cache CoherencyObject CoherencyCluster Space / ThresholdCluster Logging RecoveryCluster Membership ServiceCluster Event ServiceReliable Cluster InterconnectSingle System PresentationInterconnect I/O AbstractionUDPTCPVISDPBasis I/O and Platform AbstractionOperating System / Device Drivers
Sybase TechWave 2004
Cluster Infrastructure InterconnectProvides an abstraction layer on top of the cluster platform-specific private interconnects
Supports redundant cluster interconnects with automatic failover in the event of a link failureRedundant cluster interconnects can be used for load balancing and high availability
Supports reliable reception, OOB, multicast and remote DMA
Protocol supported: UDP
Private interconnects should be exclusively reserved.
Sybase TechWave 2004
Cluster Infrastructure Interconnect (cont.)
Fully meshed interconnection modelDedicated channels for cache coherency, lock management, cluster membership messages etc
Approaches to scale performanceRedundant interconnect for load balancing and availabilityHighly optimized interprocess communication for scalability
Sybase TechWave 2004
Cluster Infrastructure Fully meshed interconnectPseudo Connection Oriented Fully Meshed Structure
dedicated channels between any pair of nodes
Sybase TechWave 2004
Cluster Infrastructure Membership services
In-built heartbeat monitoring over the interconnect - Monitors the health of the clustered servers
Quorum disk for membership arbitration.
Maintains cluster view and cluster server statesstart, init, up, quiescence, and down.
Publishes cluster server join, shutdown and failover events
Supports dynamic configuration of nodes.
Sybase TechWave 2004
Cluster Lock ManagementCluster lock manager provides distributed locking servicesModeled after VAX cluster locking semanticsSupports dynamic hot lock re-mastering to reduce inter node messaging
Fully distributed lock management Dynamic re-distribution of global lock space management during instance join, shutdown and failure
Online failover recovery supportFreeze in-doubt resources at instance failure for online failover recovery Lock table replication based on high availability levelGraceful handling of multiple instance failures
Sybase TechWave 2004
Cluster Lock Management (cont.)Integrated cluster locking and cache coherency to enable fast cache-to-cache transfer via cluster interconnectThe following diagram shows an example of S1 requesting for shared access to a resource mastered by S2 and owned by S3.
RequesterOwnerLock MasterS1S2S33. Data transfer2 Lock granted1. Lock request2 Transfer request
Sybase TechWave 2004
Cluster Lock Management (cont.)
The following diagram shows an example of S1 requesting for exclusive access to a resource mastered by S2 and owned by S3.
RequesterOwnerLock MasterS1S2S33 Data transfer4. Lock granted1. Lock request2. Downgrade/Transfer request3 Lock Downgraded
Sybase TechWave 2004
Cluster Lock Management Distributed deadlock detectionLocal deadlock detection prior to initiation of remote deadlock detection
Use optimized distributed deadlock detection algorithmAll instances do full distributed deadlock detection Minimized remote messaging and resource usage
Sybase TechWave 2004
Cluster Cache Management
Cluster cache manager provides global cache coherency
Cluster cache manager tightly integrated with the cluster lock manager and interconnect module.
Sharing the disk means cache in a cluster provides the ability to extend memory beyond a single machineAny node in the cluster can act as an inexpensive L2 cache to other nodes.
Optimized solution to increase the read-write concurrency for row locked tables
Sybase TechWave 2004
Object Metadata Cache Coherency Object Cache Coherency ManagerObjects are kept coherent across the clusterAllows for concurrent DDL operationsSelective synchronization scheme for optimal concurrency
Sybase TechWave 2004
Cluster RecoveryBoot Recovery
Occurs in the coordinator node
Database offline till recovery is complete
Sybase TechWave 2004
Online Failover RecoveryHappens immediately after CLM recovery
Controlled by coordinator node
Database recovery is online - unaffected portions of the database are accessible while recovery executes in parallel with other tasks.
Normal flow of events tasks executing in other nodes can continue
Tasks block when they try to access a resource that needs to be recovered.
Sybase TechWave 2004
Online Failover Recovery In-doubt ResourcesAll update locks on metadata and pages that were held by the crashed node becomes inaccessible after the crash.
CLM moves the controlling lock for that resource to an in-doubt state
The in-doubt state is cleared by recovery, after the resource is recovered.
After that, resource becomes available
Only UPDATE locks matter.
Sybase TechWave 2004
Online Failover Recovery Block till Recovery CompletionAny task that asks for an in-doubt lock will be blocked till recovery for that locks controlled resource is complete
Access patterns are a factor; for e.g.If table A is updated in instance A and table B is updated in instance B, access to table B in instance B can continue unhindered even if instance A crashes.
Sybase TechWave 2004
Online Failover Recovery Resource RecoveryResources are made available as and when recovery for that resource is done, subject to transactional restrictions.
Database access increases from partial access to complete access as recovery progresses. Blocked tasks are made ready to run when the lock becomes available.
Sybase TechWave 2004
LoggingLog record size increased impacts LCT requirements
Hotspot elimination - decentralized database timestamp
Sybase TechWave 2004
AgendaASE HA ArchitectureShared Disk Cluster Feature OverviewShared Disk Cluster Architecture OverviewWhat Benefits The Customers Would See?Q&A
Sybase TechWave 2004
What Benefits the Customers would see?
Continuous Availability with Instantaneous FailoverMulti Node Availability. Availability increases as number of nodes increases.Dynamic Load BalancingWorkload based load balancing among all clustered servers for new and failover connectionsIncremental Node Growth and ShrinkIncrease the capacity by adding additional SMP boxesScalabilityScalability can be achieved with cluster-aware application partitioning using logical clustersSingle System ImageAll servers share Data. Ease of administration and application deployment.Platform independent cluster infrastructure
Sybase TechWave 2004
AgendaASE HA ArchitectureShared Disk Cluster Feature OverviewShared Disk Cluster Architecture OverviewWhat Benefits The Customers Would See?Q&A
Sybase TechWave 2004
As you will see throughout this week, Sybase creates solutions that guide your data from data center to device through the Enterprise - Unwired. To build an Unwired Enterprise, you need to do three things:
Manage enterprise information make that information useful, relevant and valuable
Unwire the information across the enterprise ensuring fast, reliable access in real-time
and unwire the people enabling them to work wherever they need to, securing data across networks, systems and devices.
Sybase Analyst Event August 2003The Unwired Enterprise is an information management strategy.
In this session, well be discussing this (POINT OUT YOUR PRODUCT OR SOLUTION AREA) section of Sybases Unwired Enterprise solution.
Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003
Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003