Adaptive Server Enterprise Shared Disk Cluster - A Technical ...

ASE121: Shared Disk Cluster Technical Review Linda LinSenior [email protected]

Ganesan Gopal Senior [email protected]

August 15-19, 2004

Sybase TechWave 2004

The Enterprise. Unwired.


The Enterprise. Unwired.

UnwirePeople

UnwireInformation

ManageInformationSybase WorkspaceIndustry and Cross Platform SolutionsAdaptive Server EnterpriseAdaptive Server AnywhereSybase IQDynamic ArchiveDynamic ODSReplication ServerOpenSwitchMirror ActivatorPowerDesignerConnectivity OptionsEAServerIndustry Warehouse StudioUnwired AcceleratorUnwired OrchestratorUnwired ToolkitEnterprise PortalReal Time Data ServicesSQL Anywhere StudioM-Business AnywherePylon Family (Mobile Email)Mobile SalesXcelleNet Frontline SolutionsPocketBuilderPowerBuilder FamilyAvantGo


AgendaASE HA ArchitectureShared Disk Cluster Feature OverviewShared Disk Cluster Architecture OverviewWhat Benefits The Customers Would See?Q&A


ASE HA ArchitectureCurrentStandby (active-passive): ASE on standby node is not started until primary node failure

Distributed workloads (active-active)12.0 implementation of companion server architecture

Work in progressConcurrent access (shared disk cluster)


Active-Passive

Disk StorageSANHA SystemASE DB QuorumTwo nodes failover cluster with single database store

ASE on primary node is active

ASE on standby node is inactive before primary failover


Active-Active Symmetric configurationSANHA SystemASE #1ASE #2ASE #1ASE #2Two node failover cluster with two database stores

Both ASE servers are active servicing disjoint applications

Both ASE servers are companions of each other for availability

Disk Storage


Motivations for Shared Disk ClusterTCOEmerging Low Cost Intel/Linux combo - 4-8 nodes with high speed interconnectAbility to consolidate multiple apps for effective utilization of unused H/W capacityEase of administration

Continuous AvailabilityUninterrupted operation in case a component fails.

High PerformanceIncrease the capacity by adding additional SMP Boxes with no restructuring of data.

Single System PresentationSingle Server appearance to the applicationApplications run at multiple nodes with Shared Data

Technology Trends


Shared Disk Cluster


Shared Disk Cluster - An instance of server failover


Shared Disk Cluster HighlightsShared Disk Cluster (Concurrent Access)Single database storeSingle system presentationAll servers provide services at all timeAll servers can access all DBs directlyIndependent of platform-specific clustering serviceIncremental node growth and shrinkCluster-aware application partitioningWorkload based load balancingInstantaneous server failoverWorkload is re-distributed at a server failure


Shared Disk Cluster


Shared Disk Cluster Feature Logical ClustersSimilar to the Logical Process Management (LPM) feature thats currently available in ASERules can be specified at a instance level rather than at the engine level (LPM will be supported as well)

DBAs can setup application, users, etc binding to the various instances.


Logical Cluster with Application Partitioning

SANS1S2S3S4

Client connections are transparently redirected to a logical cluster

Cluster-aware application partitioning improves scalability

Easy to implement Service Level AgreementPrivate InterconnectQuorumSales ApplicationFinance ApplicationFinance ClusterSales ClusterS1S2S3S4Shared Disk StorageCluster DBFinance DBSales DBSharedFinance Logical ClusterSales Logical Cluster


Logical Clusters for Disjoint Applications

SAN

Single system presentation with transparent connection redirection to logical cluster

Cluster-aware application partitioning improves scalability

Frequently accessed data is available locally and infrequently accessed data is shared by all serversQuorumSales ApplicationFinance ApplicationPrivate InterconnectFinance Logical ClusterSales Logical ClusterS1Shared Disk StorageCluster DBFinance DBSales DBShared


Logical Cluster with Incremental Node Growth

SAN

Add more low cost SMP boxes to logical cluster as demands grow - TCO

New connections are directed to the least loaded server within a logical clusterQuorumSales ApplicationFinance ApplicationSales ApplicationPrivate InterconnectShared Disk StorageCluster DBFinance DBSales DBSharedFinance Logical ClusterSales Logical Cluster


Logical Cluster with Server Failover

SAN

Continuous Availability with Instantaneous Failover

Failover connections are redirected to another server in the same logical clusterPrivate InterconnectQuorumSales ApplicationFinance ApplicationSales ApplicationShared Disk StorageCluster DBFinance DBSales DBSharedSales Logical ClusterFinance Logical Cluster


Logical Cluster Failover

SAN

Continuous Availability with Instantaneous -Failover across logical cluster

When a logical cluster fails, failover connections are redirected to an alternate logical cluster

Databases are available as long as one server is available

Private InterconnectQuorumSales ApplicationFinance ApplicationSales ApplicationS3Sales Logical ClusterShared Disk StorageCluster DBFinance DBSales DBSharedFinance Logical Cluster


Logical Cluster Recovery

SAN

After a logical cluster recovers, connections are redirected to the logical cluster.

Private InterconnectQuorumSales ClusterSales ApplicationFinance ApplicationSales ApplicationS3Sales Logical ClusterShared Disk StorageCluster DBFinance DBSales DBSharedFinance Logical Cluster


Shared Disk Cluster Installation and Administration

Topics

Platform Recommendations

Installation

Configuration and Administration

Upgrade


Platform RecommendationsHomogeneous operating systems Linux or Solaris

Similar HW configuration

Gigabit Ethernet as private interconnects

Raw devices on SAN connected storage for globally accessible databases and quorum disk devicesOther options: clustered file systems or virtualized storage with volume managers

Single installation of $SYBASE on globally mounted file system


Cluster InstallationProcedure similar to that of a standalone SMP

Done from a single node in the cluster

Installation is on globally accessible raw devices and file systems

All instances in the cluster share single installation of databases,server binaries and scriptsconfiguration files.


Cluster Administration sybclusterClient side cluster administration tool Command line interfaceJava-based program supporting multiple platforms

Communicates with Unified Agent Framework (UAF) and ASEAgent on server side

UAF and ASEAgent are configured through UAF Web Console.


Cluster Administration sybcluster architecture sybclusterServer Node 1

Server Node 2


Cluster Administration sybcluster usageExample:sybcluster addcluster cluster ..sybcluster addserver server -cluster ..sybcluster startcluster cluster ..sybcluster startserver server cluster ..sybcluster stopserver server -cluster ..sybcluster stopcluster cluster ..sybcluster dropserver server -cluster sybcluster dropcluster cluster ..sybcluster serverstatus server -cluster ..sybcluster clusterstatus cluster ..

Command line input also takes user authentication and UAF agent discovery information


Cluster Administration Integration with third party cluster admin tool start, stop, monitor scripts binaryThird party cluster admin agentstart, stop, monitor scripts binaryServer Node 1

Server Node 2

Third party cluster admin toolThird party cluster admin agent


ASE Cluster Management Clusters are contained within the clusters folder.Operations are organized by task.Servers are color coded with status.


ASE Cluster Management Cluster Summary


Upgrade to Shared Disk Cluster

Upgrade SMP Installation to Shared Disk ClusterUpgrade existing installation as in SMP serverConfigure cluster and add additional servers to the cluster

Upgrade Active-Passive Installation to Shared Disk ClusterDisable and de-configure Active-Passive ModeUpgrade existing installation as in SMP serverConfigure cluster and add additional servers to the cluster

Upgrade Active-Active Installation to Shared Disk ClusterDisable and de-configure Active-Active CompanionshipUpgrade one installation as in SMP serverDump and load databases from the other installation (if needed)Configure cluster and add additional servers to the cluster


Shared Disk Cluster Feature Single System PresentationTopicsCluster Configuration FileInterfaces FileServer NameASE Configuration ParametersStored ProceduresCluster Database AdministrationDDL and DMLRPC and DTM HandlingError LogTempdbTrace Flag HandlingLoad BalancingConnection Failover


Single System Presentation Cluster configuration fileFormat

quorum Interfaces_file < interfaces file>config_file Traceflags < trace flags separated by space>interconnect primary server server server interconnect secondary server server server


Single System Presentation Cluster configuration file (cont.)Example

cluster1quorum /dev/rdsk/c1d2s4interfaces_file /opt/sybase/interfacesconfig_file /opt/sybase/server.cfgtraceflags 3623 3605interconnect primary udpserver 1 S1 node1 24040 24051server 2 S2 node2 24040 24051server 3 S3 node3 24000 24011interconnect secondary udpserver 1 S1 node1 24060 24071server 2 S2 node2 24060 24071server 3 S3 node3 24060 24071


Single System Presentation Interfaces fileLocation must be specified in the cluster configuration file

The -i option will not be allowed in a cluster environment

Each instance must have a server entry in the interfaces file

ASE instance on each node will use the server name as it does today in a SMP environment


Single System Presentation Interfaces file (cont.)The recommended way on the client side

specify a cluster name as server name entry in the interfaces file

add query lines for all the instances in a cluster

These provide transparency on the client side in terms of not having to know which instance it is communicating to

Continuing from the cluster configuration file example, the recommended way would be


Single System Presentation Interfaces file (cont.)# Will be used by server S1S1master tcp ether node1 4048query tcp ether node1 4048# Will be used by server S2S2master tcp ether node2 4048query tcp ether node2 4048# Will be used by server S3S3master tcp ether node3 4048query tcp ether node3 4048# Will be used by the clients when connecting to the clustercluster1query tcp ether node1 4048query tcp ether node2 4048 query tcp ether node3 4048


Single System Presentation Server name@@servername Cluster vs SMP

global variableSMPCluster@@servernameServer nameCluster name@@instancenameNAInstance name


Single System Presentation Config parametersAll the instances will use the same server config file

Location of the file must be specified in the cluster config file

The -c option will not be allowed

Most of the config parameters will apply to every instanceFor example, lets assume a 2 node cluster and the config parameter number of locks is set to 1000, then both the instances comes up with 1000 locks each


Single System Presentation Config parameters (cont.)Few config parameters will be instance specific

New config block will be defined in the config file to override the global values on a per-instance basis

Example: .tcp no delay = 1.max online engines = 10..[Instance 1]max online engines = 5[Instance 2]tcp no delay = 0


Single System Presentation Config parameters (cont.)Config parameters that are currently dynamic will continue to remain as is.

2 types of static config parameters One that requires just an instance rebootExample: max online engines

The other which requires a cluster rebootExample: enable java


Single System Presentation Database administration

Dump and Load

one backup server for the entire cluster

Space and Log Management

Single set of database devices and single log as in SMP server (minimal administration impact)


Single System Presentation

No change to existing DDL and DML

Some stored procedures can have a cluster-wide scopeExample: sp_who

New set option will be added to indicate the desired scope cluster wide or instance specificProvides ability to control the scope of the execution of the applicable stored procedures


Single System Presentation RPC and DTM handlingWithin a cluster, node-to-node RPCs will not be supported

Transparent handling of Distributed transactions and Replication


Single System Presentation Error log and tempdbEach instance will use an instance specific errorlog

Each instance can be configured to use an instance specific tempdb

Worktable and #temp tables will be created in this instance specific tempdb

The system wide tempdb will be shared across all the nodes

This will be used for persistent tempdb tables


Single System Presentation Tempdb (cont)Instance specific tempdb is not recovered on a failover

This will imply that #temp tables will be lost on a failover

Persistent temp tables will be logged and recovered in the event of a failover and will be available on all instances

The overall architecture would look like:


Single System Presentation Tempdb (cont)Instance 1 specific tempdbSystem tempdb (shared by instance 1 and instance 2)Instance 2 specific tempdb


Single System Presentation Trace flagsTraceflags that are needed across all the instances need to be specified in the cluster config file

The usage of -T option will only be applicable to the instance using it, i.e, this traceflag will not be set on other instances

dbcc traceon() will be instance specific

A new dbcc option will be introduced to propagate a traceflag among all the instances in a cluster


Single System Presentation Load balancingCritical aspects on load balancing:

Connection re-direction

Client side load balancing


Workload Management Connection re-directionBased on workload,Client connecting to an available instance be redirected to a different available instanceTransparent to the applicationNew OCS capability (enabled by default) for clients that link with the new OCS libraries


Workload Management Client-side load balancingClient randomly picks a query line from the list of query lines. Example:cluster1query tcp ether node1 4048query tcp ether node2 4048 query tcp ether node3 4048query tcp ether node4 4048

Follows Round Robin once a query line is picked Example: Once node 2 is picked, client will try node 2, followed by node 3 and node 4 wrapping up with node 1 Option will be turned off by defaultCan be enabled in the OCS config file without any application change


Single System Presentation - Connection failoverConnection failover

HA Aware Client applications will continue to get the same connection failover behavior


Shared Disk Cluster ArchitectureAn Instance of Clustered ServerASE KernelData ServiceCluster Lock ManagementBuffer Cache CoherencyObject CoherencyCluster Space / ThresholdCluster Logging RecoveryCluster Membership ServiceCluster Event ServiceReliable Cluster InterconnectSingle System PresentationInterconnect I/O AbstractionUDPTCPVISDPBasis I/O and Platform AbstractionOperating System / Device Drivers


Cluster Infrastructure InterconnectProvides an abstraction layer on top of the cluster platform-specific private interconnects

Supports redundant cluster interconnects with automatic failover in the event of a link failureRedundant cluster interconnects can be used for load balancing and high availability

Supports reliable reception, OOB, multicast and remote DMA

Protocol supported: UDP

Private interconnects should be exclusively reserved.


Cluster Infrastructure Interconnect (cont.)

Fully meshed interconnection modelDedicated channels for cache coherency, lock management, cluster membership messages etc

Approaches to scale performanceRedundant interconnect for load balancing and availabilityHighly optimized interprocess communication for scalability


Cluster Infrastructure Fully meshed interconnectPseudo Connection Oriented Fully Meshed Structure

dedicated channels between any pair of nodes


Cluster Infrastructure Membership services

In-built heartbeat monitoring over the interconnect - Monitors the health of the clustered servers

Quorum disk for membership arbitration.

Maintains cluster view and cluster server statesstart, init, up, quiescence, and down.

Publishes cluster server join, shutdown and failover events

Supports dynamic configuration of nodes.


Cluster Lock ManagementCluster lock manager provides distributed locking servicesModeled after VAX cluster locking semanticsSupports dynamic hot lock re-mastering to reduce inter node messaging

Fully distributed lock management Dynamic re-distribution of global lock space management during instance join, shutdown and failure

Online failover recovery supportFreeze in-doubt resources at instance failure for online failover recovery Lock table replication based on high availability levelGraceful handling of multiple instance failures


Cluster Lock Management (cont.)Integrated cluster locking and cache coherency to enable fast cache-to-cache transfer via cluster interconnectThe following diagram shows an example of S1 requesting for shared access to a resource mastered by S2 and owned by S3.

RequesterOwnerLock MasterS1S2S33. Data transfer2 Lock granted1. Lock request2 Transfer request


Cluster Lock Management (cont.)

The following diagram shows an example of S1 requesting for exclusive access to a resource mastered by S2 and owned by S3.

RequesterOwnerLock MasterS1S2S33 Data transfer4. Lock granted1. Lock request2. Downgrade/Transfer request3 Lock Downgraded


Cluster Lock Management Distributed deadlock detectionLocal deadlock detection prior to initiation of remote deadlock detection

Use optimized distributed deadlock detection algorithmAll instances do full distributed deadlock detection Minimized remote messaging and resource usage


Cluster Cache Management

Cluster cache manager provides global cache coherency

Cluster cache manager tightly integrated with the cluster lock manager and interconnect module.

Sharing the disk means cache in a cluster provides the ability to extend memory beyond a single machineAny node in the cluster can act as an inexpensive L2 cache to other nodes.

Optimized solution to increase the read-write concurrency for row locked tables


Object Metadata Cache Coherency Object Cache Coherency ManagerObjects are kept coherent across the clusterAllows for concurrent DDL operationsSelective synchronization scheme for optimal concurrency


Cluster RecoveryBoot Recovery

Occurs in the coordinator node

Database offline till recovery is complete


Online Failover RecoveryHappens immediately after CLM recovery

Controlled by coordinator node

Database recovery is online - unaffected portions of the database are accessible while recovery executes in parallel with other tasks.

Normal flow of events tasks executing in other nodes can continue

Tasks block when they try to access a resource that needs to be recovered.


Online Failover Recovery In-doubt ResourcesAll update locks on metadata and pages that were held by the crashed node becomes inaccessible after the crash.

CLM moves the controlling lock for that resource to an in-doubt state

The in-doubt state is cleared by recovery, after the resource is recovered.

After that, resource becomes available

Only UPDATE locks matter.


Online Failover Recovery Block till Recovery CompletionAny task that asks for an in-doubt lock will be blocked till recovery for that locks controlled resource is complete

Access patterns are a factor; for e.g.If table A is updated in instance A and table B is updated in instance B, access to table B in instance B can continue unhindered even if instance A crashes.


Online Failover Recovery Resource RecoveryResources are made available as and when recovery for that resource is done, subject to transactional restrictions.

Database access increases from partial access to complete access as recovery progresses. Blocked tasks are made ready to run when the lock becomes available.


LoggingLog record size increased impacts LCT requirements

Hotspot elimination - decentralized database timestamp


What Benefits the Customers would see?

Continuous Availability with Instantaneous FailoverMulti Node Availability. Availability increases as number of nodes increases.Dynamic Load BalancingWorkload based load balancing among all clustered servers for new and failover connectionsIncremental Node Growth and ShrinkIncrease the capacity by adding additional SMP boxesScalabilityScalability can be achieved with cluster-aware application partitioning using logical clustersSingle System ImageAll servers share Data. Ease of administration and application deployment.Platform independent cluster infrastructure




As you will see throughout this week, Sybase creates solutions that guide your data from data center to device through the Enterprise - Unwired. To build an Unwired Enterprise, you need to do three things:

Manage enterprise information make that information useful, relevant and valuable

Unwire the information across the enterprise ensuring fast, reliable access in real-time

and unwire the people enabling them to work wherever they need to, securing data across networks, systems and devices.

Sybase Analyst Event August 2003The Unwired Enterprise is an information management strategy.

In this session, well be discussing this (POINT OUT YOUR PRODUCT OR SOLUTION AREA) section of Sybases Unwired Enterprise solution.

Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003

Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003Sybase Analyst Event August 2003

Adaptive Server Enterprise Shared Disk Cluster - A Technical ...

Documents

add additional servers

shared disk clusterdisable

smp serverconfigure cluster

logical cluster

config parameters

temp tables

cache coherency

query line