Mysql Nd Load Balancing

Ulf Wendel, Oracle

PECL/mysqlnd_ms 1.4

MySQL? Load? Clustering! Balancing!

The speaker says...

Clustering databases is a main stream technology. MySQL users can choose between a variety of clustering options. The opions range from asynchronous primary copy to synchronous update anywhere.

Let's quickly recap what MySQL has to offer.

● Asynchronous read scale-outMaster: writes, Slaves: readsVarious topologies supportedOther use cases: backup, HA, OLTP/warehousing

MySQL Replication

Client

Writes Reads

Master SlaveSlave Slave

New:Global Transaction

Identifier

The speaker says...

MySQL Replication is used for read scale-out. Writes are send to a MySQL master, reads are send to a MySQL slave. The master sends updates to the slaves in an asynchronous way. Slaves may not have the latest changes. This kind of replication is also called primary copy. Where goes the update: to the primary. When/how do updates happen: lazy, asynchronous.

MySQL Replication can also be used for:• Backup – perfom blocking backup on slave• High Availablility – for example, remote slaves• Warehousing – OLTP reporting on the slaves

● Real-time transactional write scale-out99,999% availabilityAuto sharding, shared nothing architectureWeb use cases: session server, tele communication

MySQL Cluster

Client

SQL Node SQL Node

Cluster Data Node Cluster Data Node Cluster Data Node

1.2 Billion UPDATEs/min

The speaker says...

MySQL Cluster is used for real-time transactional read and write scale-out. Data is auto-partitioned on cluster data nodes resulting in 99.999% availability. The cluster is accessed through MySQL Application Cluster Nodes. The most basic MySQL Application Cluster Node is a SQL Node – a MySQL Server (mysqld).

If MySQL Cluster is new to you, think of MySQL Cluster as a storage engine for the network. Like InnoDB and MyISAM are storage engines for disks. This kind of replication is also called eager (when?) update anywhere (where?).

● Tasks:Choose servers to execute statementLoad balance within candidatesMaintain connection poolAutomatic (client) fail over for High Availability

Application using any DB cluster

MySQL Server 1 MySQL Server 2 MySQL Server n

Client

The speaker says...

All PHP applications talking to a cluster of MySQL database servers are facing the same tasks.

Replacing a single database with a database cluster means changing the 1:1 relation between the application and the database into a 1:n relation. 1:n puts an additional task on the application: find the right n, find the right server to talk to.

The plugin for all of you

WordPress, Drupal, Symfony, Oxid, ZendFramework, ...

mysql, mysqli, PDO_MYSQL

mysqlnd

Balancing Connection PoolingStatement Redirection

PECL/mysqlnd_ms plugin

Failover

MySQL Server 1 MySQL Server 2 MySQL Server n

The speaker says...

PECL/mysqlnd_ms is a plugin for the MySQL native driver for PHP (mysqlnd) library. The mysqlnd library is part of PHP as of version 5.3. As of 5.4 mysqlnd is a default.

All three PHP MySQL APIs (mysql, mysqli, PDO_MySQL) can be compiled to use the mysqlnd library, thus all existing PHP MySQL application are supported by the plugin.

From an applications point of view a mysqlnd plugin can act as a transparent proxy. Depending on the use case, no code changes are needed. PECL/mysqlnd_ms can be a drop-in solution.

● Go for it! But don't miss our goodies...

Wait - what's wrong with this?

class DBRepl { private static $master_list = array(...); private static $master = NULL; private static $slave_list = array(...); private static $slave = NULL; public static function WriteConn() { if (self::$master) return self::$master; /* pick and initialize ...*/ } public static function ReadConn() { if (self::$slave) return self::$slave; /* pick and initialize ...*/ }}

The speaker says...

A client-side approach to the problem is promising, if client applications can be changed. It is no drop-in solution. The load balancer is part of the client application. It scales by client and it fails by client. There is no single point of failure as it is the case with a proxy based approach. No additional latency occurs like with a proxy. Load balancing can be adaptive for good resource usage. DBMS node failures do not block clients, fail over is possible.

PECL/mysqlnd_ms has all this, is semi-transparent and offers many more features!

● Anything missing?All PHP MySQL APIs, semi-transparent, drop-inRead write splitting, transaction awareLoad balancing, fail over (optional: automatic)Service levels: consistency, slave lag limit, trottlingOptional, automatic caching of read resultsUser can control all automatic actions75+ pages documentation: hands on and referenceStable, more test code than C codeSome 10 bug reports since 1.0

PECL/mysqlnd_ms 1.4

The speaker says...

No worry, it is not complicated. All the magic is beneath the hood, in the C code of PECL/mysqlnd_ms. The user interface is short and comprehensive.

This is all what PECL/mysqlnd_ms is about: make using any kind of MySQL Cluster easier! Let the load balancer do the work.

Provide a cluster-aware API. Clustering is mainstream. We need new APIs.

● Default: mysqlnd_ms.disable_rw_split = 0read-only: statement begins with SELECTread-only: statement begins with slave SQL hintcustom: callback picks slave for statement

Read/Write splitting

Client

Writes Reads (SELECT, SQL hint)


The speaker says...

Read/Write splitting must be done for primary copy clusters, like MySQL Replication. Clients must direct all writes to the primary (master). PECL/mysqlnd_ms automatically directs statements that begin with SELECT or a certain SQL hint to one of the configured secondaries (slaves).

An update anywhere cluster, such as MySQL Cluster, allows writes to be performed on all nodes. No read write splitting is needed. Thus, you can disable it on a global level using the configuration directive mysqlnd_ms.disable_rw_split.

● No switch allowed during a transactionAPI calls controlling transactions monitored (5.4+)SQL not checked, protocol does not annnounce stateSolution: use API (5.4+) or SQL hints!

Redirection and transactions

Client

UPDATE SET col=@my BEGIN; SELECT @my=1;


The speaker says...

Statement redirection for load balancing bares a pitfall: connection state. Session variables and transactions are part of the state. A load balancer must not switch servers in the middle of a transaction. When using primary copy, all transactions shall go to the primary (master) because it is unknown at the beginning of a transaction whether it includes updates.

The MySQL Client Server protocol does not hint whether a transaction is active. Load balancers must either parse SQL to catch BEGIN, monitor API calls or be hinted by the application. PECL/mysqlnd_ms monitors the API and understands hints.

● Use API call over SQL to control transactionsFor MySQL Replication, set: trx_stickiness = masterCheck master_on_write=1 or mysqlnd_ms_set_qos()

Transaction aware code

$mysqli = new mysqli("myapp", "username", "password", "database");

/* Disable autocommit, master used, no server switch allowed */$mysqli->autocommit(FALSE);/* ... */

if (!$mysqli->commit()) { die(sprintf("[%d] %s\n", $mysqli->errno, $mysqli->error));}

/* Transaction has ended, load balancing begins again */$mysqli->autocommit(TRUE);

The speaker says...

Use the API calls to control transactions. Give the database driver a chance to help you. Virtually any state change that you perform through API calls is handled by PECL/mysqlnd_ms. For example, if you change the charset with mysqli_set_charset(), the plugin makes sure that all opened connections use the same charset. We hide complexity, make cluster use easier.

Consider using master_on_write=1. This will keep all reads followed by the first write on the master. Handy, if you use asynchronous cluster and eventual consistency is not enough. Wait for quality-of-service, its getting better!

● PECL/mysqlnd_ms does not look worse, does it?

Is this your code?

$mysqli = DBRel::ReadConn();/* ... */$mysqli = DBRepl::WriteConn()$mysqli->autocommit(FALSE);/* ... */if (!$mysqli->commit()) { die(sprintf("[%d] %s\n", $mysqli->errno, $mysqli->error));}/* ... */$mysqli->autocommit(TRUE);$mysqli = DBRepl::ReadConn();

The speaker says...

The differences to the classic approach become obvious. The plugin code has less „hints“, the abstraction is less often informed of the need of a certain type of database.

The classic code is specifically written for a certain kind of cluster. Once you migrate the database from a lazy primary copy system (e.g. MySQL Replication) to an eager update anywhere solution (e.g. MySQL Cluster) you have additional calls in your code that are no more needed.

The plugin code needs nothing but a configuration file change to switch from MySQL Replication to MySQL Cluster.

● Built-in or user-defined through callbackRound robin, Random, Random once1.3 – kind of adaptive by setting limits for slave lag1.4 – nodes can be assigned a weight/probability

Load Balancing

Client

Writes Load Balancing of Reads


Weight = 2 Weight = 1 Weight = 1Weight = 1

The speaker says...

The best and default load balancing stategy is the one with the lowest impact on the connection state: random once. Minimize switches for the life-span of your PHP script. Pick a random slave during startup and use it for all reads. The life-span of PHP is the of a web request. It is short. Two web requests will end up on two random slaves.

The upcoming 1.4 release will allow you to assign a weight to a node. Nodes with a weight of two will get twice as many requests as nodes with a weight of one. This is not only great if your nodes have different computing power but also to prefer local nodes over remote ones by setting a higher weight.

● Caution: connection state changeDefault: disabled = raise exceptionOptional: silently failover master1.4 – remember failed, try more slaves before master

Pooling: connection failover

Client

Writes Reads


The speaker says...

Connection state changes are what speaks against automatic, silent fail over once a connection has been established and fails. If your application is written in a way that automatic reconnect upon failure is allowed, you can enable it. Otherwise, handle the error, do not close the connection but run the next query to trigger a new load balancing attempt! Failed nodes can be remembered for the duration of a request.

Automatic failover can either try to connect to the master or try to use another slave before failing over to the master. If using MySQL Cluster, the search logic still applies with the only difference that only masters are to be configured.

● Do not open connection before neededDelay open until statement executionReduce number of connections, reduce loadGreat for legacy apps – even without load balancing

Pooling: lazy connections

Client

Writes Reads


The speaker says...

By default, to reduce the connection pool size of every PHP web request, connections are not opened before executing a statement. Imagine you have 50 concurrent web requests, you configured four slaves and, you are using random once. If not using lazy connections, the plugin opens 50 x 4 = 200 slave connections but uses at max 50 of them (random once). Every connection occupies resources – ever seen a PHP application opening connections before using? There are many badly written ones...

Many connection state changing API calls are buffered and replayed when opening lazy to prevent pitfalls.

Database cluster as a service

● Don't bother about node selection – MS does!mysqlnd_ms_set_qos()Eventual, session and strong consistencyAllow only certain slave lag, enable caching

Quality of service

Client

Node NodeNode Node

PECL/mysqlnd_ms

The speaker says...

The basics (load balancing, failover) are solved. Abstraction can start. We can start to see the cluster as a service. The load balancer knows how to use the cluster to achieve a certain quality of service. The application formulates its service quality requirements, the load balancer takes appropriate action. For example, state the consistency level needed. Is reading stale data from a slave allowed (eventual consistency) and if so, is there any limit on freshness?

Having API calls to know the client demands is a great help for load balancer and cluster vendors!

● Nodes may or may not serve the latest copiesNodes may serve stale copies or not have copies at allDefault consistency level of MySQL Replication

Eventual consistency

MySQL Node

MySQL Node MySQL Node

GET X, NULL

SET X = 1

GET X, X = 0

The speaker says...

An eventual consistent node may or may not serve the latest copy. In fact, there is no promise that a particular copy is available from every node. Many systems that default to eventual consistency reach strong consistency over time. Eventually all nodes get synchronized. This model is similar to that of a cache.

Eventual consistency is good enoug for browsing product catalogs or other infrequently changing contents in areas where stale data is acceptable. It is the default consistency level with MySQL Replication. Why bother ?! Can't vendors provide proper solutions?

● There is no one-fits-all replication solutionThe dangers of replication and a solution (Jim Gray, Pat Helland, 1996): a ten-fold increase in nodes and traffic gives a thousand fold increase in deadlocks or reconciliationsCAP theorem (Eric Brewer, 2000): consistency, availability and paritition tolerance can't be achieved together… vendors are forced to offer compromises… vendors are faced with a highly complex problem… you are forced to know about consistency :-(

Clustering databases is complex

The speaker says...

I would love to stop talking about consistency. But replication has limits. Keep on pushing but know the limits. Every synchronous, update anywhere solution has a scalability limit. Partitioning (sharding) only shifts the limits and creates other issues (rebuilding aggregates). Until its solved...

Applications shall state their QoS/consistency needs:• To allow vendors to relax consistency for performance

reasons, but only if feasible, if allowed by the app• To allow load balancers to make an optimal node choice

= PECL/mysqlnd_ms 1.2+ build around GTIDs...

● Better than master_on_write = 1


/* read-write splitting: master used */

$mysqli->query("INSERT INTO orders(item) VALUES ('elePHPant')");

/* Request session consistency: read your writes */

mysqlnd_ms_set_qos($mysqli, MYSQLND_MS_QOS_CONSISTENCY_SESSION);

/* Plugin picks a node which has the changes, here: master */

$res = $mysqli->query("SELECT item FROM orders WHERE order_id = 1");

var_dump($res->fetch_assoc());

/* Back to eventual consistency: stale data allowed */

mysqlnd_ms_set_qos($mysqli, MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL);

/* Plugin picks any slave, stale data is allowed */

$res = $mysqli->query("SELECT item, price FROM specials");

Setting QoS for consistency

The speaker says...

An eager update anywhere replication system such as MySQL Cluster delivers strong consistency. Classic users of MySQL Replication achieve different level of consistency using the simple rules: eventual consistency – any slave, session and strong consistency – master only.Using the master for any level beyond eventual consistency has a negative impact on read scale out. MySQL 5.6 Global Transactions Identifiers help to achieve session consistency even when reading from slaves – reliable read your writes has become possible. PECL/mysqlnd_ms implements the node selection logic.

● Combination of server id and sequence numberEmulation: PECL/mysqlnd_ms 1.2+, MySQL ProxyBuilt-in: MySQL 5.6

Global transaction identifier

MySQL Master

Log 7, Pos 34, GTID M:1: UPDATE x=1Log 7, Pos 35, GTID M:2: UPDATE x=9

MySQL Slave 1 MySQL Slave 2

… , GTID M:1: UPDATE x=1… , GTID M:1: UPDATE x=1… , GTID M:2: UPDATE x=9

The speaker says...

A global transaction identifier is a cluster-wide unique transaction identifier. MySQL 5.6 can generate it automatically. MySQL Proxy and PECL/mysqlnd_ms 1.2 feature client-side emulations for use with any MySQL version. SQL can be used to access the GTIDs.

GTIDs have been been created to make MySQL Replication failover easier. However, they are useful for load balancing as well in a primary copy system.

● With or without GTID: primary onlyOnly the primary has all comitted transactions

MySQL Replication: strong con.

MySQL Master

MySQL Slave MySQL Slave

SET X = 1

Client A

GET X, X = 1

Client B

The speaker says...

Configuring PECL/mysqlnd_ms for use with a MySQL Replication cluster and calling mysqlnd_ms_set_qos($conn, MYSQLND_MS_QOS_CONSISTENCY_STRONG) instructs PECL/mysqlnd_ms to use only the MySQL Replication master server for requests.

In a lazy primary copy system there is only one node that is guaranteed to have all comitted updates: the primary. Please note, that its possible to achieve higher consistency levels than eventual consistency in an lazy primary copy system by appropriately choosing nodes.

● Use GTID to find „synchronous“ slaveCheck slave status using SQLReduce read load on master

MySQL Replication: session con.

MySQL Master

..., GTID M:1: UPDATE x=1

..., GTID M:2: UPDATE x=9


… , GTID M:1: UPDATE x=1… , GTID M:1: UPDATE x=1… , GTID M:2: UPDATE x=9

SET X = 9

GET X, X = 9

The speaker says...

Global transaction identifier help to find „up-to-date“ slaves that have already replicated the latest updates of a client. Thus, session consistency can now be achieved by reading from the master and selected „up-to-date“ slaves. This works with the GTID emulation of PECL/mysqlnd_ms 1.2+ and any MySQL version. And, it works with MySQL 5.6 that has built-in GTIDs.

PECL/mysqlnd_ms 1.4 can either wait for a slave to catch up or search all slaves before considering the master. The wait effectively means throttling slaves to reduce slave lag!

● With or without GTID: all slavesOptional QoS level: upper slave lag limit MySQL estimates slave lag!

MySQL Replication: eventual c.

MySQL Master


Slave lag = 1 second

SET X = 9

GET X, X = 8

Slave lag = 7 seconds

The speaker says...

A MySQL Replication slave is eventual consistent – it may or may not have the latest updates. There is no need to filter nodes with regards to consistency.

Slaves can be filtered by replication lag: mysqlnd_ms_set_qos($conn, MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL, MYSQLND_MS_QOS_OPTION_AGE, 5) filters out all slaves that report an estimated lag of more than five seconds.

● Same logic whenever slaves are to be filteredapplied for: session consistency + GTIDapplied for: eventual consistency + LagStage 1: send SQL to check status to all slavesStage 2: fetch replies in orderStage 3: apply filter logicSQL is documented in the manual

Slave selection logic

The speaker says...

Whenever PECL/mysqlnd_ms has to check the slave status for filtering slaves, the check is done in two stages. First, all slaves are queried. Then, replies are fetched from the slaves in order. Usually, many requests can be send out in stage one before the servers reply.

The SQL details are documented at php.net/mysqlnd_ms.

● Automatically cache data no older than n secondsReduce slave loadCaches: Memcache, process memory, …PHP 5.4, PECL/mysqlnd_qc 1.1

QoS: Cache Integration


Slave lag = 1 second Slave lag = 7 seconds

Max_TTL = 10 – 1 = 9 TTL = MAX_AGE - 7TTL = MAX_AGE - 1

Cache

1st GET X, X = 8

2nd GET X, X = 8

The speaker says...

If the load balancer knows that eventual consistency is good and a certain degree of stale data (age, slave lag) is allowed, a slave access can be transparently replaced with a cache access. This lowers the overall slave load in the cluster and reduces latency.

Replacing a slave access with a cache access is transparent from an application point of view. PECL/mysqlnd_ms is using PECL/mysqlnd_qc (qc = query cache) for the caching. PECL/mysqlnd_qc can also be used as a standalone cache to store query results in Memcache, process memory and many more caches.


/* no caching */

$res = $mysqli->query("SELECT headline FROM latest_news");

var_dump($res->fetch_all(MYSQLI_ASSOC));

/* use cache, if slave lag allows */

mysqlnd_ms_set_qos($mysqli,

MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL,

MYSQLND_MS_QOS_OPTION_CACHE, 60));

$res = $mysqli->query("SELECT headline FROM old_news");

var_dump($res->fetch_all(MYSQLI_ASSOC));

Transparent caching

The speaker says...

Complicated logic behind but easy to use – PHP should look out for new cluster aware APIs.

Let's close with an example of a minimal configuration for use with MySQL Replication.

/* php.ini */

mysqlnd_ms.enable=1

mysqlnd_ms.config_file=/path/to/mysqlnd_ms_plugin_json.conf

/* /path/to/mysqlnd_ms_plugin_json.conf contents */

{ "myapp": {

"master": {

"master_0": {

"host": "localhost",

"socket": "/tmp/mysql.sock"

}},

"slave": {

"slave_0": {

"host": "192.168.2.27",

"port": "3306"

}}}}

Speaking code: configuration


$pdo = new PDO('mysql:host=myapp;dbname=database', 'username', 'password');

$mysql = mysql_connect("myapp", "username", "password");

/* Statements will be run on the master */

$mysqli->query("DROP TABLE IF EXISTS test");

$mysqli->query("CREATE TABLE test(id INT)");

$mysqli->query("INSERT INTO test(id) VALUES (1)");

/* Goes to the slave - don't let slave lag fool you! */

$res = $mysqli->query("SELECT id FROM test");

var_dump($res->fetch_assoc());

$mysqli->close();

Connect and use... - as ever

The speaker says...

Imagine we would have failover that could be bound to certain error codes. For example, assume the SELECT returns „Error: 1051 SQLSTATE: 42S02 (ER_BAD_TABLE_ERROR), Message: Unknown table '%s' „ because you forgot about possiblity of a lag between the CREATE and on the master and the SELECT on the slave. What if PECL/mysqlnd_ms would automatically fail over to the master to retry the SELECT... Sure, the better solution is to as for session consistency to avoid this issue at all.

Let us know what additional features you would like to see in PECL/mysqlnd_ms. We are ready to code...

THE END

Contact: [email protected], @Ulf_Wendel

Load Balancing layers overview

Kvack?Bonus slides!

● Simple – 1:1 mapping!Requires synchronous clusterNo fault tolerance: node failure = stack failureInefficient resource usage: no balancing

Whole stack cluster

HTTP ServerHTTP Server

App Server (PHP)

MySQL Node

App Server (PHP) App Server (PHP)

HTTP Server


The speaker says...

In a synchronous cluster, for example, if using MySQL Cluster, all nodes have all the data. Thus, every application server could be assigned to one DBMS node. Easy, fast and good for MySQL Cluster but with limits. No good for asynchronous MySQL Replication.

Limit: DBMS node failure includes application node failure. Client should have additional fail over logic.

Limit: Over- and undercapacity of a DBMS node cannot be compensated. Clients cannot switch to more powerful nodes. Overcapacity of a MySQL node cannot be used by other clients. Different hardware size is ignored.

● Free and open source MySQL ProxyNo application changes, free 3rd party proxy scriptsMySQL node failure does not stop application serverGood resource usage, dynamic adoption possible

Proxy approach - Pro


App Server (PHP)

MySQL Node


HTTP Server


MySQL Proxy

The speaker says...

A transparent MySQL Proxy based solution requires no application changes. Clients connect to the Proxy using MySQL Client Server Protocol, as if the MySQL Proxy was a MySQL Server.

Proxy can compensate for DBMS failures. It can react to temporary and permant outages.

Proxy can adapt load balancing dynamically. Taking into account node hardware size, current node load, latency, location, … fantasy sets the limits here for self-written or 3rd party Proxy scripts.

● Additional componentComplex architecture, can be single point of failureC and Lua are no natural match for PHP loversAdds latency: from client to proxy to node

Proxy approach - Con


App Server (PHP)

MySQL Node


HTTP Server


MySQL Proxy

The speaker says...

MySQL Proxy is a great performer!

But, … MySQL Proxy adds complexity to the stack. MySQL Proxy needs to be managed. MySQL Proxy is build around C and Lua. C and PHP would be a better match for PHP users. Wrongly used, MySQL Proxy becomes a single point of failure. It is single threaded. This bares the risk of tasks (Proxy scripts) blocking each other.

But, … MySQL Proxy adds latency. Although, it can be minimized significantly running MySQL Proxy on the App Server to avoid use of TCP/IP between PHP and Proxy.

● Client application handles load balancing No single point of failureNo additional latencyHighly configurable and adaptive

Client-side load balancer - Pro


App Server (PHP)

MySQL Node


HTTP Server


<?php .. ?> <?php ...?> <?php ?>

The speaker says...

A client-side approach to the problem is promising, if client applications can be changed. It has most Pro's of the previous approaches.

The load balancer is part of the client application. It scales by client and it fails by client. Scalability is given and there is no single point of failure. No additional latency occurs.

Load balancing can be adaptive for good resource usage. DBMS node failures do not block clients, fail over is possible.

● Client application handles load balancing Application must be designed for (specific) cluster useNo drop-in solution for existing applicationsPHP is stateless: adaptive load balancing is difficult

Client-side load balancer - Con


App Server (PHP)

MySQL Node


HTTP Server


<?php .. ?> <?php ...?> <?php ?>

The speaker says...

The major downside of a client-side application based solution is the need to change the application. Application changes must be done even for basic cluster use cases. Modifications to source code may not be possible. They may complicate upgrades of standard software. They may cause lots of work thus become expensive.

Load balancing is part of PHP thus stateless. This is both a Pro and a Con. It is difficult to hint other clients about node failures. On the other hand, shared nothing is not a bad idea either.

● Client-side and driver based: PECL/mysqlnd_msPro: all previousSynchronous cluster: no application changesAsynchronous cluster: no change for basic cases

Is driver based the solution?


App Server (PHP)

MySQL Node


HTTP Server


PECL/mysqlnd_ms PECL/mysqlnd_ms PECL/mysqlnd_ms

The speaker says...

A driver based client-side solution has all Pro's!

Considering load balancing aspect only, no application changes are required. If using MySQL Cluster, a synchronous cluster, no application changes are required. If using MySQL Replication, an asynchronous cluster that needs read-write splitting, the need for application changes is significantly reduced!

The PECL mysqlnd_ms installation can be part of the next PHP deployment, done anyway for security considerations.No new procedure. If any issues, simply disable extension.

Mysql Nd Load Balancing

Documents