How To Set Up SQL Load Balancing with HAProxy - Slides

Confidential

Load Balancing with HAProxy and MySQL

!

!

July 9, 2014

Alex Yu [email protected]

mailto:[email protected]

Copyright Severalnines AB

Webinar Housekeeping

!This webinar is being recorded

!A link to the recording & slides will be posted on severalnines.com

!We welcome questions: enter questions into the chat box and we will respond at the end of the presentation

!Think of something later? ! Email Severalnines at [email protected]

2



Agenda

!What Is HAProxy?

!SQL Load Balancing with HAProxy

! Failure detection with MySQL

!Fault Tolerance with HAProxy

! Active/Passive setup

!Deployment Scenarios

! MySQL Cluster, Galera Cluster and MySQL Replication

!Load Balancing Alternatives

3


Why Load Balance?

!Optimising server utilisation ! Queuing and throttling

!Maximise availability ! No SPOF

!Maximise throughput ! Distribute workload across a set of servers

!Scalability ! Site traffic grows ! Add DB servers to provide more throughput

4

LB

DB DB DB …

!Reliable High Performance TCP/HTTP Load Balancer ! v1.5 stable released June 2014

! Native SSL support, IPv6, full HTTP keepalive, HTTP Compression … ! v1.4 most deployed and stable version, released Feb 2010

! client-side keep alive, TCP speedups, source base stickiness … ! Single Process Event-Driven. Can easily saturate 10Gb link with a single instance

! 108 000 session/s, http://www.haproxy.org/10g.html (2009) !Reverse Proxy

! load balancing, scaling out, failover (health checks) !High traffic sites like

! reddit, DISQUS, GitHub, Imgur, StackOverflow/Server Fault, Instagram …


HAProxy?

5

http://www.haproxy.org/10g.html


SQL Load Balancing for MySQL

!Configuration file ! Sections; global, defaults, frontend, backend, listen ! /etc/haproxy/haproxy.cfg

!Global (process wide)

6

MySQL[WSREP]

Galera Replication (Synchronous)

LB

MySQL[WSREP]

MySQL[WSREP]

R/W R/W R/W

Client Client Client

global log 127.0.0.1 local0 pidfile /var/run/haproxy.pid daemon user nobody group nobody stats socket /tmp/haproxy.socket … node HAProxy_1 description HAProxy 1 maxconn 40000 spread-checks 3 quiet


SQL Load Balancing for MySQL (cont.)

!defaults

7

MySQL[WSREP]


LB

MySQL[WSREP]

MySQL[WSREP]

R/W R/W R/W


defaults log global mode tcp maxconn 40000 option dontlognull option tcp-smart-accept option tcp-smart-connect option redispatch retries 3 timeout queue 3500ms timeout connect 3500ms timeout client 120s timeout server 120s



! listen (frontend + backend)

8

listen s9s1_33306_defaul_LB mode tcp bind *:33306 timeout client 60000ms timeout server 60000ms balance leastconn option httpchk #option allbackups default-server port 9200 inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100 server db1 10.0.3.70:3306 check server db2 10.0.3.60:3306 check server db3 10.0.3.50:3306 check



! listen (stats ui)

9

userlist STATSUSERS group admin users admin user admin insecure-password admin !listen admin_page 0.0.0.0:9600 mode http stats enable stats refresh 60s stats uri / acl AuthOkay_ReadOnly http_auth(STATSUSERS) acl AuthOkay_Admin http_auth_group(STATSUSERS) admin stats http-request auth realm admin_page unless AuthOkay_ReadOnly


10

!Session rate ! number of new connections per second

!Sessions ! current number of sessions/connections

!LastChk ! health check

!Wght ! server weight


MySQL Health Checks

11

!Built-in MySQL health check ! option mysql-check user <username> ! Sends two MySQL packages

! Authentication and Quit ! No errors unless MySQL server is down

!Complete HTTP Request ! option httpchk ! Response: HTTP/1.1 200 OK ! Check Host and/or MySQL server state

! Galera Node State

MySQL[WSREP]


LB

MySQL[WSREP]

MySQL[WSREP]

R/W R/W R/W



MySQL Health Checks (cont.)

12

!Galera Node ! Check 1: Get node state

! SYNCED or DONOR ! Check 2: If Node is a “DONOR”

! Get the SST method ! xtrabackup

! Return “200 OK” ! rsync or mysqldump

! Return “503 Service Unavailable” Galera Node State


MySQL Health Checks (cont.)

13

!xinetd service ! mysqlchk.sh (shell script) ! spams syslog by default

! log_on_success =

#/etc/xinetd.d/mysqlchk # default: on # description: mysqlchk service mysqlchk { flags = REUSE socket_type = stream port = 9200 wait = no user = nobody server = /usr/local/bin/mysqlchk.sh log_on_failure += USERID disable = no #only_from = 0.0.0.0/0 per_source = UNLIMITED }


Network Tuning

!TCP tuning

14

#/etc/sysctl.conf net.ipv4.ip_nonlocal_bind=1 net.ipv4.tcp_tw_reuse=1 # reuse TIME-WAIT sockets net.ipv4.ip_local_port_range=1024 65023 # increase max num of ports net.ipv4.tcp_max_syn_backlog=40000 #Increase the number of outstanding syn requests net.ipv4.tcp_max_tw_buckets=400000 # Maximal number of timewait sockets net.ipv4.tcp_max_orphans=60000 net.ipv4.tcp_max_syn_backlog=40000 # TCP SYN Flood Protection net.ipv4.tcp_synack_retries=3 # TCP SYN Flood Protection net.core.somaxconn=40000 # burst connection rate net.ipv4.tcp_fin_timeout=5 # how long to keep sockets in FIN-WAIT-2


Fault Tolerance for HAProxy

15

Client

HAProxy 1

MySQL MySQL MySQL

IP: 10.10.10.20


Fault Tolerance for HAProxy (cont.)

!Active/Passive IP failover ! keepalived ! VRRP protocol provides router/IP failover

!VIP, “floating IP”: 10.10.10.10 ! net.ipv4.ip_nonlocal_bind=1

!Master - HAProxy 1 ! IP: 10.10.10.20

!Backup - HAProxy 2 ! IP: 10.10.10.30

16


Fault Tolerance for HAProxy (cont.)

!keepalived configuration file

17

#/etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script "killall -0 haproxy" # verify the pid existence interval 2 # check every 2 seconds weight 2 # adjust priority by this weight } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 # ID for this route priority 101 # 101 on master # 100 on backup virtual_ipaddress { 10.10.10.10 # the virtual IP } track_script { chk_haproxy } }


ClusterControl HAProxy + Keepalived

!Install HAProxy with our admin scripts ! http://www.severalnines.com/blog/using-haproxy-database-clusters

18

$ git clone https://github.com/severalnines/s9s-admin.git $ cd s9s-admin/cluster/ # install haproxy1 on 10.10.10.20 listening on 33306 $ ./s9s_haproxy --install -i 1 -n <ip db1>,<ip db2>,<ip db3> -h10.10.10.20 # install haproxy2 on 10.10.10.30 listening on 33306 $ ./s9s_haproxy --install -i 1 -n <ip db1>,<ip db2>,<ip db3> -h10.10.10.30

# scale out db nodes on haproxy1 $ ./s9s_haproxy --add-node -i 1 -h 10.10.10.20 -n <ip db4> $ ./s9s_haproxy --add-node -i 1 -h 10.10.10.20 -n <ip db5> # scale out db nodes on haproxy2 $ ./s9s_haproxy --add-node -i 1 -h 10.10.10.30 -n <ip db4> $ ./s9s_haproxy --add-node -i 1 -h 10.10.10.30 -n <ip db5>

http://www.severalnines.com/blog/using-haproxy-database-clusters


ClusterControl HAProxy + Keepalived (cont.)

!Deployment and configuration automated from ClusterControl

19


ClusterControl HAProxy + Keepalived (cont.)

20


Deployment Scenarios

21

!MySQL Cluster / NDB Engine ! Active/Active, Shared nothing architecture ! SQL, Data and MGMT Nodes ! HAProxy co-located with MGMT Nodes

! MGMT is a light weight process ! Re-use “idle” servers ! Connections originate from HAProxy server

! Additional latency ! App Server <-> HAProxy <-> SQL Node <-> Data Node

NDB Data Node

NDB Data Node

(MGMT)

HAProxy

(MGMT)

HAProxy

SQL Node MySQL Server



Deployment Scenarios (cont.)

22

NDB Data Node

NDB Data Node

!MySQL Cluster / NDB Engine ! Active/Active, Shared nothing architecture ! SQL, Data and MGMT Nodes ! HAProxy co-located with App servers

! No extra latency ! Extra load on app servers ! Health check loads on SQL nodes from all LBs

! Alt: MySQL Connector/J’s JDBC driver ! jdbc:mysql:loadbalance://host-1,host-2,…host-n/database?

loadBalanceBlacklistTimeout=5000 ! Config changes propagate to all app servers

(App)

HAProxy

(App)

HAProxy





23

NDB Data Node

NDB Data Node

!MySQL Cluster / NDB Engine ! Active/Active, Shared nothing architecture ! SQL, Data and MGMT Nodes ! HAProxy co-located with SQL nodes

! No extra latency ! Extra load on SQL nodes ! Do not use JDBC load balancing ! VIP

(App) (App)



HAProxy HAProxy

…



24

MySQL SlaveMySQL Master

!MySQL Replication Master/Slave ! 1 Master for Writes ! N Slaves for Reads ! Separate front and backends configs for Master and Slaves

!ex: backend configuration for SlavesHAProxy

MySQL Slave

MySQL Slave

Writes

Readsfrontend front_reads mode tcp bind 192.168.100.110:3316 default_backend cluster_reads backend cluster_reads mode tcp balance roundrobin server Slave1 10.10.10.30:3306 check server Slave2 10.10.10.40:3306 check



25

!Galera Cluster ! Synchronous Replication, Read & Write to any Node ! Cluster wide optimistic locking

! Higher probability for “deadlocks” ! Pessimistic locking on the Node

MySQL[WSREP]


LB

MySQL[WSREP]

MySQL[WSREP]

R/W R/W R/W




26

!Galera Cluster ! Synchronous Replication, Read & Write to all Nodes ! Cluster wide optimistic locking

! Higher probability for “deadlocks” ! Pessimistic locking on the Node

MySQL[WSREP]


LB

MySQL[WSREP]

MySQL[WSREP]

R/W R/W R/W




27

MySQL[WSREP]


LB

MySQL[WSREP]

MySQL[WSREP]

R/W R/W R/W


listen s9s1_33307_LB1 bind *:33307 mode tcp timeout client 60000ms timeout server 60000ms balance leastconn option httpchk default-server port 9200 inter 2s <…> server db1 10.0.3.70:3306 check server db2 10.0.3.60:3306 check backup server db3 10.0.3.50:3306 check backup

!Handle “dead locks” ! Write to only one Node - Use HAProxy ! Expect deadlocks and make sure applications

handles it properly. Retry the transaction. ! wsrep_retry_autocommit=1 ! Changes to the App code to minimise data hotspots.


Alternatives!MySQL’s Connector/J JDBC driver

!jdbc:mysql:loadbalance://host-1,host-2,…

!PHP mysqlnd

! http://www.php.net/manual/en/intro.mysqlnd-ms.php

!Oracle’s MySQL Proxy 0.84

! http://dev.mysql.com/downloads/mysql-proxy/

!SkySQL’s MaxScale 1.0.0. beta

! https://github.com/skysql/MaxScale

!ProxySQL

! https://github.com/renecannao/proxysql

!Pen 0.24.0

! http://morestuff.siag.nu/category/pen/

28

http://www.php.net/manual/en/intro.mysqlnd-ms.php

http://dev.mysql.com/downloads/mysql-proxy/

https://github.com/skysql/MaxScale

https://github.com/renecannao/proxysql

http://morestuff.siag.nu/category/pen/


Thank You!

!Tutorial – MySQL Load Balancing with HAProxy ! http://www.severalnines.com/resources/clustercontrol-mysql-haproxy-load-

balancing-tutorial

!Severalnines Blog ! www.severalnines.com/blog

!s9s-admin tools ! https://github.com/severalnines/s9s-admin

!More Questions? Contact us at: ! [email protected]

29


How To Set Up SQL Load Balancing with HAProxy - Slides

Technology

extra load

db nodes

galera replication

pessimistic

extra latency

galera cluster

read amp

higher probability