HandlerSocket plugin HandlerSocket plugin for MySQL for MySQL Jun 29, 2010 Jun 29, 2010 DeNA Technology Seminar @ Yoy DeNA Technology Seminar @ Yoy ogi ogi IT Platform Dept., System Management Division IT Platform Dept., System Management Division DeNA Co.,Ltd. DeNA Co.,Ltd. Akira Higuchi <higuchi dot akira at dena dot j Akira Higuchi <higuchi dot akira at dena dot j p> p>
51
Embed
HandlerSocket plugin for MySQL Jun 29, 2010 DeNA Technology Seminar @ Yoyogi IT Platform Dept., System Management Division DeNA Co.,Ltd. Akira Higuchi.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
HandlerSocket plugin for HandlerSocket plugin for MySQLMySQL
Jun 29, 2010 Jun 29, 2010 DeNA Technology Seminar @ YoyogiDeNA Technology Seminar @ Yoyogi
IT Platform Dept., System Management DivisionIT Platform Dept., System Management DivisionDeNA Co.,Ltd.DeNA Co.,Ltd.Akira Higuchi <higuchi dot akira at dena dot jp>Akira Higuchi <higuchi dot akira at dena dot jp>
Who am I?Who am I?
Akira Higuchi, Ph.D. in science Akira Higuchi, Ph.D. in science IT Platform Dept., DeNA Co.,Ltd. IT Platform Dept., DeNA Co.,Ltd.
system-wide performance optimizationsystem-wide performance optimization middleware developmentmiddleware development
The creator of HandlerSocket pluginThe creator of HandlerSocket plugin Using GNU/Linux since 1993Using GNU/Linux since 1993
About HandlerSocket pluginAbout HandlerSocket plugin
What is HandlerSocket?What is HandlerSocket?
Non-SQL interface for MySQLNon-SQL interface for MySQL
What HandlerSocket aimsWhat HandlerSocket aims
Executes simple CRUD operations Executes simple CRUD operations fastfast Omit SQL parsingOmit SQL parsing Combine multiple requests on the server Combine multiple requests on the server
side side Allows SQL on the same databaseAllows SQL on the same database
Only simple operations can be faster Only simple operations can be faster Seamless migration from SQL queriesSeamless migration from SQL queries
HandlerSocket pluginHandlerSocket plugin
Offers a direct and non-SQL interface to MySQL Offers a direct and non-SQL interface to MySQL storage enginesstorage engines
Own TCP/IP listenerOwn TCP/IP listener Talks a text protocolTalks a text protocol There is a C++ and a Perl client librariesThere is a C++ and a Perl client libraries Only works with LinuxOnly works with Linux The source code is here:The source code is here:
08/mycached-memcac.html08/mycached-memcac.html Works with any storage enginesWorks with any storage engines Talks the memcached protocolTalks the memcached protocol
NDB APINDB API http://dev.mysql.com/doc/ndbapi/en/index.http://dev.mysql.com/doc/ndbapi/en/index.
htmlhtml Dedicated for the ndbcluster engineDedicated for the ndbcluster engine
Handlersocket executes simple read queries 4x Handlersocket executes simple read queries 4x faster than mysqld/libmysqlfaster than mysqld/libmysql
Very effective when many columns are retrievedVery effective when many columns are retrieved The reason is described laterThe reason is described later
Commands supported by Commands supported by HandlerSocket (for reading HandlerSocket (for reading
data)data) In pseudo-SQL...In pseudo-SQL...
SELECT f1, .. , fn FROM db.tableSELECT f1, .. , fn FROM db.table
WHERE k1, ... , km = v1, ... , vmWHERE k1, ... , km = v1, ... , vm
ORDER BY index_i LIMIT offset, limitORDER BY index_i LIMIT offset, limit (k1, ... , km) are the key fields (or a (k1, ... , km) are the key fields (or a
prefix) of the index_iprefix) of the index_i =, >=, >, <=, and < can be used for a =, >=, >, <=, and < can be used for a
comparatorcomparator
Commands supported by Commands supported by HandlerSocket (for modifying HandlerSocket (for modifying
data)data)
UPDATE, DELETE, and INSERTUPDATE, DELETE, and INSERT Does not support transactionsDoes not support transactions Modifications are recorded to the Modifications are recorded to the
binary log in the row-based formatbinary log in the row-based format Modifications are durableModifications are durable
Command exampleCommand example
create table db1.table1 (k int key, v char(20))create table db1.table1 (k int key, v char(20)) insert into db1.table1 values (234, 'foo'), (678, ‘bar’)insert into db1.table1 values (234, 'foo'), (678, ‘bar’)
$ telnet localhost 9998Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.P 0 db1 table1 PRIMARY k,v0 10 = 1 2340 2 234 foo0 = 1 6780 2 678 bar
opens the PK
find k = 234
find k = 678
Why fast?Why fast?
No SQL parsingNo SQL parsinglow CPU usagelow CPU usage
Executes multiple requests in bulkExecutes multiple requests in bulklow CPU/Disk usagelow CPU/Disk usage
libmysql/mysqldlibmysql/mysqld Much CPU time spent in mysqldMuch CPU time spent in mysqld Parsing SQL is slowParsing SQL is slow schedule() is called frequentlyschedule() is called frequently
HandlerSocketHandlerSocket Most CPU time is consumed in the kernelMost CPU time is consumed in the kernel schedule() is not called frequentlyschedule() is not called frequently Inside mysqld, innodb eats most CPU Inside mysqld, innodb eats most CPU
timetime
Executing multiple requests Executing multiple requests in bulkin bulk
Threading modelThreading model
mysqld:mysqld: Thread per connection (MySQL 5)Thread per connection (MySQL 5) Thread pooling (MySQL 6?)Thread pooling (MySQL 6?)
Threading modelThreading model
HandlerSocket:HandlerSocket: Small number of threadsSmall number of threads Many connections per threadMany connections per thread
Uses epoll()Uses epoll() Virtually unlimited number of concurrent Virtually unlimited number of concurrent
connectionsconnections Small memory footprintSmall memory footprint
InnoDB:InnoDB: Reader threads don’t blockReader threads don’t block Only one writer thread can be executed at the Only one writer thread can be executed at the
same timesame time
HandlerSocket requests are deadlock-freeHandlerSocket requests are deadlock-free Only simple operations are supportedOnly simple operations are supported
Client/server protocolClient/server protocol
MySQL C/S protocolMySQL C/S protocol
write(3, "L\0\0\0\3select column0,column1,column2,column3,column4 from write(3, "L\0\0\0\3select column0,column1,column2,column3,column4 from db_1.table_1 where k=15", 80) = 80db_1.table_1 where k=15", 80) = 80
Result-set metadata become very large if Result-set metadata become very large if a result-set has many columnsa result-set has many columns
Neither a HANDLER statement nor a Neither a HANDLER statement nor a server-side prepared statement does not server-side prepared statement does not help to avoid this problemhelp to avoid this problem
handlersocket_threads = 16handlersocket_threads = 16 Number of reader threadsNumber of reader threads Recommended value is the number of logical Recommended value is the number of logical
Number of file descriptors mysqld can openNumber of file descriptors mysqld can open HandlerSocket can handle up to 65000 HandlerSocket can handle up to 65000
concurrent connectionsconcurrent connections
Other configuration optionsOther configuration options
innodb_adaptive_hash_index = 1innodb_adaptive_hash_index = 1 Adaptive has index is fast, but consume Adaptive has index is fast, but consume
memorymemory
Options related to durabilityOptions related to durability
10000000 records10000000 records SELECT v from table1 where k = ?SELECT v from table1 where k = ? Random accessRandom access
Write benchmark:Write benchmark: 10000000 records10000000 records UPDATE table SET v = ? where k = ?UPDATE table SET v = ? where k = ? Random accessRandom access Durable writeDurable write