Basic MySQL Troubleshooting for Oracle Database Administrators Sveta Smirnova Principal Support Engineer October, 29, 2015
Basic MySQL Troubleshootingfor Oracle Database Administrators
Sveta SmirnovaPrincipal Support Engineer
October, 29, 2015
Table of Contents
•MySQL Server overview
•Basic troubleshooting techniques
•High concurrency issues
•Replication
2 www.percona.com
MySQL Server overview
3 www.percona.com
MySQL architecture
Connectors: C, JDBC, ODBC, Python, ...
Connection Pool: Authentication, Caches
SQL interface
Parser
Optimizer
Caches and Buffers:Global
Engine-specific
Storage engines: InnoDB, TokuDB, ...
File system: Data, Index, logs, other files
• Base• Installation layout• Log files
• Connectors• Clients, APIs
• Optimizer• Cache and buffers• Storage engines• Management
4 www.percona.com
Typical installation layout
• datadir• Schema directory
• Table and trigger files: *.frm, *.ibd, *.MYD,*.par, etc.
• Schema• ...• InnoDB shared tablespace• Log files
• InnoDB redo log files• Binary, relay log files• Error log• Slow query log• General query log
• Configurable• You can setup
custom path foreach component
• Including custompaths for tables
5 www.percona.com
Connectors for MySQL server
• Clients• MySQL CLI• MySQL Workbench• Many other graphical and web-based
• APIs• Exist for most popular programming languages• C, C++, JDBC, PHP, Python, Net, ODBC, more
6 www.percona.com
Monitoring solutions
• Command-line• Percona Toolkit• MySQL Utilities
• With GUI• MySQL Enterprise Monitor (MEM)• MEM plugin for Oracle Enterprise Manager• VividCortex• Many others
7 www.percona.com
Plugins
• Storage engines
• Authentication
• Audit
• Query rewrite
• More
8 www.percona.com
Storage engines
• Own data
• Own index format
• Own locking model
• Own diagnostic
• Own log files
• CHECK TABLE
9 www.percona.com
Basic troubleshootingtechniques
10 www.percona.com
Error processing
• Warnings, errors, and notesmysql> select max (f1) from t1;
ERROR 1630 (42000): FUNCTION test.max does not exist.
Check the ’Function Name Parsing and Resolution’ section
in the Reference Manual
mysql> select * from t1 where "f1"=1;
Empty set, 1 warning (0.05 sec)
mysql> show warnings;
+-----------+--------+----------------------------------------------------+
| Level | Code | Message |
+-----------+--------+----------------------------------------------------+
| Warning | 1292 | Truncated incorrect DOUBLE value: ’f1’ |
+-----------+--------+----------------------------------------------------+
1 row in set (0.00 sec)
11 www.percona.com
Error handling in applications
• Error information• mysql error• mysql errno
• Warnings and notes• mysql info• mysql sqlstate• mysql warning count
12 www.percona.com
Error handling in stored routines
• GET DIAGNOSTICS
• GET DIAGNOSTICS rows = ROW COUNT,conditions = NUMBER;
• GET DIAGNOSTICS CONDITION 1 code =RETURNED SQLSTATE, msg = MESSAGE TEXT;
• SIGNAL/RESIGNAL
• Can be used for custom errors
13 www.percona.com
More information about errors
[sveta@delly ~]$ perror 1630
MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION):
FUNCTION %s does not exist. Check the ’Function Name Parsing and Resolution’
section in the Reference Manual
[sveta@delly ~]$ perror 1292
MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE):
Truncated incorrect %.32s value: ’%.128s’
[sveta@delly ~]$ perror 2
OS error code 2: No such file or directory
[sveta@delly ~]$ perror 150
MySQL error code 150: Foreign key constraint is incorrectly formed
14 www.percona.com
MySQL Access Privilege System
• No roles by default, limited user limits• All records are in the mysql database (schema)• Pluggable authentication since version 5.5• Connections
• TCP/IP with login-password• Socket (Unix)• Named pipe (Windows)
15 www.percona.com
Common user access issues
• Privileged client cannot connect
• Unprivileged client can connect
• Privileged user cannot perform operation
• Unprivileged user has undesired access
16 www.percona.com
Common queries for access issuesmysql> select user, host from mysql.user order by user desc, host desc;
+------+------------+
| user | host |
+------+------------+
| root | localhost |
| root | delly |
| root | ::1 |
| root | 127.0.0.1 |
| foo | % |
| | localhost |
+------+------------+
6 rows in set (0.00 sec)
• Most descriptive host first, then wildcard• Socket connection by default on Unix
17 www.percona.com
Wrong access checklist
• SHOW GRANTS [FOR user@host]• Grant tables
• mysql.db• mysql.tables priv• mysql.columns priv• mysql.procs priv• mysql.proxies priv
• SELECT USER(), CURRENT USER()
18 www.percona.com
Wrong access examplemysql> show grants;
+--------------------------------------------------------------------+
| Grants for root@localhost |
+--------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO ’root’@’localhost’ WITH GRANT OPTION|
| GRANT PROXY ON ’’@’’ TO ’root’@’localhost’ WITH GRANT OPTION |
+--------------------------------------------------------------------+
2 rows in set (0.02 sec)
mysql> show grants for foo@’%’;
+-----------------------------------------------------+
| Grants for foo@% |
+-----------------------------------------------------+
| GRANT USAGE ON *.* TO ’foo’@’%’ |
| GRANT ALL PRIVILEGES ON ‘test‘.* TO ’foo’@’%’ |
+-----------------------------------------------------+
2 rows in set (0.02 sec)
19 www.percona.com
Connection statistics in P S
• Tables accounts, users, hostsmysql> select user, host, current_connections as cur,
-> total_connections as total from performance_schema.accounts;
+------+-----------+-----+-------+
| user | host | cur | total |
+------+-----------+-----+-------+
| foo | localhost | 0 | 3 |
| root | localhost | 1 | 3 |
| NULL | NULL | 14 | 17 |
+------+-----------+-----+-------+
3 rows in set (0.01 sec)
• HOST CACHE
20 www.percona.com
Connection statistics in P S
• Tables accounts, users, hosts• HOST CACHE
• Content of DNS cache• Errors from:
• Name Server
• Connection
• Authentication
• max connect errors, max user errors, etc.
• Your first assistant in case of connection issue
20 www.percona.com
Performance Schema• Monitors interval operations
• Statements• Stages• Locks• Memory• Variables• Replication• IO• Mutexes and waits
• Similar to Oracle wait interface21 www.percona.com
What can affect query execution?
• You run a query, it does not return an error, but stillbehaves not as expected
• It can be:• Startup options or system variables• How optimizer creates query plan• Storage engine used• Parallel execution - next big section
22 www.percona.com
System variables and options: scope
• Global• Control parameters, necessary for all server
processes• Location of server files: datadir etc.• Shared buffers• More
• Session• Control connection-specific parameters
• MySQL option tables
23 www.percona.com
System Variables: how to set
• SET [GLOBAL] var name = NEW VAL
• Command-line option
• Configuration file
• In default location
• Specified by option –defaults-file
24 www.percona.com
System Variables: who can change
• Global options and few session options
• A user with privilege SUPER
• Session options
• Anybody
• There are no limits!
25 www.percona.com
Buffers: when allocated
• Those which control behavior of whole server• Once at server startup• Can start with low values, then grow to specified
• Connection options• For every connection when connection opens
• Operation-specific• For every operation when needed• Can be allocated more than once per query
26 www.percona.com
System variables: control before 5.7• SHOW [GLOBAL] STATUS• Tables in Information Schema
• GLOBAL ,SESSION VARIABLES• GLOBAL ,SESSION STATUS
• GLOBAL• Since server start
• SESSION• For operations in current session• Can be reset
• FLUSH STATUS
27 www.percona.com
System status variables: example
mysql> show global status like ’Handler_read_rnd_next’\G
*************************** 1. row ***************************
Variable_name: Handler_read_rnd_next
Value: 27
1 row in set (0.00 sec)
mysql> show status like ’Handler_read_rnd_next’\G
*************************** 1. row ***************************
Variable_name: Handler_read_rnd_next
Value: 7
1 row in set (0.00 sec)
28 www.percona.com
Information Schema
• Contains metadata information• Tables• Indexes• Other
• Allows to create plugins• InnoDB plugins
• Similar to Data Dictionary Views in Oracle
29 www.percona.com
System variables: control in 5.7
• Performance Schema tables• variabes by*• user variables by*• status by*
• Statistics grouped by• Global• Session• Thread• Account/Host/User
30 www.percona.com
System variables: best practices
• Record currently used variables• SHOW [GLOBAL] VARIABLES
• Make change dynamically if possible• SET [GLOBAL] var name=NEW VAL
• Test in one session first• Then change global variable• Change configuration file after you are happy with
results31 www.percona.com
When affecting option is not known
• Record currently used variables• SHOW [GLOBAL] VARIABLES
• Start mysqld with option –no-defaults• This option must be first one!
• Check if problem is solved• Change variable values one-by-one until you find one
which leads to the problem
32 www.percona.com
MySQL Optimizer
• EXPLAIN is less powerful if compare with Oracle• Visual EXPLAIN in MySQL Workbench
• EXPLAIN EXTENDED• Should be followed by SHOW WARNINGS
• EXPLAIN PARTITIONS• EXPLAIN FORMAT=JSON• INFORMATION SCHEMA.TRACE• Status variables ’Handler %’
33 www.percona.com
EXPLAIN in Oracle
• http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htmEXPLAIN PLAN SET statement_id = ’example_plan4’ FOR
SELECT h.order_number, l.revenue_amount, l.ordered_quantity
FROM so_headers_all h, so_lines_all l
WHERE h.customer_id = :b1
AND h.date_ordered > SYSDATE30
AND l.header_id = h.header_id ;
Plan
--------------------------------------------------
SELECT STATEMENT
NESTED LOOPS
TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL
INDEX RANGE SCAN SO_HEADERS_N1
TABLE ACCESS BY INDEX ROWID SO_LINES_ALL
INDEX RANGE SCAN SO_LINES_N1
34 www.percona.com
EXPLAIN in MySQLmysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.user\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: user
type: index
possible_keys: NULL
key: PRIMARY
key_len: 228
ref: NULL
rows: 4
filtered: 100.00
Extra: Using index
1 row in set, 1 warning (0.00 sec)
mysql> SHOW WARNINGS\G
******* 1. row *******
Level: Note
Code: 1003
Message: /* select#1 */ select ‘mysql‘.‘user‘.‘User‘ AS ‘user‘, ‘mysql‘.‘user‘.‘Host‘ AS ‘host‘ from ‘mysql‘.‘user‘
35 www.percona.com
EXPLAIN: overview
mysql> explain extended select * from t1 join t2 where t1.int_key=1;
+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | f... | Extra |
+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+------------------+
| 1 | SIMPLE | t1 | ref | int_key,ik | int_key | 5 | const | 4 | 100. | NULL |
| 1 | SIMPLE | t2 | index | NULL | pk | 9 | NULL | 6 | 100. | Using index; |
Using join buffer |
(Block Nested Loop) |
+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+------------------+
2 rows in set, 1 warning (0.00 sec)
Note (Code 1003): /* select#1 */ select ‘test‘.‘t1‘.‘pk‘ AS ‘pk‘,‘test‘.‘t1‘.‘int_key‘ AS ‘int_key‘,‘test‘.‘t2‘.‘pk‘
AS ‘pk‘,‘test‘.‘t2‘.‘int_key‘ AS ‘int_key‘ from ‘test‘.‘t1‘ join ‘test‘.‘t2‘ where (‘test‘.‘t1‘.‘int_key‘ = 1)
Number of select
Select type
Tables, for which information is printed
How data is accessed
Possible keys
Key, which was actually used
Length of the key
Which columns were compared with the index
Number of examined rows
% of filtered rowsrows x filtered / 100 — number of rows,which will be joined with another table
Additional information
Table, for which information is printedProduct of rows here: how many rows in all tables will be accessed
For this example estimated value is 4*6 = 24
Actual (optimized) query as executed by MySQL Server
36 www.percona.com
EXPLAIN FORMAT=JSONmysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.user\G
*************************** 1. row ***************************
EXPLAIN: {
"query_block": {
"select_id": 1,
"table": {
"table_name": "user",
"access_type": "index",
"key": "PRIMARY",
"used_key_parts": [
"Host",
"User"
],
"key_length": "228",
"rows": 8,
"filtered": 100,
"using_index": true
}
}
}37 www.percona.com
When EXPLAIN lies: Handler % variablesmysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> select count(*) from employees join titles using(emp_no)
-> where title=’Senior Engineer’\G
*************************** 1. row ***************************
count(*): 97750
1 row in set (3.24 sec)
mysql> SHOW STATUS LIKE ’Handler_read_%’;
+----------------------------+--------+
| Variable_name | Value |
+----------------------------+--------+
| Handler_read_first | 1 |
| Handler_read_key | 300027 |
| Handler_read_last | 0 |
| Handler_read_next | 397774 |
...
38 www.percona.com
MySQL Optimizer: other tools
• INFORMATION SCHEMA.OPTIMIZER TRACE
• join preparation, join optimization, join execution• considered execution plans, refine plan, more
• Query Analyzer in MEM
• Visual EXPLAIN in MySQL Workbench
39 www.percona.com
MySQL Storage Engines
• Own way to handle• Corruption• Index statistics• CHECK TABLE to check for errors
• They care about physical data, so all data information ison their level
• Options usually start from engine name• myisam *, innodb *, custom *
40 www.percona.com
Storage Engines explained
• InnoDB• TokuDB• Popular non-transactional engines
• MyISAM - Previous default engine• Memory• Federated• CSV• Blackhole
41 www.percona.com
Storage Engines explained
• InnoDB
• Most used storage engine• Physical layout
• *frm file – table definition
• Shared tablespace
• *ibd file – tablespace for individual table - –innodb file per table = 1
• Redo log files
• Automatic startup check
41 www.percona.com
Storage Engines explained
• InnoDB
• Own locking model: row-based• Troubleshooting tools
• InnoDB Monitors (SHOW ENGINE InnoDB STATUS)
• Tables in Information Schema
• Instrumentation in Performance Schema
41 www.percona.com
Storage Engines explained
• TokuDB
• Fractal tree• Optimized for compression and writes• Troubleshooting tools
• SHOW ENGINE tokudb STATUS
• Information schema tables
• Status variables
41 www.percona.com
Storage Engines explained
• Popular non-transactional engines
• Server provides locking model• Table-level locks• Can have own instrumentation• MyISAM
• Status variables
• Performance Schema
• Set of utilities, names start from myisam *
41 www.percona.com
High concurrency issues
42 www.percona.com
Common concurrency issues
• Query or transaction waits a lock, held by another one
• Fight for system resources
• Resource overload
• Resource underload
43 www.percona.com
Lock types and transactions
• Lock types• Levels
• MDL• Table-level• Row-level
• What do they lock• Read locks
• Block writes
• Write locks• Block reads and writes
• Transactions• Server-level
• MDL locks• Table locks
• Engine-level• Table locks• Row locks
• AUTOCOMMIT• supported
44 www.percona.com
Locks diagnostic• SHOW [FULL] PROCESSLIST• SHOW ENGINE INNODB STATUS• INFORMATION SCHEMA
• PROCESSLIST• InnoDB table
• Performance Schema• METADATA LOCKS• TABLE HANDLES• EVENTS TRANSACTIONS *
• Both server-level and engine-level• Contain GTID information
45 www.percona.com
Locks diagnostic
• SHOW [FULL] PROCESSLISTmysql> select id, state, info from information_schema.processlist\G
******************** 1. row ********************
id: 5
state:
info: NULL
******************** 2. row ********************
id: 4
state: Waiting for table metadata lock
info: alter table titles add column f1 int
******************** 3. row ********************
id: 2
state: executing
info: select id, state, info from information_schema.processlist
45 www.percona.com
Locks diagnostic
• Performance Schemamysql> select processlist_id, object_type, lock_type, lock_status, source
-> from metadata_locks join threads on (owner_thread_id=thread_id)
-> where object_schema=’employees’ and object_name=’titles’\G
*************************** 1. row ***************************
processlist_id: 4
object_type: TABLE
lock_type: EXCLUSIVE
lock_status: PENDING -- waits
source: mdl.cc:3263
*************************** 2. row ***************************
processlist_id: 5
object_type: TABLE
lock_type: SHARED_READ
lock_status: GRANTED -- holds
source: sql_parse.cc:5707
45 www.percona.com
Locks diagnostic
• SHOW ENGINE INNODB STATUS
-------------
TRANSACTIONS
-------------
Trx id counter 0 26243837
Purge done for trx’s n:o < 0 26243834 undo n:o < 0 0
History list length 2
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0 26243836, ACTIVE 4 sec, OS thread id 101514240 starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 320, 1 row lock(s)
MySQL thread id 4485, query id 25022137 localhost root Updating
update t set a=36 where a=6
45 www.percona.com
InnoDB Monitors
• SHOW ENGINE INNODB STATUS
• Permanent output
• innodb status output
• innodb status output locks
• innodb-status-file - Deleted on normal shutdown!
46 www.percona.com
Other InnoDB diagnostic
• INFORMATION SCHEMA• INNODB TRX• INNODB LOCKS• INNODB LOCK WAITS• INNODB METRICS
• Options innodb monitor *
• Option innodb print all deadlocks
47 www.percona.com
Locks diagnostic summary• Table-level
• PROCESSLIST: ”Waiting for table lock”• P S.TABLE HANDLES
• Row-level• InnoDB monitors
• SHOW ENGINE INNODB STATUS
• Tables in INFORMATION SCHEMA• Option –innodb print all deadlocks
• MDL• PROCESSLIST
• ”Waiting for metadata lock”
• P S.METADATA LOCKS48 www.percona.com
Replication
49 www.percona.com
MySQL Replication Overview• Always available, needs to be setup• Asynchronous master-slave• Master
• Keeps all updates in binary log file• Two formats: ROW and STATEMENT
• Slave• IO thread read updates from master and stores in
relay log file• SQL thread executes updates
• Multiple SQL threads since 5.6• Multiple masters since 5.7
• GTIDs since 5.650 www.percona.com
Main troubleshooting tools
• SHOW SLAVE STATUS
• Tables in Performance Schema and system database
• Multi-threaded slave (MTS) has additional output foreach worker
• Error log file
51 www.percona.com
Main troubleshooting tools• SHOW SLAVE STATUS
mysql> show slave status \G
************************ 1. row ************************
Slave_IO_State: Queueing master event to the relay log
...
Master_Log_File: master-bin.000003
Read_Master_Log_Pos: 191
Relay_Log_File: slave-relay-bin.000006
Relay_Log_Pos: 4
Relay_Master_Log_File: master-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
Last_Errno: 0
Last_Error:
...
Exec_Master_Log_Pos: 319
...
51 www.percona.com
Main troubleshooting tools• Tables in Performance Schema and system database
• replication connection configuration• replication applier configuration• replication connection status• replication applier status• replication applier status by coordinator• replication applier status by worker - MTS only• mysql.slave master info• mysql.slave relay log info• mysql.slave worker info
51 www.percona.com
Main troubleshooting tools• Multi-threaded slave (MTS)
mysql> select * from replication_applier_status_by_worker\G
*************************** 1. row ***************************
CHANNEL_NAME:
WORKER_ID: 1
THREAD_ID: 25
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION:
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
CHANNEL_NAME:
WORKER_ID: 2
THREAD_ID: 26
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: d0753e78-14ec-11e5-b3fb-28b2bd7442fd:770
...
51 www.percona.com
IO thread: communication issues
• Check slave error log• SHOW SLAVE STATUS• P S.replication connection status• Access error
• Try to connect using normal MySQL client usingslave credentials
• SHOW GRANTS
• Fix privileges on master• Restart slave
52 www.percona.com
SQL thread: typical issues
• Simple master-slave• Data is different on master and slave
• Replication event can not be applied
• Different errors on master and slave• Slave lags far behind the master
• Circular replication or other writes in addition to slaveSQL thread
• Data is different on master and slave
53 www.percona.com
Data is different on master and slave
• Was the table modified besides the SQL thread?• How?• Can it affect content of the table in the wrong way?
• Are the table definitions same on master and slave?• MySQL Utilities
• mysqlrplsync, mysqldbcompare, mysqldiff
• Maybe master events were applied in wrong order?• Use mysqlbinlog to find queries caused the issue• Check application, running on master
54 www.percona.com
Events applied in different order
• Happens only with STATEMENT format• Row lock issues• Triggers
• SET GLOBAL slave skip counter - No GTIDs!• Skip transaction - GTIDs• Synchronize tables!
• Different options• Start slave with master’s options, then check• Happens in older versions
55 www.percona.com
Slave lags behind master
• Threads• Master runs in multiple update threads• Slave uses single
• Seconds behind master is growing - Not 100 % reliable!• Tune slave performance
• Multi-threaded slave• One thread per database in 5.6• Affected by concurrency issues
• Indexes on slave only• For statement-based replication
56 www.percona.com
More information
57 www.percona.com
Further reading
• MySQL Troubleshooting book
• Planet MySQL
• MySQL User Reference Manual
• Bug trackers
• http://bugs.mysql.com• https://bugs.launchpad.net/percona-server/
58 www.percona.com
Place for your questions
???
59 www.percona.com
Thank you!
http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
60 www.percona.com