MySQL Performance schema missing_manual_flossuk

PERFORMANCE_SCHEMA in MySQL 5.6: The Missing Manual

Valerii Kravchuk, Principal Support [email protected]

www.percona.com

Who am I?

Valerii (aka Valeriy) Kravchuk:● MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012

○ Bugs Verification Team all this time○ Support issues related to bugs, InnoDB, performance and more○ Trainings (mostly informal) for new team members○ All kinds of decision making committees…

● Principal Support Engineer in Percona, 2012 - ...○ Doing more or less the same as before, but better (I hope)...○ Plus I try to speak and write about MySQL in public now

● http://mysqlentomologist.blogspot.com - my blog about MySQL (mostly bugs)● https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about

MySQL (mostly bugs…)● http://bugs.mysql.com - my personal playground. 70 bugs reported in 2014

www.percona.com

What is this session about?

● How to find the root causes of MySQL server performance problems?○ http://method-r.com/faq/35-what-is-method-r○ Where time is spent?

● Proper instrumentation of the code, is it really important?○ http://tkyte.blogspot.com/2005/06/instrumentation.html○ http://en.wikipedia.org/wiki/Observer_effect_(physics)

● OS-level debug, trace and profiling tools, aren’t they allow to pinpoint any performance problem in MySQL?

● Instrumentation in MySQL - PERFORMANCE_SCHEMA○ History and engineers involved○ Features in MySQL 5.6○ How to use it?○ Any alternatives in MySQL world?○ Sources of additional information and tools

www.percona.com

Method R - common sense approach

1. Identify the most important task (the bottleneck)2. Measure its response time (R) in detail

a. Time spent on useful processingb. Time spent on waiting (I/O, locks, mutexes)

3. Optimize that response time in the most (economically) efficient waya. Process (do) lessb. Try to reduce waits of all kinds

4. Repeat until your system is (economically) optimal

www.percona.com

On Proper Instrumentation of Code

“Code instrumentation, what is it? To me, it is the fine art of making approximately every other line of your developed code “debug” of some sort. Trace information. With timestamps. And meaningful information.” - Tom Kyte

“In science, the term observer effect refers to changes that the act of observation will make on a phenomenon being observed. This is often the result of instruments that, by necessity, alter the state of what they measure in some manner.” - Wikipedia

www.percona.com

Sources of information on performance

● Trace files from -debug binaries, optimizer trace files● Extended slow query log● show global status;● show engine innodb status\G● show engine innodb mutex;● InnoDB-related tables in the INFORMATION_SCHEMA● userstat (Percona Server and other builds)● show profiles;● Poor man’s profiler (pt-pmp) or real profilers (perf, oprofile)● DTrace● tcpdump analysis (see https://vividcortex.com/blog/2014/02/25/performance-schema-slow-

query-log-tcp-sniffing/)● In MySQL 5.6+ it’s time to use PERFORMANCE_SCHEMA!

www.percona.com

Extended slow query log

● Feature of Percona Server (5.1+)● http://www.percona.com/doc/percona-server/5.5/diagnostics/slow_extended_55.html

# User@Host: mailboxer[mailboxer] @ [192.168.10.165]# Thread_id: 11167745 Schema: board# Query_time: 1.009400 Lock_time: 0.000190 Rows_sent: 4 Rows_examined: 1543719 Rows_affected: 0 Rows_read: 4# Bytes_sent: 278 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0# InnoDB_trx_id: 1500# QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: No Tmp_table_on_disk: No# Filesort: No Filesort_on_disk: No Merge_passes: 0# InnoDB_IO_r_ops: 6415 InnoDB_IO_r_bytes: 105103360 InnoDB_IO_r_wait: 0.001279# InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000# InnoDB_pages_distinct: 6430SET timestamp=1346844943;SELECT id,title,production_year FROM title WHERE title = 'Bambi';

www.percona.com

Percona Server and userstat

http://www.percona.com/doc/percona-server/5.6/diagnostics/user_stats.html:

● Introduced in Google patch, allows to solve many performance issues● Additional tables in INFORMATION_SCHEMA:

○ CLIENT_STATISTICS○ THREAD_STATISTICS○ USER_STATISTICS○ TABLE_STATISTICS○ INDEX_STATISTICS

● Easy to configure: ○ http://dom.as/2013/04/17/on-performance-schemas/

● Low performance impact as some sources say:○ http://www.mysqlperformanceblog.com/2012/06/02/how-expensive-is-user_statistics/

● But some sources say it adds overhead:○ “and then, once userstat=1 the regression is horrible.. over 30%... “

● Hardly this can help with mutexes or waits…

www.percona.com

Time to try PERFORMANCE_SCHEMA● We know how to find functions where most of the time was spent (profiling)● We know how to find mutex waits that were long enough to force OS wait, or

how to find blocking row locks...● If we need more details, specifically time spent waiting… it’s time to use

PERFORMANCE_SCHEMA it seems● Sample output while test (for Bug #68079) was running, naive attempt

(the time is in picoseconds, 1E-12, 0.000 000 000 001):

mysql> select event_name, count_star, sum_timer_wait from performance_schema.events_waits_summary_global_by_event_name where event_name like 'wait/synch/mutex/innodb%' and count_star > 0 order by sum_timer_wait desc limit 5;+--------------------------------------------+------------+----------------+| event_name | count_star | sum_timer_wait |+--------------------------------------------+------------+----------------+| wait/synch/mutex/innodb/os_mutex | 71754726 | 11380485535846 || wait/synch/mutex/innodb/trx_sys_mutex | 60057 | 4191727883 || wait/synch/mutex/innodb/buf_pool_mutex | 8655 | 468303388 || wait/synch/mutex/innodb/mutex_list_mutex | 10310 | 427385546 || wait/synch/mutex/innodb/rw_lock_list_mutex | 8304 | 316946749 |+--------------------------------------------+------------+----------------+5 rows in set (0.00 sec)

www.percona.com

Some history behind...

● Read the WorkLogs:○ http://dev.mysql.com/worklog/task/?id=2360 - “Why we're doing this”○ http://dev.mysql.com/worklog/task/?id=4674 - “Setup for Actors”○ http://dev.mysql.com/worklog/task/?id=4513 - “Implement it…”

● Where the name comes from?“Proposals for the schema name were DYNAMIC (but that is a reserved word in standard SQL), PERFORMANCE_SCHEMA (Jim Winstead's suggestion), PERFORMANCE (Peter's preference), and DBTrace because it sounds like DTrace. There was a vote (see dev-private thread "WL#2360 Vote: schema name and table name"). The winner was PERFORMANCE_SCHEMA.” - Peter Gulutzan

“

www.percona.com

Basic ideas and terms

PERFORMANCE_SCHEMA (aka PS, P_S, PFS):● Inspired by Oracle Wait Interface● Separate database and storage engine● Records run time statistics via built-in instrumentation points● Designed to let us get the answers to the following questions:

○ Where was time spent?○ Who has used most of resources/time?○ What was executed to to use these?○ When it was executed?

● Appeared in MySQL 5.5, improved and ON by default in MySQL 5.6 (52 tables), and is documented and explained in many details already:○ http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html

● Implemented by Marc Alff (http://marcalff.blogspot.com/)● Deal with instruments (including wait events), consumers, objects (tables for

now) that are monitored, threads (some of them background) that run and actors (users) that execute SQL statements (in stages - thread statuses).

www.percona.com

Overview of PERFORMANCE_SCHEMA tables

:

www.percona.com

What to do with these tables?

● Table names are lowercase and long (events_statements_summary_by_digest)● Column names are uppercase● Tables are “in memory”, no indexes, content disappear or is reinitiated upon restart● In general you are supposed to SELECT … FROM ..., UPDATE selected columns

(setup_* tables), INSERT or DELETE (setup_actors, setup_objects) rows or run TRUNCATE TABLE (events_* tables and *_summary_* tables)use performance_schemaDESC events_waits_current;SHOW CREATE TABLE setup_timers\GUPDATE setup_instruments SET ENABLED=”YES”, TIMED=”YES”;UPDATE setup_consumers SET ENABLED=”YES”;SELECT * FROM threads;TRUNCATE TABLE events_statements_summary_by_digest;

● TRUNCATE TABLE for summary tables resets summary columns to 0 or NULL○ All but events_statements_summary_by_digest - digest rows are removed

● You need SELECT, UPDATE, … and DROP privileges, accordingly

:

www.percona.com

So, how to use P_S?

● The amount of information in the manual is huge, it seems not easy to start using P_S○ I wish it would be as easy as setting TIMED_STATISTICS to TRUE with ALTER

SYSTEM + few queries to V$SESSTATS, V$SYSSTATS or V$SESSION_WAIT● Marc Alff explains in http://marcalff.blogspot.com/2013/04/on-configuring-

performance-schema.html:○ “Define your goals” - for example, we have waits and we need to find out what

waits are most important based on time wasted (or waited)○ “Define what to instrument” - we need to instrument InnoDB waits mutex waits

and internal lock waits, but how to do this? Details next, but:■ "wait/synch/mutex/%", for mutexes■ "wait/synch/rwlock/%", for read write locks

○ “Define how much detail to collect” - depends on goals...○ “Provide sizing data” - autosizing based on max_connections (mostly)○ “Monitor sizing problems” - memory used vs dropped statistics and impact

:

www.percona.com

Define your goals

● From casual monitoring (“non-blocking SHOW”) to debugging “hangs” and solving complex scalability problems...

● Examples include:○ Monitoring file I/O globally for capacity planning○ Extended threads monitoring (“better SHOW PROCESSLIST”)○ Performance of SQL statements by stages (“extended slow query log”)○ Load by database and by table, unused indexes etc (“userstat”, setup_objects)○ Find InnoDB scalability bottlenecks (“hot” mutex or rwlock)

● Set realistic goals. PERFORMANCE_SCHEMA (in 5.6) is NOT any good for:○ Checking row locks in InnoDB tables (or explaining deadlocks)○ Checking metadata locks (until 5.7.3)○ Finding memory leaks (until 5.7.x)○ Keeping historical data about performance (buffers are limited, no disk storage)○ Explaining optimizer decisions (there is optimizer trace for this in 5.6)

● Instruments to enable, level of details to collect and overhead depend on goals and deployment (production server vs development machine)

● It makes sense to concentrate on things you can control/change

www.percona.com

Define what to instrument

● It’s controlled by the content of setup_instuments table● By default the following instruments are enabled (for slow SQL or I/O mostly):

○ File I/O○ Table I/O○ Table lock○ Statements○ Idle

● Dynamic (all but wait/synch/%), can be set in my.cnf alsomysql> select * from performance_schema.setup_instruments where enabled='YES' limit 5;+---------------------------------+---------+-------+| NAME | ENABLED | TIMED |+---------------------------------+---------+-------+| wait/io/file/sql/map | YES | YES || wait/io/file/sql/binlog | YES | YES || wait/io/file/sql/binlog_index | YES | YES || wait/io/file/sql/relaylog | YES | YES || wait/io/file/sql/relaylog_index | YES | YES |+---------------------------------+---------+-------+5 rows in set (0.00 sec)

www.percona.com

How instruments are named?● http://dev.mysql.com/doc/refman/5.6/en/performance-schema-instrument-

naming.html:○ "wait/io/file/%" - file io○ "wait/io/socket/%" - network io○ "wait/io/table/%" - table io○ "wait/lock/table/%" - table locks○ "wait/synch/cond/%" - conditions○ "wait/synch/mutex/%" - mutexes○ "wait/synch/rwlock/%" - read write locks○ "stage/%" - stages of statement execution (see show profile)○ "idle" - clients idle time.○ "statement/com/%" - commands in the client/server protocol○ "statement/sql/%" - SQL queries (and some more statement/…)

Hint: bold means “new in MySQL 5.6”, blue color means enabled by default

www.percona.com

Define how much details to collect

● It’s controlled by the content of setup_consumers table● Dynamic, can be set in my.cnf also● By default the following instruments are enabled (global, thread, statements digest

and details about statements), they are ring buffers (sizing discussed later):mysql> select * from performance_schema.setup_consumers;+--------------------------------+---------+| NAME | ENABLED |+--------------------------------+---------+| events_stages_current | NO | # current stage of execution per thread| events_stages_history | NO | # 10 recent stage events per thread| events_stages_history_long | NO | # 10000 recent stage events overall| events_statements_current | YES | # P_S.events_statements_current (details…)| events_statements_history | NO | # historical data, 10 recent per thread| events_statements_history_long | NO | # historical data, 10000 recent overall| events_waits_current | NO | # current wait event per thread| events_waits_history | NO | # 10 recent wait events per thread| events_waits_history_long | NO | # 10000 recent wait events overall| global_instrumentation | YES | # P_S enabled or not essentially (1 table)| thread_instrumentation | YES | # Statistics collection per thread (userstat)| statements_digest | YES | # P_S.events_statements_summary_by_digest+--------------------------------+---------+12 rows in set (0.00 sec)

www.percona.com

Example: setup for checking waits...

● http://mysqlentomologist.blogspot.com/2013/01/how-to-use-performanceschema-to-check.html - my experience back at MySQL 5.6 RC stage...

● To set up InnoDB mutex waits monitoring via PERFORMANCE_SCHEMA in general with MySQL 5.6.x you should do at least the following:1. Start MySQL server with all mutex related instruments enabled at startup

(performance_schema=ON by default), like this:

mysqld_safe --performance_schema_instrument='wait/synch/mutex/innodb/%=on' &

2. Connect to server and set up proper consumers explicitly, like this:

UPDATE performance_schema.setup_consumers SET enabled = 'YES' WHERE name like 'events_waits%';

3. Run your problematic load.4. Check waits using whatever tables you need

www.percona.com

Example: everything enabled in my.cnf

To set up all instruments and consumers (once and forever) in my.cnf, add:

performance_schema_instrument = '%=on'

performance_schema_consumer_events_stages_current = ONperformance_schema_consumer_events_stages_history = ONperformance_schema_consumer_events_stages_history_long = ONperformance_schema_consumer_events_statements_current = ONperformance_schema_consumer_events_statements_history = ONperformance_schema_consumer_events_statements_history_long = ONperformance_schema_consumer_events_waits_current = ONperformance_schema_consumer_events_waits_history = ONperformance_schema_consumer_events_waits_history_long = ONperformance_schema_consumer_global_instrumentation = ONperformance_schema_consumer_thread_instrumentation = ONperformance_schema_consumer_statements_digest = ON

www.percona.com

Define proper sizing for data collected

● Default sizes may just work (autosizing happens):mysql> show global variables like 'performance_schema%';+--------------------------------------------------------+-------+| Variable_name | Value |+--------------------------------------------------------+-------+| performance_schema | ON || performance_schema_accounts_size | 100 || performance_schema_digests_size | 10000 || performance_schema_events_stages_history_long_size | 10000 || performance_schema_events_stages_history_size | 10 |...| performance_schema_max_mutex_instances | 15906 |...

● The following 4 server variables define autosizing heuristic:○ max_connections○ table_definition_cache○ table_open_cache○ open_files_limit (it also depends on ulimit at OS level and defines size of 3

above if it’s not high enough to accept settings)

www.percona.com

Monitor sizing problems● Too big means a lot of memory used (no memory leaks though!)● Too small means partial data collected and performance degradation● When this happens proper status variable *_lost (23 in total) is incremented:

mysql> show global status like 'performance_schema%';+-----------------------------------------------+-------+| Variable_name | Value |+-----------------------------------------------+-------+| Performance_schema_accounts_lost | 0 || Performance_schema_cond_classes_lost | 0 || Performance_schema_cond_instances_lost | 0 || Performance_schema_digest_lost | 1 | # <-- problem...

● It’s easy to check the amount of memory used with SHOW ENGINE...:mysql> show engine performance_schema status;...| performance_schema | events_statements_summary_by_digest.row_size | 1472 || performance_schema | events_statements_summary_by_digest.row_count | 10000 || performance_schema | events_statements_summary_by_digest.memory | 14720000 |...| performance_schema | performance_schema.memory | 421469376 |

www.percona.com

What you can find out with default P_S

● What every foreground thread is doing:mysql> select * from performance_schema.events_statements_current\G*************************** 1. row *************************** THREAD_ID: 32 EVENT_ID: 240 END_EVENT_ID: NULL EVENT_NAME: statement/sql/select SOURCE: mysqld.cc:939 TIMER_START: 450538953130416024 TIMER_END: NULL TIMER_WAIT: NULL LOCK_TIME: 18001000000 SQL_TEXT: select * from performance_schema.events_statements_current... ERRORS: 0 WARNINGS: 0 ROWS_AFFECTED: 0 ROWS_SENT: 0 ROWS_EXAMINED: 0CREATED_TMP_DISK_TABLES: 0 CREATED_TMP_TABLES: 0... NO_INDEX_USED: 1 NO_GOOD_INDEX_USED: 0

www.percona.com

What you can find out with default P_S● What threads are running:

mysql> select thread_id, name, processlist_id from performance_schema.threads;+-----------+----------------------------------------+----------------+| thread_id | name | processlist_id |+-----------+----------------------------------------+----------------+| 1 | thread/sql/main | NULL || 2 | thread/innodb/io_handler_thread | NULL |...| 10 | thread/innodb/io_handler_thread | NULL || 11 | thread/innodb/io_handler_thread | NULL || 14 | thread/innodb/srv_lock_timeout_thread | NULL || 15 | thread/innodb/srv_monitor_thread | NULL || 16 | thread/innodb/srv_purge_thread | NULL || 17 | thread/innodb/srv_error_monitor_thread | NULL || 18 | thread/innodb/srv_master_thread | NULL || 19 | thread/innodb/page_cleaner_thread | NULL || 20 | thread/sql/shutdown | NULL || 21 | thread/sql/con_sockets | NULL || 32 | thread/sql/one_connection | 11 |+-----------+----------------------------------------+----------------+20 rows in set (0.00 sec)

www.percona.com

What you can find out with default P_S

● What is in the host cache:mysql> select host, last_error_seen, count_host_acl_errors

-> from performance_schema.host_cache;+-------+---------------------+-----------------------+| host | last_error_seen | count_host_acl_errors |+-------+---------------------+-----------------------+| pc-PC | 2014-03-02 17:42:09 | 1 |+-------+---------------------+-----------------------+1 row in set (0.00 sec)

● What tables are used a lot or caused a lot of waits:mysql> select object_schema, object_name, count_star, sum_timer_wait

-> from performance_schema.table_io_waits_summary_by_table-> order by 4 desc limit 5;

+---------------+-----------------------+------------+----------------+| object_schema | object_name | count_star | sum_timer_wait |+---------------+-----------------------+------------+----------------+| test | t | 19 | 41330631677220 || test | test2 | 4 | 26094282300 || test | tt | 4 | 122427090 || mysql | db | 0 | 0 || mysql | time_zone_leap_second | 0 | 0 |+---------------+-----------------------+------------+----------------+5 rows in set (16.33 sec)

www.percona.com

Example 1: slow statement execution

# Make sure we have historical events collected about stagesmysql> select count(*) from performance_schema.events_stages_history_long;+----------+| count(*) |+----------+| 2893 |+----------+1 row in set (0.09 sec)

# Find query that spent longest time on some stage, can you guess why from the below?mysql> select stg.event_name, stmt.sql_text,

-> stg.timer_wait/1000000000000 wait_s-> from performance_schema.events_stages_history_long stg-> join performance_schema.events_statements_history_long stmt-> on (stg.nesting_event_id=stmt.event_id)-> where 1=1-> /*stg.event_name like '%Sending data%'*/-> and stmt.rows_examined < 100000 /* stmt.rows_sent < 100000 */-> and stg.timer_wait > 10*1000000000000-> order by stg.timer_wait desc limit 1\G

*************************** 1. row ***************************event_name: stage/sql/updating sql_text: update tt set c1=c1+1

wait_s: 51.06471 row in set (0.03 sec)

www.percona.com

Example 2: current waits# Make sure we have all waits we care about instrumentedmysql> select e.event_name, e.source, e.operation, t.processlist_info

-> from performance_schema.events_waits_current e-> join performance_schema.threads t on t.thread_id=e.thread_id-> where type='foreground' and t.processlist_id <> connection_id()\G

*************************** 1. row *************************** event_name: wait/io/table/sql/handler source: handler.cc:2687 operation: fetchprocesslist_info: update tt set c1=c1+11 row in set (0.08 sec)

2686 MYSQL_TABLE_IO_WAIT(m_psi, PSI_TABLE_FETCH_ROW, MAX_KEY, 0, 2687 { result= rnd_next(buf); }) 2688 DBUG_RETURN(result);

mysql> select e.event_name, e.source, e.operation, t.processlist_info-> from performance_schema.events_waits_current e-> join performance_schema.threads t on t.thread_id=e.thread_id-> where type='foreground' and t.processlist_id <> connection_id()\G

*************************** 1. row *************************** event_name: idle source: mysqld.cc:909 operation: idleprocesslist_info: NULL1 row in set (0.15 sec)

www.percona.com

Example 3: hot files for disk I/Omysql> select event_name, count_star, sum_timer_wait/1000000000000 wait_s fromperformance_schema.file_summary_by_event_name order by 3 desc limit 5;+---------------------------+------------+--------+| event_name | count_star | wait_s |+---------------------------+------------+--------+| wait/io/file/sql/FRM | 1982 | 4.2914 || wait/io/file/sql/binlog | 136 | 1.3193 || wait/io/file/myisam/kfile | 1608 | 0.8984 || wait/io/file/myisam/dfile | 739 | 0.4980 || wait/io/file/archive/data | 12 | 0.2637 |+---------------------------+------------+--------+5 rows in set (0.01 sec)

# This is where to look for file namesmysql> select file_name, sum_timer_wait/1000000000000 wait_s from performance_schema.file_summary_by_instance order by 2 desc limit 5;+-----------------------------------------------------------------+--------+| file_name | wait_s |+-----------------------------------------------------------------+--------+| C:\ProgramData\MySQL\MySQL Server 5.6\data\pc-PC-bin.000013 | 0.2703 || C:\ProgramData\MySQL\MySQL Server 5.6\data\test\#sql-233c_e.frm | 0.2052 || C:\ProgramData\MySQL\MySQL Server 5.6\data\pc-PC-bin.000010 | 0.2002 || C:\ProgramData\MySQL\MySQL Server 5.6\data\pc-PC-bin.000014 | 0.1344 || C:\ProgramData\MySQL\MySQL Server 5.6\data\pc-PC-bin.index | 0.1253 |+-----------------------------------------------------------------+--------+5 rows in set (0.00 sec)

www.percona.com

Example 4: better PROCESSLISTmysql> select -> processlist_id as Id, processlist_user as User, -> processlist_host as Host, -> processlist_db as db, processlist_command as Command, -> processlist_time as Time, processlist_state as State, -> processlist_info as Info -> from performance_schema.threads -> where type='FOREGROUND'\G*************************** 1. row *************************** Id: 1 User: root Host: localhost db: testCommand: Query Time: 0 State: Sending data Info: selectprocesslist_id as Id, processlist_user as User,processlist_host as Host,processlist_db as db, processlist_command as Command,processlist_time as Time, processlist_state as State,processlist_info as Infofrom performance_schema.threadswhere type='FOREGROUND'1 row in set (0.00 sec)

www.percona.com

Example 5: SHOW PROFILES remakemysql> show warnings\G*************************** 1. row *************************** Level: Warning Code: 1287Message: '@@profiling' is deprecated and will be removed in a future release.1 row in set (0.06 sec)

# we may have no SHOW PROFILES in MySQL 5.7! Get ready:mysql> select event_id as Query_ID, timer_wait/1000000000000 as Duration, sql_text as Query

-> from performance_schema.events_statements_history_long statements-> join performance_schema.threads threads-> on threads.thread_id = statements.thread_id-> where threads.processlist_id = connection_id()-> order by Duration desc limit 2\G

*************************** 1. row ***************************Query_ID: 50Duration: 0.8202 Query: desc information_schema.schemata*************************** 2. row ***************************Query_ID: 1333Duration: 0.7237 Query: select event_id as Query_ID, timer_wait/1000000000000 as Duration, sql_text as Queryfrom performance_schema.events_statements_history_long statementsjoin performance_schema.threads threads...2 rows in set (0.02 sec)

www.percona.com

Still seems complicated?

www.percona.com

ps_helper To the Rescue!

● Developed by Mark Leith, MySQL expert @ Oracle○ worked in MySQL Support team since 2005○ senior software development manager @Oracle now (MEM, maybe MEB)○ good Oracle RDBMS background

● Easier to use comparing to “raw” PERFORMANCE_SCHEMA● Ready to use collection of views, stored routines and...● Recently even an easy way to create ps, P_S, PFS or whatever synonym you like!● git clone https://github.com/MarkLeith/dbahelper.git dbahelper - if you prefer...● Installed in a separate schema (ps_helper)● IO/latency/waits/statement digests● Tools to simplify P_S configuration changes, convenient data presentation, and more

Links:● http://www.markleith.co.uk/ps_helper/● https://github.com/MarkLeith/dbahelper/archive/master.zip● https://www.slideshare.net/Leithal/performance-schema-andpshelper

www.percona.com

ps_helper: basicsInstall:openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot < ps_helper_56.sql

Check:mysql> show tables;+-------------------------------------------------+| Tables_in_ps_helper |+-------------------------------------------------+| _digest_95th_percentile_by_avg_us || _digest_avg_latency_by_avg_us || check_lost_instrumentation |...53 rows in set (0,00 sec)

mysql> select routine_name from information_schema.routines where routine_schema='ps_helper';+-------------------------------+| routine_name |+-------------------------------+| analyze_statement_digest |...

20 rows in set (0,00 sec)

www.percona.com

ps_helper: configuration-related SPs

● reset_to_default(FALSE) - default configuration no matter what is in my.cnf● save_current_config() - save current configuration in temporary tables● reload_saved_config() - restore saved configuration● truncate_all(FALSE) - truncate all the events and summary tables:

mysql> select count(*) from performance_schema.events_statements_summary_by_digest;+----------+| count(*) |+----------+| 6 |+----------+1 row in set (0,00 sec)

mysql> call ps_helper.truncate_all(FALSE);Query OK, 0 rows affected (0,03 sec)

mysql> select count(*) from performance_schema.events_statements_summary_by_digest;+----------+| count(*) |+----------+| 1 |+----------+1 row in set (0,00 sec)

www.percona.com

ps_helper: slow statement execution

● Let’s assume you’ve noted some slow enough statement:select digest, digest_text from performance_schema.events_statements_summary_by_digest\G...*************************** 16. row *************************** digest: f1a302160f89f76cbe60a86761a8c58ddigest_text: INSERT INTO `t1` ( `c1` ) SELECT `c1` FROM `t1`

● You can try to analyze it in details:mysql> call ps_helper.analyze_statement_digest('f1a302160f89f76cbe60a86761a8c58d', 30, 1.0, FALSE, FALSE);...+------------+-------------+-----------+-----------+---------------+---------------+------------+------------+| executions | exec_time | lock_time | rows_sent | rows_affected | rows_examined | tmp_tables | full_scans |+------------+-------------+-----------+-----------+---------------+---------------+------------+------------+| 1 | 00:01:08.84 | 428.00 us | 0 | 262144 | 524288 | 1 | 1 |+------------+-------------+-----------+-----------+---------------+---------------+------------+------------+1 row in set (30,36 sec)...

www.percona.com

ps_helper.analyze_statement_digest()...+-------------------------------------+-------+-----------+| event_name | count | latency |+-------------------------------------+-------+-----------+| stage/sql/Sending data | 1 | 3.53 s || stage/sql/converting HEAP to MyISAM | 1 | 326.79 ms || stage/sql/query end | 1 | 128.05 ms || stage/sql/removing tmp table | 1 | 16.78 ms || stage/sql/init | 2 | 299.73 us || stage/sql/Creating tmp table | 1 | 109.72 us || stage/sql/Opening tables | 1 | 104.24 us || stage/sql/statistics | 1 | 54.66 us || stage/sql/closing tables | 1 | 51.30 us || stage/sql/System lock | 1 | 43.07 us || stage/sql/freeing items | 1 | 42.29 us || stage/sql/preparing | 1 | 36.15 us || stage/sql/checking permissions | 2 | 18.98 us || stage/sql/optimizing | 1 | 14.15 us || stage/sql/end | 1 | 8.06 us || stage/sql/executing | 1 | 5.57 us || stage/sql/cleaning up | 1 | 3.71 us || stage/sql/logging slow query | 1 | 2.32 us |+-------------------------------------+-------+-----------+18 rows in set (30,37 sec)

# and more outputs...

www.percona.com

ps_helper: hot tables...mysql> select * from ps_helper.schema_table_statistics_with_buffer limit 1\G*************************** 1. row *************************** table_schema: test table_name: t1 rows_fetched: 524306 fetch_latency: 20.97 s rows_inserted: 524288 insert_latency: 00:01:47.45 rows_updated: 0 update_latency: 0 ps rows_deleted: 0 delete_latency: 0 ps io_read_requests: 14 io_read: 1.42 KiB io_read_latency: 73.96 us io_write_requests: 7992 io_write: 254.04 MiB io_write_latency: 1.11 s io_misc_requests: 253 io_misc_latency: 20.18 s innodb_buffer_allocated: 107.17 MiB # these are from I_S.INNODB_BUFFER_PAGE innodb_buffer_data: 99.16 MiB innodb_buffer_pages: 6859innodb_buffer_pages_hashed: 6859 innodb_buffer_pages_old: 6859 innodb_buffer_rows_cached: 4734931 row in set (2,89 sec)

www.percona.com

ps_helper: better PROCESSLIST# smart view to get a lot of useful information easilymysql> SELECT * FROM ps_helper.processlist WHERE conn_id = CONNECTION_ID()\G*************************** 1. row *************************** thd_id: 24 conn_id: 5 user: root@localhost db: test command: Query state: Sending data time: 0 current_statement: SELECT * FROM ps_helper.proces ... HERE conn_id = CONNECTION_ID() last_statement: NULLlast_statement_latency: NULL lock_latency: 4.98 ms rows_examined: 0 rows_sent: 0 rows_affected: 0 tmp_tables: 1 tmp_disk_tables: 0 full_scan: YES last_wait: wait/io/file/myisam/dfile last_wait_latency: 4.74 us source: mi_dynrec.c:2011 row in set (0,03 sec)

www.percona.com

ps_helper: why is it useful?Have some real fun:openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot ps_helper < procedures/create_synonym_db.sqlWarning: Using a password on the command line interface can be insecure.

mysql> call create_synonym_db('performance_schema', 'ps');Query OK, 0 rows affected (2,35 sec)

mysql> use psReading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -A

Database changedmysql> show tables;+----------------------------------------------------+| Tables_in_ps |+----------------------------------------------------+| accounts |...

Use ps_helper.format_time(SUM(sum_timer_wait)):● Check Bug #70252 to find out what may happen if you don’t use it● Check https://www.slideshare.net/Leithal/performance-schema-andpshelper

www.percona.com

ps_helper: why is it useful?

Saves a lot of keystrokes...More fancy views:

● statement_analysis● statements_with_runtimes_in_95th_percentile● statements_with_full_table_scans● statements_with_sorting● statements_with_temp_tables● …

More fancy stored procedures and functions:● dump_thread_stack() - allows to produce .dot files → .png or .pdf● format_statement()● format_time()● format_bytes()● format_path()

www.percona.com

PERFORMANCE_SCHEMA - The Good

● Monitoring is available on all platforms supported by MySQL● Activating the Performance Schema causes no (few) changes in server behavior

○ No persistent on-disk storage○ Memory allocation at server startup○ Processing happens mostly at events retrieval○ No new keywords or threads added

● Overhead is small comparing to some other ways to monitor● Problems/failures in P_S code should not prevent server from working normally● It is easy to add new instrumentation points (they say)● Under active development:

○ Very few bugs - 4 active in MySQL 5.6.16 as of February 22, 2014○ Even more features in MySQL 5.7: check Bug #69527 (MDL locks!)

● Used by Oracle internally a lot for real things (MEM, performance improvements)● As it’s based on proper ideas, eventually it’s going to become an ultimate solution

for MySQL performance monitoring (and replace everything else)

www.percona.com

PERFORMANCE_SCHEMA - The Bad

● Performance overhead in default configuration is notable in some cases (10-30%, see bug reports and 3rd party studies)

● Additional memory use is notable (see Bug #68514):○ “Why is this not a bug? On my x86-64 with 5.6.12 when

performance_schema=1 then the extra RAM is:max_connections * 196kb” - Mark Callaghan

● Mutex and rw-lock waits instrumentation must be enabled at server startup

● How complete is the instrumentation?● There are bugs and missing parts in the manual● There are (few) bugs in implementation● Is this kind of instrumentation really good? (SQL, no trace)

www.percona.com

Bugs in PERFORMANCE_SCHEMA

Bug #68413 - Facebook: “performance_schema overhead is at least 10%”Bug #71204 - Timing for stages in P_S is different from what SHOW PROFILE reportsBug #71634 - P_S digest looks wrong for system variables, shown as @ @ variable…Bug #71682 - PROCESSLIST_ID for slave sql thread is missing

Bug #68574 - “No instrumentation for InnoDB files in P_S on windows” - same with 5.6.16, by the way...Bug #68785 - new features (connection attributes) work well together (or not)...

Bug #66589 - still not fixed (do not use mysqlhotcopy). One of the last bugs I verified in Oracle...Bug #68514 - beware of memory usage impact with P_S

www.percona.com

Missing manual: verified bug reports

Bug #68097 - Manual does not explain that some P_S instruments must be enabled at startupBug #71274 - Manual does not provide enough details about background threads in P_S.THREADSBug #71276 - Manual does not explain PROCESSLIST_STATE in P_S.THREADS properlyBug #71293 - Manual page for P_S.FILE_INSTANCES table does not explain EVENT_NAME valuesBug #71303 - Manual page on P_S build configuration does not provide enough detailsBug #71304 - Manual does not provide enough details about automatic sizing of P_S parameters

These were reported by me and tagged with “missing manual”. There are more...

www.percona.com

PERFORMANCE_SCHEMA - The Ugly● Names: schema name, long table names… you know whom to blame (and thank

for ps_helper). I wish we have public synonyms in MySQL, so I’d create V$SYSSTATS (Bug #68855). You can create views, but that’s overhead, no?

● Why additional views and procedures (ps_helper) are not there by default (Bug #71207)? - going to be solved with SYS schema

● Some important things and storage engines are not properly instrumented yet, MDL locks, for example… ups, seems fixed in 5.7.3

● Heuristics to control size (read Bug #68514 again)… less than obvious and I’d prefer more control

● Make up your mind on what should be in P_S vs I_S vs SHOW statements. Think about SHOW PROCESSLIST vs I_S.PROCESSLIST vs P_S.THREADS

● Want to know what else users care about? Check feature requests for P_S (11 “Verified” as of November 2, 2013), some of them are limitations “by design”:

○ Bug #69881 - “Try to compute elapsed time for an event which has not completed” - Todd Farmer○ Bug #66515 - “I have a problem with PS instrumentation that can only be enabled at server start.” -

Mark Callaghan○ Bug #70383 - “Implement a SELECT-syntax/statement as a synonym for SHOW ENGINE

performance_schema STATUS” - Peter Laursen

www.percona.com

PERFORMANCE_SCHEMA in MySQL 5.6

Useful reading (and names to remember):● http://marcalff.blogspot.com/2013/04/on-configuring-performance-schema.html - article from Mark

Alff on how to configure PERFORMANCE_SCHEMA

● http://dev.mysql.com/doc/refman/5.6/en/performance-schema-configuration.html - RTFM

● http://dimitrik.free.fr/blog/archives/2013/07/mysql-performance-why-performance-schema-overhead.html - overhead explained and measured by Dimitri Kravtchuk:

● http://www.percona.com/webinars/using-mysql-56-performance-schema-troubleshoot-typical-workload-bottlenecks - webinar by Peter Zaitsev (May 15, 2013)

● http://mysql.wisborg.dk/category/mysql/performance_schema/ - useful posts from Jesper Krogh on the topic. He had created yet another set of tools to make out life easier, ps_tools: http://mysql.wisborg.dk/2013/10/10/mysql-connect-2013-ps_tools/

● http://mysqlblog.fivefarmers.com/tag/performance_schema/ - practical use of PERFORMANCE_SCHEMA, by Todd Farmer

● http://www.slideshare.net/SvetaSmirnova/performance-schema-for-mysql-troubleshooting - practical guide from Oracle’s Senior Principal Support engineer Sveta Smirnova

www.percona.com

Percona Live 2014, April 1-4, 2014

http://www.percona.com/live/mysql-conference-2014/home

Two sessions on PERFORMANCE_SCHEMA there:● Is performance_schema worth the overhead?● MySQL 5.7: Performance Schema Improvements ● Many more on MySQL performance in general...

www.percona.com

Thank you!

Questions and Answers

Please, report bugs at:http://bugs.mysql.comUse “Affects Me” button!