MySQL 8.0: Performance & Scalibility•Metadatais information about data in the RDBMS –Column definitions, Index definitions, Foreign key definitions... • Data Dictionarycollects

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |

MySQL 8.0:Performance & ScalibilityBill PappMySQL Solutions Architect

Copyright © 2017, Oracle and/or its affiliates. All rights reserved.

1


Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Improved UTF-8 Support


Improved UTF-8 Support in MySQL 8.0 • Support for the latest Unicode 9.0• utf8mb4 made default character set!

–utf8mb4_0900_ai_ci default collation• Accent and case sensitive collations

– Including 20+ language specific collations–Now also Japanese and Russian

• Significantly improved performance

5


What Is in MySQL 5.7 and Earlier Versions?• Default charset is “latin1” and default collation is “latin1_swedish_ci”• utf8 = utf8mb3: support BMP Plane 0 only• utf8mb4 character set:

–Only accent and case insensitive collations–Default collation is utf8mb4_general_ci, compares all characters beyond

BMP, e.g. emojis, to be equal–20+ language specific collations–Recommend to use: utf8mb4_unicode_520_ci

6


New Default Character Set• No changes to existing tables• Only has effect on new tables/schemas where character set is not

explicitly defined.• Separating character set/collation change from server upgrade

– Upgrade first, change charset/collation afterwards

• Recommend users to not mixing collations– Error “Illegal mix of collations”– Slower query because index can no longer be used

7


Transactional Data Dictionary


Data Dictionary Definition

• Metadata is information about data in the RDBMS– Column definitions, Index definitions, Foreign key definitions...

• Data Dictionary collects metadata for all things in RDBMS– User table structure, Stored program definitions...

ID NAME WEIGHT HEIGHT GENDER

3 Bob 80 185 M

5 Liz 55 165 F

Metadata

Data


MySQL Data Dictionary before MySQL 8.0

10

Data DictionaryFiles

FRM TRG OPT

System Tables (mysql.)

user procevents

InnoDB System Tables

MyISAM

File system

InnoDB

SQL


MySQL Data Dictionary before MySQL 8.0

• Performance of INFORMATION_SCHEMA• Inconsistencies between persistent storage in files and tables• Inconsistencies between InnoDB DD and Server DD• Showstopper for crash-safe / atomic DDL• Difficult for replication• Hard to extend

11

Problems


What do you get with the Transactional Data Dictionary?

• All DD tables and system tables in a DD tablespace (mysql)• Atomic DDL (this is actually a big thing!!)

– Especially important for replicated topologies– Crash situations will not end up in out-of-synch information sql-layer/SE-layer

• Reliability and Redundancy– Serialized Dictionary Information

• Better scaling INFORMATION_SCHEMA queries– Reimplemented with views over DD tables– Scaling both on database size and query load

• Improved Upgrade

12


MySQL 8.0: Transactional Data DictionaryAtomic and crash-safe DDL

13

Data Dictionary

InnoDBSQL DD TableDD TableDD Table


Transactional Data Dictionary in MySQL 8.0

• All dictionary metadata stored in tables• Single repository of metadata for SEs and the MySQL server• Reliable, crash-safe InnoDB tables• INFORMATION_SCHEMA tables implemented as VIEWs over DD tables

– Queries can be optimized by the optimizer– Improved performance– Simpler, uniform implementation, easier to maintain

Main features



• Extensible–The data dictionary schema is based on the SQL standard definitions–The data dictionary is designed to be easily extended for new

requirements–Designed to provide automated upgrade of dictionary data–Designed to allow plugins to add and extend system tables, I_S tables,

Performance Schema tables

Main features, cont’ed



16

Overview

InnoDB

Data Dictionary

DD Table User Table

INFORMATIONSCHEMA

Views

Archive

User Table

CSV

User Table

Optimizer


Transactional Data Dictionary API in MySQL 8.0

• The only way to deal with Data Dictionary– For the server core– For SEs– For other plugins

• Less Error-prone• Internal API (non-stable) and external API (stable)• Provide a way to handle SE private data

17

Design goals


InnoDB


18

Architecture

Query Executor

Parser Optimizer

Data Dictionary Tablespace

Data Dictionary Internal API

Internal SE

Data Dictionary External API

Plugin

Storage Engine

Plugin

User TableTablespace


Reliability and Redundancy



SDI: Serialized Dictionary Information- Used for data migration and is a redundant write-only copy in normal

operationThe InnoDB Data Dictionary tablespace ”mysql” is the metadata storage

20

Reliability and Redundancy

InnoDB User TablespacesInnoDB Data Dictionary tablespace

mysql.tables

SDI

ID Name

1 User Table 1

2 User Table 2

User Table


Example - DROP SCHEMA at high level

21

MySQL 5.7• Delete tables

– Metadata, TRN/TRG/FRM files– Data, InnoDB tables

• Delete stored programs– Metadata, rows in MyISAM (non-

transactional)• Delete schema

– Metadata, DB.OPT fileMix of filesystem, non-transactional/transactional storage and multiple commits

MySQL 8.0• Delete tables

– Metadata, rows in InnoDB– Data, InnoDB tables

• Delete stored programs– Metadata, rows in InnoDB

• Delete schema– Metadata, rows in InnoDB

Updates to transactional storage, one commit


Reliability – Disaster Recovery

• All data stored in InnoDB (transactional SE)– Metadata stored in an SDI embedded in .ibd– Data stored in .ibd

• InnoDB logging DDL operations as «Dictionary Storage Engine»– After restart and recovery, the DDL operation will be complete or rolled back

22

Use case – IMPORT using Serialized Dictionary Information


Reliability – Automated Dictionary Upgrade

• The Data Dictionary definition will have a version attached• The MySQL Server executable will support upgrading dictionary tables

23

Use case – Upgrade


Reliability – Protecting Dictionary Tables

• Data Dictionary tables are protected against user DML– This means there will be no possibility for corruption from user DML on these tables– All information in the dictionary tables will be available through

INFORMATION_SCHEMA

24


CREATE PROCEDURE p1(v INT)SQL SECURITY INVOKERBEGIN...END

Data Dictionary

Data Dictionary

Table Definitions SP Definitions

View Definitions Plugins

ACL

CREATE TABLE customers(id INT AUTO_INCREMENT...PRIMARY KEY (id),INDEX ...FOREIGN KEY ...

)


Hot Row Contention Improvements


SELECT ... FOR UPDATE SKIP LOCKED• Common problem:

– Hot row contention, multiple worker threads accessing the same rows

• Solution 1:– Only read rows that are not locked – InnoDB skips a locked row, and the next one goes to the result set

• Example:– Booking system: Skip orders that are pending

27

START TRANSACTION;SELECT * FROM seats WHERE seat_no BETWEEN 2 AND 3 AND booked = 'NO'

FOR UPDATE SKIP LOCKED;


SELECT… FOR UPDATE NOWAIT• Common problem:

– Hot row contention, multiple worker threads accessing the same rows

• Solution 2:– If any of the rows are already locked, the statement should fail immediately– Without NOWAIT, have to wait for innodb_lock_wait_timeout (default: 50 sec) while

trying to acquire lock

• Usage:

28

START TRANSACTION;SELECT * FROM seats WHERE seat_no BETWEEN 2 AND 3 AND booked = 'NO'

FOR UPDATE NOWAIT;ERROR 3572 (HY000): Statement aborted because lock(s) could not be acquired …

https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout


Index Extensions


Invisible Index• Index is maintained by the SE, but ignored by the Optimizer• Primary key cannot be INVISIBLE• Use case: Check for performance drop BEFORE dropping an index

• To see an invisible index: set optimizer_switch='use_invisible_indexes=on';

30

ALTER TABLE t1 ALTER INDEX idx INVISIBLE;mysql> SHOW INDEXES FROM t1;+---------+------------------+----------------------+---------------+| Table | Key_name | Column_name | Visible | +---------+------------------+----------------------+---------------+| t1 | idx | a | NO |+---------+------------------+----------------------+---------------+


Descending Index

• In 5.7: Index in ascending order is created, server scans it backwards• In 8.0: Index in descending order is created, server scans it forwards• Works on B-tree indexes only• Benefits:

– Use indexes instead of filesort for ORDER BY clause with ASC/DESC sort key– Forward index scan is slightly faster than backward index scan

31

CREATE TABLE t1 (a INT,b INT,INDEX a_b (a DESC, b ASC)

);


Cost Model Improvements


Motivation for Improving the MySQL Cost Model• Produce more correct cost estimates

– Better decisions by the optimizer should improve performance

• Adapt to new hardware architectures– SSD, larger memories, caches

• More maintainable cost model implementation– Avoid hard coded “cost constants”– Refactoring of existing cost model code

• Configurable and tunable• Make more of the optimizer cost-based

Fasterqueries

33


• Time to do a table scan of 10 million records:

• Adjust cost model to support different storage technologies

• Provide configurable cost constants for different storage technologies

New Storage Technologies

Memory 5 sSSD 20 - 146 sHard disk 32 - 1465 s

Provide a program that could measure performance and suggest good cost constant configuration for a running

MySQL server?

34


• Storage engines:– Estimate for how much of data and

indexes are in a memory buffer– Estimate for hit rate for memory buffer

• Optimizer cost model:– Take into account whether data is

already in memory or need to be read from disk

Memory Buffer Aware Cost Estimates

Server

Storageengine

Disk data

Queryexecutor

Database buffer

35


MySQL 8.0: Disk vs Memory Access• New defaults for cost constants:

• InnoDB reports for each table/index percentage of data cached in buffer pool

• Note: Query plan may change between executions

Confidential – Oracle Internal/Restricted/Highly Restricted 36

Cost MySQL 5.7 MySQL 8.0Read a random disk page 1.0 1.0Read a data page from memory buffer 1.0 0.25Evaluate query condition 0.2 0.1Compare keys/rows 0.1 0.05


Execution time (MySQL 8.0.3)

In-memory Disk-boundPlan A 5.8 secs 9 min 47 secsPlan B 77.5 secs 3 min 49 secs

Selected plan

In-memory Disk-boundMySQL 5.6 Plan B MySQL 5.7 Plan AMySQL 8.0 Plan A Plan B

39

DBT-3 Query 8

DBT-3 Scale factor 10In-Memory: innodb_buffer_pool_size = 32 GBDisk-bound: innodb_buffer_pool_size = 1 GB


Histograms

• Provides the optimizer with information about column value distribution• To create/recalculate histogram for a column:

ANALYZE TABLE table UPDATE HISTOGRAM ON column WITH n BUCKETS;

• May use sampling– Sample size is based on available memory (histogram_generation_max_mem_size)

• Automatically chooses between two histogram types:– Singleton: One value per bucket– Equi-height: Multiple value per bucket

40

Column statistics


Histograms

EXPLAIN SELECT * FROM customer JOIN orders ON c_custkey = o_custkeyWHERE c_acctbal < -1000 AND o_orderdate < '1993-01-01';

41

Example query

id select type table type possible keys key key

len ref rows filtered extra

1 SIMPLE orders ALL i_o_orderdate, i_o_custkey NULL NULL NULL 15000000 31.19 Using

where

1 SIMPLE customer eq_ref PRIMARY PRIMARY 4 dbt3.orders.

o_custkey 1 33.33 Using where


Histograms

ANALYZE TABLE customer UPDATE HISTOGRAM ON c_acctbal WITH 1024 buckets;

EXPLAIN SELECT * FROM customer JOIN orders ON c_custkey = o_custkeyWHERE c_acctbal < -1000 AND o_orderdate < '1993-01-01';

42

Create histogram to get a better plan



1 SIMPLE customer ALL PRIMARY NULL NULL NULL 1500000 0.00 Using where

1 SIMPLE orders ref i_o_orderdate, i_o_custkey i_o_custkey 5

dbt3.customer.c_custkey

15 31.19 Using where


Contention-AwareTransaction Scheduling(CATS)


What is Contention-Aware Transaction Scheduling?

• Nearly all database systems rely on some variant of the First-In-First-Out (FIFO) policy.

• In a nutshell, FIFO grants locks to those who requested it first (i.e., transactions that are ahead of the queue, unless they’re incompatible with the locks that are currently granted).

• CATS is a new algorithm (developed by PhD Students at the University of Michigan) which can dramatically reduce latency and increase throughput when used in place of a FIFO approach.

• .44


How Does CATS Work?

• The CATS algorithm is based on a simple intuition: not all transactions are equal, and not all objects are equal.

• When a transaction already has a lock on many popular objects, it should get priority when it requests a new lock.

• In other words, unblocking such a transaction will indirectly contribute to unblocking many more transactions in the system, which means higher throughput and lower latency overall.

45


Hints


Hints: Join Order • Hints to control table order for join execution• 5.7: STRAIGHT_JOIN to force the listed order in FROM clause• 8.0:

– JOIN_FIXED_ORDER /* replacement for STRAIGHT_JOIN*/– JOIN_ORDER /* use specified order */– JOIN_PREFIX /* use specified order for first tables */– JOIN_SUFFIX /* use specified order for last tables */– No need to reorganize the FROM clause to add join order hints like you will for

STRAIGHT_JOIN

47


Join Order Hints - Example

EXPLAIN SELECT /*+ JOIN_ORDER(customer, orders) */ * FROM customer JOIN orders ON c_custkey = o_custkeyWHERE c_acctbal < -1000 AND o_orderdate < '1993-01-01';

48

Change join order with hint



1 SIMPLE customer ALL PRIMARY NULL NULL NULL 1500000 33.33 Using where

1 SIMPLE orders ref i_o_orderdate, i_o_custkey i_o_custkey 5

dbt3.customer.c_custkey

15 31.19 Using where

Alternatives with same effect for this query:JOIN_PREFIX(customer) JOIN_SUFFIX(orders) JOIN_FIXED_ORDER()


Comparing Join Order

0

2

4

6

8

10

12

14

16

Que

ry E

xecu

tion

Tim

e (s

econ

ds)

orders → customer customer → orders

Performance


Hints: Index Merge• Index merge: Merge rows from multiple range scans on a single table• Algorithms: union, intersection, sort union• Users can specify which indexes to use for index merge

– /*+ INDEX_MERGE() */– /*+ NO_INDEX_MERGE() */

50

10INDEX(a)

10INDEX(b)

a=10 AND b=10Result:

Intersection

SELECT * FROM t1 WHERE a=10 AND b=10


Index Merge Hint - ExampleEXPLAIN SELECT count(*) FROM users WHERE user_type=2 AND status=0 AND parent_id=0;

51

id select type table type possible

keys key key len ref rows Extra

1 SIMPLE users index_merge

parent_id, status, user_type

user_type, status, parent_id

1,1,4 NULL 2511Using intersect (user_type, status, parent_id);Using where; Using index

mysql> SELECT count(*) FROM users WHERE user_type=2 AND status=0 AND parent_id=0;...1 row in set (1.37 sec)mysql> SELECT /*+ INDEX_MERGE(users user_type, status) */ count(*)

-> FROM users WHERE user_type=2 AND status=0 AND parent_id=0;...1 row in set (0.18 sec)

Low selectivity


Hints: Set session variables• Set a session variable for the duration of a single statement• Examples:

SELECT /*+ SET_VAR(sort_buffer_size = 16M) */ name FROM people ORDER BY name;INSERT /*+ SET_VAR(foreign_key_checks = OFF) */ INTO t2 VALUES (1, 1), (2, 2), (3, 3);SELECT /*+ SET_VAR(optimizer_switch = 'condition_fanout_filter = off') */ * FROM customer JOIN orders ON c_custkey = o_custkeyWHERE c_acctbal < 0 AND o_orderdate < '1993-01-01';

• NB! Not all session variables are settable through hints:mysql> SELECT /*+ SET_VAR(max_allowed_packet=128M) */ * FROM t1;Empty set, 1 warning (0,01 sec)

Warning (Code 4537): Variable 'max_allowed_packet' cannot be set using SET_VAR hint.

52


Better IPv6 and UUID Support


Improved Support for UUID

• Five “-” separated hexadecimal numbers • MySQL uses version 1, the first three numbers are generated from the low,

middle, and high parts of a timestamp.• 36 characters, inefficient for storage

Convert to BINARY(16) datatype, only 16 bytes

54

mysql> select uuid();+---------------------------------------------------------+| uuid() |+---------------------------------------------------------+| aab5d5fd-70c1-11e5-a4fb-b026b977eb28 |+---------------------------------------------------------+


Improved Support for UUID

• UUID_TO_BIN(string_uuid, swap_flag) • BIN_TO_UUID(binary_uuid, swap_flag)• IS_UUID(string_uuid)

55

Functions to convert UUID to and from binary

Feature Requestfrom Developers


UUID_TO_BIN Optimization• Binary format is now smaller• Shuffling low part with the high part improves index performance

56

25 26 27 28 29

Insert Performance OptimizedOriginal

11e678fe53303f87a4778c89a52c4f3b

53303f87-78fe-11e6-a477-8c89a52c4f3bFrom VARCHAR(36)

To BINARY(16)


IPv6 • New! Bit-wise operations on binary data types

– Designed with IPv6 in mind:– INET6_ATON(address) & INET6_ATON(network)– No longer truncation beyond 64 bits

57



• Improved Query Scan Performance• GROUPING()• Hint: Merge/materialize derived

table/view • JSON:

– Partial update– Improved performance of sorting and

grouping of JSON values– Path expressions: Array index ranges– JSON_MERGE_PATCH()

• Skip index dives for FORCE INDEX • Parser Refactoring

58

All These Features and More…

Try it out!

Give us feedback!


New INFORMATION_SCHEMA in MySQL 8.0


INFORMATION_SCHEMA in MySQL 8.0

• New option: information_schema_stats_expiry– Default 24h– Cache dynamic values in mysql.index_stats and mysql.table_stats– Set to 0, always fetch data from SE and no «stats» will be stored– ANALYZE will update mysql.index_stats and mysql.table_stats

• Improved performance fetching dynamic data from SE (InnoDB)– Typically 4-5 times faster than 5.7 (but note that this is only relevant for a few values)

60


NEW INFORMATION_SCHEMA in MySQL 8.0

61

Uniform, simpler implemention makes it a lot faster

MySQL Client

I_S Query Results

MySQL Server

Optimizer preparesexecution plan.

Executor reads metadata fromdata dictionary tables.

InnoDB storage engine

Return rows to user.

INFORMATION_SCHEMA in 8.0

MySQL Client

I_S Query Results

MySQL Server

Create temporary table.

Heuristic optimization.

Read metadata from File system orfrom MyISAM/InnoDB engine.

.

TEMP TABLE

Return rows to user.

INFORMATION_SCHEMA in 5.7

File system / MyISAM/ InnoDB engine


INFORMATION_SCHEMA Performance and Scalability

• Typically 30X performance improvements over MySQL 5.7• More than 100X for some queries like: List all InnoDB table columns

62

I_S queries scale, both with database size and query load

0 20 40 60 80 100 120 140 160

List all InnoDB tables columns 5k tables

List all InnoDB tables columns 10k tables

MySQL 8.0MySQL 5.7

Time in Seconds (Lower is better)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 63

100 schemas times 50 tables (5000 tables)

INFORMATION_SCHEMA Performance

0 0.5 1 1.5 2 2.5 3 3.5 4

Count All Schemas

Schema aggregate size stats

All Dynamic Table Info

All Static Table Info

Auto Increments Near Limit

Count All Columns

Count All Indexes

MySQL 8.0MySQL 5.7



Performance and Scalability Improvements

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal/Restricted/Highly Restricted

MySQL Performance SchemaEvolution form 5.5 to 8.0

65

MySQL 5.5

• Event Waits

• Mutexes

• Files

• Threads

MySQL 5.6

• Statement Instrumentation

• Lower Overhead

MySQL 5.7

• Memory Instrumentation

• Prepared Statements Instrumentation

• Transactions Instrumentation

• Scalable Memory Allocation

• Bundled SYS schema

• Lower Overhead

MySQL 8.0

• Histograms

• Indexes

• Data Locks instrumentation

• SQL Errors instrumentation

• Variables Instrumentation

• Table plugin

• Improved Defaults


Performance Schema: Benchmarks

0. 10. 20. 30. 40.

Query Time MySQL 8.0MySQL 5.7

SELECT * FROM sys.session1000 active sessions


Over 30x Faster


RESOURCE GROUP

• New mechanism to control resource usage for DML load– Can be used to assign execution threads to CPUs– Use “hints” to assign to a resource group per query

• RESOURCE GROUP management– CREATE/DROP groups– Assign priority– Assign CPUs– Enable/disable

• Supported in INFORMATION_SCHEMA and PERFORMANCE_SCHEMA

67


MySQL 8.0: Resource Group Example

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

No Resource Group With Resource Group

Que

ries p

er S

econ

d

Select

Update

System Configuration :Oracle Linux 7, Intel(R) Xeon(R) CPU E7-4860 2.27GHz 40 cores-HT

(40 Cores Shared) (40 Cores for Select)(10 Cores for Update RG)


MySQL 8.0 BenchmarksCopyright © 2015, Oracle and/or its affiliates. All rights reserved.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 73

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

1,000,000

1 2 4 8 16 32 64 128 256 512 1,024

Que

ries p

er S

econ

d

Users

MySQL 8.0

MySQL 5.7

MySQL 5.6

MySQL 8.0: SysBench Read Write (Mixed)30% Faster than MySQL 5.7

OS : Oracle Linux 7.4CPU : 48cores-HT Intel Skylake 2.7Ghz (2CPU sockets, Intel(R) Xeon(R) Platinum 8168 CPU)RAM: 256GB Storage : x2 Intel Optane flash devices(Intel (R) Optane (TM) SSD P4800X Series)


0

50,000

100,000

150,000

200,000

250,000

300,000

1 2 4 8 16 32 64 128 256 512 1,024

Que

ries p

er S

econ

d

Users

MySQL 8.0

MySQL 5.7

MySQL 5.6

MySQL 8.0: SysBench Read Write (update nokey)2x Faster than MySQL 5.7

Oracle Confidential – Internal/Restricted/Highly Restricted 74



MySQL 8.0: SysBench IO Bound Read Only (Point Selects)2x Faster than MySQL 5.7


0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1 2 4 8 16 32 64 128 256 512

Que

ries p

er S

econ

d

Users

MySQL 8.0

MySQL 5.7

MySQL 5.6



0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

1,000,000

1 2 4 8 16 32 64 128 256 512 1,024

Que

ries p

er S

econ

d

Users

MySQL 8.0

MySQL 5.7

MySQL 5.6

MySQL 8.0: SysBench OLTP Read Only (Mixed - utf8mb4)40% Faster than MySQL 5.7




MySQL 8.0 Overview

Bill PappPrincipal Solutions Architect



MySQL Innovation: 5.7 -> 8.0

- 3x Better Performance- Replication Enhancements- Optimizer Cost Model- JSON Support- Improved Security- Sys & Performance Schema- GIS

MySQL 5.7 (GA)

MySQL InnoDB Cluster (GA)- MySQL Group Replication- MySQL Router- MySQL Shell

MySQL 8.0- Document Store- Data Dictionary- Roles- Unicode- CTEs- Window Functions- Security- Replication- SysSchema- GIS

2 Years in Development400+ Worklogs5000+ Bugs Fixed500 New Tests

8.0

GA

24x7 at Scale

Mobile First

Developer First

Data Driven


MySQL Document Store

MySQLRelational Tables Foreign Keys

NoSQLJSON DocumentsSchemaless JSON Collections

X Dev API SQL CRUD


NoSQLJSON DocumentsSchemaless JSON Collections

MySQLRelational TablesForeign Keys

X Dev APISQL

CRUD

MySQLDocument

Store


#1 New Feature: MySQL Document Store

An easy, straight forward way to work with JSON documents in MySQL

91

#1 New Feature


Document Oriented Databases

• Schemaless: No centralized database schema– Data model enforcement and validation (if any) at application layer– Simpler schema updates (no ALTER TABLE penalty)

• NoSQL APIs: Simpler programming interfaces– No specialized language for queries and data manipulation– Complex queries handled at application layer (no complex SELECTs, JOINs)– Document in, document out, manipulations at client side

• Scalability, but some drawbacks:– Limited database features (no foreign keys, no transactions, etc.)– Weak consistency guarantees

92

Usability & Scalability


NoSQL, MySQL, Why not…• Have both schema-less and schema in the same technology stack?• One that checks all the boxes of all stakeholders:

93

Developers✔ Schemaless or/and Schema✔ Rapid Prototyping/Simpler APIs✔ Document Model✔ Transactions

Operations✔ Performance Management/Visibility✔ Robust Replication, Backup, Restore✔ Comprehensive Tooling Ecosystem✔ Simpler application schema upgrades

Business Owner✔ Don’t lose my data = ACID transactions✔ Capture all my data = Extensible/Schemaless✔ Help Comply with Regulations = ‘Make Sure It’s Secure!’✔ Products On Schedule/Time to Market = Rapid Development


MySQL Document Store: Components• MySQL X Plugin

• Introduces X Protocol for relational- and document operations

• Maps CRUD operations to standard SQL (relational tables, JSON datatype and functions)

• X Protocol• New MySQL client protocol based on top of

industry standard (Protobuf)• Works for both, CRUD and SQL operations

• InnoDB Cluster• Read-Scaling, Write-Scaling, HA

• X DevAPI• New, modern, async developer API for CRUD

and SQL operations on top of X Protocol• Introduces Collections as new Schema obj.

• MySQL Shell• Offers interactive X DevAPI mode for app

prototyping

• MySQL Connectors• Support for X DevAPI for

• JavaScript, Python, PHP, Java, C#, C++

94


How does the Document Store work?Architecture from the Application’s POV

95

CRUD requests + JSON

JSON

Frontend Backend

Application

MySQL

Document Store


How does the MySQL Document Store work?Architecture & Components

96

Application Connector MySQLX PluginDevAPI Protobuf / X Protocol / TCP/IP SQL

InnoDB


Read Scale-Out

Async Replication + Auto Failover

Write Scale-Out

Sharding

S1

S2

S3

S4

MySQL Mission – 4 Steps

Timeline

MySQL Document Store

Relational & Document Model

MySQL HA

Out-Of-Box HA

✔ ✔

✔


MySQL Shell

• Scripting for Javascript, Python, and SQL mode• Supports MySQL Standard and X Protocols• Document and Relational Models• CRUD Document and Relational APIs via scripting• Traditional Table, JSON, Tab Separated output results formats• Both Interactive and Batch operations

Interface for Development and Administration of MySQL


MySQL Shell: What’s New• Customizable prompt

• Include context and session information

• Custom font and color support

• Persistent command line history

• Auto-complete / Content Assistance

• Full Unicode support

99


JSON


JSON Support• Native File Format• Virtual Columns• 20+ Functions• New! Search Functions• New! Aggregations Functions

– Query structured data and semi-structured JSON data


JSON Functions

102

MySQL 5.7 and 8.0

JSON_ARRAY_APPEND()

JSON_ARRAY_INSERT()

JSON_ARRAY()

JSON_CONTAINS_PATH()

JSON_CONTAINS()

JSON_DEPTH()

JSON_EXTRACT()

JSON_INSERT()

JSON_KEYS()

JSON_LENGTH()

JSON_MERGE[_PRESERVE]()

JSON_OBJECT()

JSON_QUOTE()

JSON_REMOVE()

JSON_REPLACE()

JSON_SEARCH()

JSON_SET()

JSON_TYPE()

JSON_UNQUOTE()

JSON_VALID()

JSON_PRETTY()

JSON_STORAGE_SIZE()

JSON_STORAGE_FREE()

JSON_ARRAYAGG()

JSON_OBJECTAGG()

JSON_MERGE_PATCH()

JSON_TABLE()

https://dev.mysql.com/doc/refman/5.7/en/json-functions.html


UTF8 / Unicode


• Emoji characters used as input • MySQL 8.0 defaults to utf8mb4• Latest Unicode 9.0 Support• New collations based on DUCET,

accent and case sensitive collations, Japanese, Russian

MySQL 8.0: UTF-8 Encoding


0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

1,000,000

1 2 4 8 16 32 64 128 256 512 1,024

Que

ries p

er S

econ

d

Users

MySQL 8.0

MySQL 5.7

MySQL 5.6

MySQL 8.0: SysBench OLTP Read Only (Mixed - utf8mb4)40% Faster than MySQL 5.7




MySQL 8.0: Boosts Developer & DevOps Productivity• New! CTEs and Recursive CTEs• New! Window Functions• New! SKIP LOCKED and NOWAIT• UUIDs and Bitwise Functions

106


New! Invisible Indexes• Indexes are “hidden” to the MySQL Optimizer

– Not the same as “disabled indexes”– Contents are fully up to date and maintained by DML

• Two use cases:– Soft Delete (Recycle Bin)– Staged Rollout

107

Feature Requestfrom DBAs


MySQL 8.0: New SQL Syntax• Two of our most requested features:

– Common Table Expressions (CTEs)– Window Functions

108



High Availability


• Virtually all organizations require their most critical systems to be highly available

110

100%


InnoDB Cluster

111

App Servers withMySQL Router

MySQL Group Replication

MySQL ShellSetup, Manage,

Orchestrate

“High Availability becomes a corefirst class feature of MySQL!”


MySQL Shell: DBA Admin API• The global variable 'dba' is used to access the

MySQL AdminAPI• mysql-js> dba.help()

• Perform DBA operations– Manage MySQL InnoDB clusters

• Create clusters• Validate MySQL instances • Configure MySQL instances• Get cluster info • Modify clusters• and much more ...




Orchestrate

112


MySQL Group Replication: Database HA• Group Replication library

– Implementation of Replicated Database State Machine• MySQL GCS is based on our home-grown Paxos implementation

– Provides virtually synchronous replication for MySQL 5.7+• Guarantees eventual consistency

– Automates operations • Conflict detection and resolution• Failure detection, fail-over, recovery• Group membership management and reconfiguration

“Multi-master update anywhere replication plugin for MySQL with built-in conflict detection and resolution, automatic distributed recovery, and group membership.”

113




Orchestrate

http://www.inf.usi.ch/faculty/pedone/Paper/2002/2002DPDT.pdf

https://en.wikipedia.org/wiki/Paxos_(computer_science)


Data Dictionary


New! Transactional Data Dictionary• Crash-safe Database

– Common data dictionary for the server and InnoDB

• Crash-safe & Atomic DDL– CREATE USER <userlist>, DROP

DATABASE with all-or-none semantic– Simplifies replication failure cases

116

• Meta-data locking for FK constraints–FK moved from InnoDB to server

layer• Scalable Information Schema

– Queries are now executed as set of SQL views on tables

– Large performance improvements


Performance Improvements


New! Performance Improvements• Improved Query Consistency

• Histograms• Improved Cost Model

• Faster Table/Range Scans

119

• Parallel Replication• UTF8MB4• Information Schema• Performance Schema Indexes


0

50,000

100,000

150,000

200,000

250,000

300,000

1 2 4 8 16 32 64 128 256 512 1,024

Que

ries p

er S

econ

d

Users

MySQL 8.0

MySQL 5.7

MySQL 5.6

MySQL 8.0: SysBench Read Write (update nokey)2x Faster than MySQL 5.7




MySQL 8.0: SysBench IO Bound Read Only (Point Selects)2x Faster than MySQL 5.7


0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1 2 4 8 16 32 64 128 256 512

Que

ries p

er S

econ

d

Users

MySQL 8.0

MySQL 5.7

MySQL 5.6



Security


• Improving MySQL Access Controls

• Easier to manage user and applications rights

• As standards compliant as practically possible

• Multiple default roles

• Export the role graph in GraphML

123

Directly

IndirectlySet Role(s)

Default Role(s)Set ofACLS

Set ofACLS

New! MySQL 8.0: RolesFeature Request

from DBAs


New! MySQL 8.0: Dynamic PrivilegesProvides finer grained administrative level access controls• Too often super is required for tasks when less privilege is really needed

– Support concept of “least privilege”

• Needed to allow adding administrative access controls – Now can come with new components– Examples

• Replication• HA • Backup

• Give us your ideas

124


MySQL Password Features• New! Password history - provides DBAs more password management

– Require new passwords not reuse old ones - By number of changes and/or time. – Establish password-reuse policy globally as well as on a per-account basis.

• New! SHA2 with Caching– Strong and Fast– Strong - SHA-256 password hashing (many rounds, seeds, …) – Fast - Caching

• Greatly reduces latency

• New! Supports for more connection protocols• New! Seamless RSA password-exchange capabilities (No linking OpenSSL)

125


MySQL 8.0: File Encryption• New! AES 256 encryption of UNDO and REDO Logs• Super Simple to manage - Set

– innodb_undo_log_encrypt=ON/OFF– innodb_redo_log_encrypt=ON/OFF

• And– ON - Pages written after setting are encrypted– OFF - Pages written after setting are not.

126


MySQL 8.0: Upgrade Checker

130

• Quick and Easy MySQL Shell Utility– JavaScript– Python

• Identifies Issues Based on Severity– No Issues– Potential Errors– Errors that must be fixed before Upgrading

• Recommends Fixes – Schema, Configuration – Data on Server, etc.