New data dictionary an internal server api that matters

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

New Data Dictionary:An Internal Server APIThat Matters

Alexander Nozdrin, Principle Software Developer

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL Community Reception @ Oracle OpenWorldMingle with the MySQL community and the MySQL team from Oracle for a fun and informative evening!

• Time: September 30 (Tue) @ 7pm• Jillian’s at Metreon 175 Fourth Street, San Francisco, CA

At the corner of Howard and 4th st.; only 2-min walk from Moscone Center

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Agenda

What is a Data Dictionary?

The MySQL Traditional Data Dictionary

New Data Dictionary

Benefits for users

Q & A

1

2

3

4

5

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Data Dictionary

What is it

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Data Dictionary

• Metadata is information about user data– User table structure– Column definitions– Index definitions– Foreign key definitions– Stored program definitions...

• Data Dictionary collects all metadata in RDBMS

Definition

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

CREATE PROCEDURE p1(v INT)SQL SECURITY INVOKERBEGIN...END

Data DictionaryOverview

Data Dictionary

Table Definitions SP Definitions

View Definitions Plugins

Time zones

Privileges

CREATE TABLE customers( id INT AUTO_INCREMENT ... PRIMARY KEY (id), INDEX ... FOREIGN KEY ...)

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 8

MySQL Server

Data Dictionary DefinitionThe ecosystem

Query Executor

Optimizer

PerformanceSchema InnoDBSE

SQLstatement

Client Parser

Result

Information Schema

DataDictionary

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

The MySQL Traditional Data Dictionary

MySQL 5.6 and earlier

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

The MySQL Traditional Data Dictionary• A mix of files and tables:– File based• Tables: FRM• Triggers: TRN, TRG...

– Table based• mysql.time_zone...

• InnoDB has a separate data dictionary

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 11

The MySQL Traditional Data DictionaryData Dictionary

Files

FRM TRG OPT

System tables (mysql.)

user proctime_zone

InnoDB internal data dictionary

CSV

Archive

InnoDB

INFO

RMATIO

N_SCH

EMA

File Scan

Table Scan

Intern.Access

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 12

The MySQL Traditional Data Dictionary

• Poor INFORMATION_SCHEMA performance• Makes crash-safe / transactional DDL impossible• Inconsistencies between files and tables• Inconsistencies between DD in InnoDB and the server• File-system dependency (lower-case-table-names)• Makes replication of DDL statements difficult• Too difficult to extend

Problems

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

New Data Dictionary

A great leap forward

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

• Stored in InnoDB tables• Reliable & crash-safe• Single repository of metadata– for the MySQL server – for Storage Engines– for Plugins

• Redundancy• Data Dictionary API

• INFORMATION_SCHEMA SQL VIEWs–Queries can be optimized– Improved performance

• Metadata versioning• Extendable– Simplify metadata upgrades– Designed with plugins in mind

New Data Dictionary : Main Features

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 15

New Data DictionaryTransition

Data DictionaryFiles

FRM TRG OPT

System tables (mysql.)

user proctime_zone

InnoDB internal dictionary

CSV

Archive

InnoDBINFO

RMAT

ION

_SCH

EMA

DD TableSQL VIEW

File Scan

Table Scan

Intern.Access

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 16

New Data DictionaryOverview

InnoDB

Data Dictionary

DD Table User Table

INFORMATIONSCHEMA

Views

Archive

User Table

CSV

User Table

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 17

InnoDB

New Data DictionaryArchitecture

Query Executor

Parser Optimizer

Data Dictionary Tablespace

Data Dictionary Internal API

Data Dictionary External API

Plugin Plugin

Plugin Plugin Plugin

Archive

User TableTablespace Storage Engine

Storage Engine

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 18

New Data Dictionary

• WL#6379: Schema definitions for new DD• InnoDB Data Dictionary Tablespace• Designed with INFORMATION_SCHEMA in mind• Ability to store SE-specific data• Use PK / FK to ensure consistency

Data Dictionary tables

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 19

New Data Dictionary API

• The only way to access Data Dictionary– For the server core– For Storage Engines– For plugins

• Hard to misuse• Internal API (non-stable) and external API (stable)• Provide a way to handle SE specific data

Design goals

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 20

New Data Dictionary

SDI : Serialized Dictionary Information

Redundancy

InnoDB

Single User TS

User Table

SDI

General TS

User Table

SDI

System TS

User Table

SDI

Data Dictionary

User Table Definition

Stored Program

Privileges

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 21

New Data DictionaryFRM shipping for MySQL Cluster?

SELECT ... FROM t1

FRMSDI

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 22

• No FRM files• New INFORMATION_SCHEMA• Migrated to InnoDB:– time zone tables– help tables–mysql.plugins–mysql.servers

• Draft of Data Dictionary API

• http://labs.mysql.com– Do NOT use it in production– Install on a spare server

• MTR can be run

New Data Dictionary : Labs Release

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

New Data Dictionary

Why does it matter for YOU?

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 24

INFORMATION_SCHEMA performance improvementsGet per table size Blog post by Shlomi Noach: http://tinyurl.com/y8cnj7o

SELECT TABLE_SCHEMA, TABLE_NAME, ENGINE,SUM(DATA_LENGTH+INDEX_LENGTH) AS size,SUM(INDEX_LENGTH) AS index_size

FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA NOT IN ('mysql', 'INFORMATION_SCHEMA') AND ENGINE IS NOT NULLGROUP BY TABLE_SCHEMA, TABLE_NAME;

Version Time

5.7.5-m15 0.38 sec

Labs Release 0.08 sec

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 25

Case: server crash

• There are some data files• No / outdated backup• FRM files lost• How to use those data files?

The problem

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 26

Traditional Data Dictionary• “Move FRM files around”• CREATE TABLE t1 (...)

• ALTER TABLE t1 DISCARD TABLESPACE

• ALTER TABLE t1 IMPORT TABLESPACE ...

• Easy to make mistakes

New Data Dictionary• Self-descriptive tablespaces (SDI)• Dedicated IMPORT statement• Goal: error-proof procedure

Case: server crash

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 27

For Plugin Developers• A way to access Data Dictionary• Persistent Storage for plugins– Store/restore custom data

• Plugins can extend INFORMATION_SCHEMA & PERFORMANCE_SCHEMA– Add new tables– Add new columns to existing tables

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Data Dictionary

Summary

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 29

Data Dictionary : Takeaways• Fundamental component in RDBMS• Critical for performance• Critical for reliability• Critical for scalability

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

The MySQL TraditionalData Dictionary

• Mix of files and tables• Server DD and InnoDB DD• Inefficient INFORMATION_SCHEMA• Difficult to extend

New Data Dictionary

• Crash-safe InnoDB tables• Single repository• INFORMATION_SCHEMA as VIEWs• Designed to be extendable• Aims for backward compatibility• Huge reengineering

Data Dictionary : Takeaways

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Questions?

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL Community Reception @ Oracle OpenWorldMingle with the MySQL community and the MySQL team from Oracle for a fun and informative evening!

• Time: September 30 (Tue) @ 7pm• Jillian’s at Metreon 175 Fourth Street, San Francisco, CA

At the corner of Howard and 4th st.; only 2-min walk from Moscone Center

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Oracle University MySQL Training ServicesPrepare Your Organization to Enable Reliable and High-Performance Web-Based Database Applications

“Training and team skill have the most significant impact on overall performance of technology and success of technology projects.” - IDC, 2013

Premier Support customers eligible to save 20% on learning credits.

Benefits

Expert-led training to support your MySQL learning needs Flexibility to train in the classroom or online Hands-on experience to gain real world experience Key skills needed for database administrators and developers

•MySQL for BeginnersMySQL for Database AdministratorsMySQL Performance TuningMySQL Cluster – NEW - Register Your Interest!MySQL and PHP - Developing Dynamic Web ApplicationsMySQL for DevelopersMySQL Developer Techniques

MySQL 5.6 Database Administrator MySQL 5.6 Developer

To find out more about available MySQL Training & Certification offerings, go to: education.oracle.com/mysql

Top Courses for Administrators and Developers

Top Certifications

RECENTLY RELEASEDALL NEW! MySQL Cluster Training To Register your interest to influence the

schedule on this newly released course – go to education.oracle.com/mysql and click on the

MySQL Cluster Course

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Thank You!

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.