Copyright © 2016, Oracle and/or its affiliates. All rights reserved. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Manyi Lu Director MySQL Optimizer Team, Oracle April, 2018 Combining SQL and NoSQL with MySQL
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.
Manyi Lu
DirectorMySQL Optimizer Team, OracleApril, 2018
Combining SQL and NoSQL withMySQL
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Agenda
3
JSON data type
JSON functions
Indexing JSON data
Partial update of JSON
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE employees (data JSON);INSERT INTO employees VALUES('{"id": 1, "name": "Jane"}'), ('{"id": 2, "name": "Joe"}');SELECT * FROM employees;+-------------------------------------+| data |+-------------------------------------+| {"id": 1, "name": "Jane"} || {"id": 2, "name": "Joe"} |+-------------------------------------+
• Validation on INSERT
• No reparsing on SELECT
• Optimized for read
• Dictionary of sorted keys
• Can compare JSON/SQL
• Can convert JSON/SQL
• Supports all native JSON datatypes
• Also supports date, time, timestamp etc.
4
The New JSON Datatype
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Agenda
5
JSON data type
JSON functions
Indexing JSON data
Partial update of JSON
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
JSON FunctionsMySQL 5.7 and 8.0
6
JSON_ARRAY_APPEND()
JSON_ARRAY_INSERT()
JSON_ARRAY()
JSON_CONTAINS_PATH()
JSON_CONTAINS()
JSON_DEPTH()
JSON_EXTRACT()
JSON_INSERT()
JSON_KEYS()
JSON_LENGTH()
JSON_MERGE[_PRESERVE]()
JSON_OBJECT()
JSON_QUOTE()
JSON_REMOVE()
JSON_REPLACE()
JSON_SEARCH()
JSON_SET()
JSON_TYPE()
JSON_UNQUOTE()
JSON_VALID()
JSON_PRETTY()
JSON_STORAGE_SIZE()
JSON_STORAGE_FREE()
JSON_ARRAYAGG()
JSON_OBJECTAGG()
JSON_MERGE_PATCH()
JSON_TABLE()
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Inlined JSON Path Expressions
• SELECT * FROM employees WHERE data->'$.id'= 2;
• ALTER … ADD COLUMN id INT AS (data->'$.id') …
• CREATE VIEW .. AS SELECT data->'$.id', data->'$.name' FROM …
SELECT * FROM employees WHERE data->”$.name” = “Jane";
Is a short hand for
SELECT * FROM employees WHERE JSON_EXTRACT(data, “$.name” ) = “Jane”;
[[database.]table.]column->”$<path spec>”
7
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
JSON_TABLE
8
• JSON is not limited to CRUD, can also be used in complex queries
• JSON_TABLE creates a relational view of JSON data
– Each object in a JSON array as a row
– JSON values within an JSON object as column values
• Query the result of JSON_TABLE() as a relational table using SQL
• Leverage existing framework for aggregation
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
JSON_TABLE
CREATE TABLE t1(json_col JSON);
INSERT INTO t1 VALUES ('{ "people": [
{ "name":"John Smith", "address":"780 Mission St, San Francisco, CA 94103"}, { "name":"Sally Brown", "address":"75 37th Ave S, St Cloud, MN 94103"}, { "name":"John Johnson", "address":"1262 Roosevelt Trail, Raymond, ME 04071"}
] }');
9
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
JSON_TABLE
SELECT people.* FROM t1,
JSON_TABLE(json_col, '$.people[*]' COLUMNS (name VARCHAR(40) PATH '$.name',address VARCHAR(100) PATH '$.address')) people;
10
Convert JSON documents to relational tables
name address
John Smith 780 Mission St, San Francisco, CA 94103
Sally Brown 75 37th Ave S, St Cloud, MN 9410
John Johnson 1262 Roosevelt Trail, Raymond, ME 04071
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
JSON_TABLE
SELECT people.* FROM t1,
JSON_TABLE(json_col, '$.people[*]' COLUMNS (name VARCHAR(40) PATH '$.name',address VARCHAR(100) PATH '$.address')) people;
WHERE people.name LIKE 'John%';
11
Filter JSON data
name address
John Smith 780 Mission St, San Francisco, CA 94103
John Johnson 1262 Roosevelt Trail, Raymond, ME 04071
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. 12
JSON_TABLE – Nested Arrays
id father married child_id child age
1 John 1 1 Eric 12
1 John 1 2 Beth 10
2 Paul 0 1 Sarah 9
2 Paul 0 2 Noah 3
2 Paul 0 3 Peter 1
[ { "father":"John", "mother":"Mary",
"marriage_date":"2003-12-05", "children": [
{ "name":"Eric", "age":12 },{ "name":"Beth", "age":10 } ] },
{ "father":"Paul", "mother":"Laura", "children": [
{ "name":"Sarah", "age":9},{ "name":"Noah", "age":3} ,{ "name":"Peter", "age":1} ] }
]
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. 13
JSON_TABLE – Nested Arrays
id father married child_id child age
1 John 1 1 Eric 12
1 John 1 2 Beth 10
2 Paul 0 1 Sarah 9
2 Paul 0 2 Noah 3
2 Paul 0 3 Peter 1
JSON_TABLE (families, '$[*]' COLUMNS (id FOR ORDINALITY,father VARCHAR(30) PATH '$.father',married INTEGER EXISTS PATH
'$.marriage_date',NESTED PATH '$.children[*]' COLUMNS (
child_id FOR ORDINALITY,child VARCHAR(30) PATH '$.name',age INTEGER PATH '$.age‘ ) )
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
JSON_TABLE
SELECT father, COUNT(*) "#children", AVG(age) "age average"FROM t, JSON_TABLE (families, '$[*]' COLUMNS (
id FOR ORDINALITY,father VARCHAR(30) PATH '$.father',NESTED PATH '$.children[*]' COLUMNS (age INTEGER PATH '$.age‘ ) ) AS fam
GROUP BY id, father;
14
SQL aggregation on JSON data
father #children age average
John 2 11.0000
Paul 3 4.3333
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
JSON Aggregation
15
Combine JSON documents in multiple rows into a JSON array
CREATE TABLE t1 (id INT, grp INT, jsoncol JSON);
INSERT INTO t1 VALUES (1, 1, '{"key1":"value1","key2":"value2"}');
INSERT INTO t1 VALUES (2, 1, '{"keyA":"valueA","keyB":"valueB"}');
INSERT INTO t1 VALUES (3, 2, '{"keyX":"valueX","keyY":"valueY"}');
SELECT JSON_ARRAYAGG(jsoncol) FROM t1;
[{"key1":"value1","key2":"value2"}, {"keyA":"valueA","keyB":"valueB"}, {"keyX":"valueX","keyY":"valueY"}]
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
JSON Aggregation
16
Combine JSON documents in multiple rows into a JSON object
CREATE TABLE t1 (id INT, grp INT, jsoncol JSON);
INSERT INTO t1 VALUES (1, 1, '{"key1":"value1","key2":"value2"}');
INSERT INTO t1 VALUES (2, 1, '{"keyA":"valueA","keyB":"valueB"}');
INSERT INTO t1 VALUES (3, 2, '{"keyX":"valueX","keyY":"valueY"}');
SELECT grp, JSON_OBJECTAGG(id, jsoncol)FROM t1GROUP BY grp;
1 | {"1":{"key1":"value1","key2":"value2"}, "2":{"keyA":"valueA","keyB":"valueB"}} 2 | {"3":{"keyX":"valueX","keyY":"valueY"}}
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Agenda
17
JSON data type
JSON functions
Indexing JSON data
Partial update of JSON
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Indexing JSON data
• Use Functional Indexes,
• STORED and VIRTUAL types are supported
18
CREATE TABLE employees (data JSON);
ALTER TABLE employees ADD COLUMN name VARCHAR(30) AS (data->>”$.name”) VIRTUAL, ADD INDEX name_idx (name);
• Functional index approach
• Use inlined JSON path or JSON_EXTRACT to specify field to be indexed
• Support both virtual and stored generated columns
• data->>”$.name” is a shorthand for JSON_UNQUOTE(data->”$.name”)
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Simplified Functional Index Syntax Upcoming!
19
• In 5.7: Create a virtual column, then add an index on this column
• In 8.0: Add a functional index directly, hide creation of virtual column
ALTER TABLE employeesADD COLUMN name VARCHAR(30) AS (data->>”$.name”) VIRTUAL, ADD INDEX name_idx (name);
ALTER TABLE employeesADD INDEX name_idx ((CAST(data->>"$.name" AS VARCHAR(30))));
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Indexing JSON Array Upcoming!
20
• 5.7: Single value index, one index record <-> one data record
• 8.0: Multi value index, multiple index records <-> one data record
• Multi-value index for efficient queries against array fields
• Use CAST( <expr> AS <type> ARRAY) to create multi-valued index
– Expr is any expression that returns JSON array
– Type is any SQL type supported by CAST()
ALTER TABLE employeesADD INDEX name_idx ((CAST(data->>"$.address" AS VARCHAR(30) ARRAY));
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. 21
Indexing JSON Array Upcoming!
id Families
1 {"father":"John", "mother":"Mary","marriage_date":"2003-12-05",
"children": [ {"name":"Eric","age":12,"eyes":"blue"},{"name":"Beth","age":10, "eyes": "green"}]}
2 {"father":"Paul", "mother": "Laura", "children": [
{"name":"Sarah","age":9,"eyes":"brown"},{"name":"Noah","age":3,"eyes": "green"} ,{"name":"Peter","age":1,"eyes": "green"}]}
CREATE TABLE t (id int primary key, families JSON);ALTER TABLE t ADD INDEX eye_color((CAST(families->>"$.children[*].eyes" AS CHAR(10) ARRAY)))
eye_color id
blue 1
brown 2
green 1
green 2
Index
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Indexing JSON Array Upcoming!
22
id father mother
2 Paul Laura
Find parents with children having brown eyes:
SELECT id, families->>"$.father", families->>"$.mother" FROM tWHERE "brown" IN (families->>"$.children[*].eyes")
eye_color id
blue 1
brown 2
green 1
green 2
Index
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Indexing JSON Array Upcoming!
23
Find parents with children having either blue or brown eyes.
SELECT id, families->>"$.father", families->>"$.mother" FROM t WHERE JSON_OVERLAPS(families->>"$.children[*].eyes",
'["blue", "brown"]')
eye_color id
blue 1
brown 2
green 1
green 2
id father mother
1 John Mary
2 Paul Laura
Index
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Agenda
24
JSON data type
JSON functions
Indexing JSON data
Partial update of JSON
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Partial Update of JSON
25
• In 5.7: Updating a JSON document always causes the entire document to be written
• In 8.0: Automatically detect potential in-place updates
– JSON_SET, JSON_REPLACE, JSON_REMOVE
– Input and target columns are the same
– Updating existing values, not adding new ones
– New value is equal to or smaller than the original value, or previous partial update has left sufficient space
• In 8.0: Only diff is shipped through binlog (default full image)
• In 8.0: InnoDB supports partial update of BLOB
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Agenda
27
JSON data type
JSON functions
Indexing JSON data
Partial update of JSON
JSON or column?
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Storing as JSON
• More flexible way to represent data that is hard to model in schema
• No painful schema changes
• Easier prototyping–Fewer types to consider
–No enforced schema
28
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Storing as Column
• Easier to apply schema to your application
• Schema may make applications easier to maintain over time as change is controlled;–Do not have to expect as many permutations
–Allows some constraints over data
29