1 Agile Database Development with JSON Chris Saxon Developer Advocate, @ ChrisRSaxon & @ SQLDaily blogs.oracle.com/ sql youtube.com/c/ TheMagicofSQL asktom.oracle.com
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
-
1
Agile Database Development with JSON
Chris Saxon
Developer Advocate, @ChrisRSaxon & @SQLDaily
blogs.oracle.com/sql
youtube.com/c/TheMagicofSQL
asktom.oracle.com
https://twitter.com/chrisrsaxonhttps://twitter.com/sqldailyhttps://blogs.oracle.com/sqlhttps://www.youtube.com/c/TheMagicofSQLhttps://asktom.oracle.com/
-
Image by Semevent from Pixabay
https://pixabay.com/users/Semevent-4278751https://pixabay.com/
-
Photo by Jon Tyson on Unsplash
https://unsplash.com/@jontyson?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyTexthttps://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText
-
Photo by Johannes Plenio on Unsplash
https://unsplash.com/@jplenio?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyTexthttps://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText
-
Photo by Brannon Naito on Unsplash
https://unsplash.com/@brannon_naito?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyTexthttps://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText
-
{ JSON }
-
9
Agile Database Development with JSON
Chris Saxon
Developer Advocate, @ChrisRSaxon & @SQLDaily
blogs.oracle.com/sql
youtube.com/c/TheMagicofSQL
asktom.oracle.com
https://twitter.com/chrisrsaxonhttps://twitter.com/sqldailyhttps://blogs.oracle.com/sqlhttps://www.youtube.com/c/TheMagicofSQLhttps://asktom.oracle.com/
-
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor
http://www.oracle.com/investor
-
User Story #1
We must be able to store product & order details
-
create table products (
product_id integer
not null
primary key,
);
create table orders (
order_id integer
not null
primary key,
);
-
create table products (
product_id integer
not null
primary key,
product_json ##TODO##
not null,
);
create table orders (
order_id integer
not null
primary key,
order_json ##TODO##
not null,
);
-
create table products (
product_id integer
not null
primary key,
product_json ##TODO##
not null,
check (
json_data is json
)
);
create table orders (
order_id integer
not null
primary key,
order_json ##TODO##
not null,
check (
json_data is json
)
);
-
create table products (
product_id integer
not null
primary key,
product_json ##TODO##
not null,
check (
json_data is json
)
);
create table orders (
order_id integer
not null
primary key,
order_json ##TODO##
not null,
check (
json_data is json
)
);
-
"Small" documents varchar2
"Large" documents ???
-
"Small" documents varchar2
"Large" documents blob
JSON data type
coming in 20c
Avoids character set conversionsLess storage than clob
-
create table products (
product_id integer
not null
primary key,
product_json blob
not null,
check (
json_data is json
)
);
create table orders (
order_id integer
not null
primary key,
order_json blob
not null,
check (
json_data is json
)
);
-
insert into products ( product_json )
values ( utl_raw.cast_to_raw ( '{
"productName": "..."
}' ) );
-
select product_json from products;
PRODUCT_JSON
7B202274686973223A20227468617422207D
-
select json_serialize (
product_json
returning clob
pretty
) jdata
from products;
JDATA
{
"productName": "..."
}
Added in 19c
-
select json_query (
product_json,
'$' returning clob
pretty
) jdata
from products;
JDATA
{
"productName": "..."
}
-
select json_query (
product_json,
'$' returning clob
pretty
) jdata
from products;
JDATA
{
"productName": "..."
}
clob added in 18c
-
User Story #2
Customers must be able to search by price
-
{
"productName": "GEEKWAGON",
"descripion": "Ut commodo in …",
"unitPrice": 35.97,
"bricks": [ {
"colour": "red", "shape": "cube",
"quantity": 13
}, {
"colour": "green", "shape": "cube",
"quantity": 17
}, …
]
}
-
select * from products p
where p.product_json.unitPrice implicit conversion!
-
select * from products p
where json_value (
product_json,
'$.unitPrice' returning number
)
-
select * from products p
where p.product_json.unitPrice.number()
-
User Story #3
Customers must be able to view their orders
-
{
"customerId" : 2,
"orderDatetime" : "2019-01-01T03:25:43",
"products" : [ {
"productId" : 1,
"unitPrice" : 74.95
}, {
"productId" : 10,
"unitPrice" : 35.97
}, …
]
}
Join to products
-
select o.order_json.products[*].productId
from orders o;
PRODUCTS
[2,8,5]
[3,9,6]
[1,10,7,4]
...
-
select json_query (
order_json, '$.products[*].productId'
with array wrapper
)
from orders o;
PRODUCTS
[2,8,5]
[3,9,6]
[1,10,7,4]
...
Need to convert to rows…
-
json_table
-
with order_items as (
select order_id, t.*
from orders o, json_table (
order_json
columns (
customerId,
nested products[*] columns (
productId,
unitPrice
) )
) t
)
Simplified syntax 18c
-
with order_items as (
select order_id, t.*
from orders o, json_table (
order_json
columns (
customerId,
nested products[*] columns (
productId,
unitPrice
) )
) t
)
-
with order_items as (
select order_id, t.*
from orders o, json_table (
order_json
columns (
customerId,
nested products[*] columns (
productId,
unitPrice
) )
) t
)
-
with order_items as (
select order_id, t.*
from orders o, json_table (
order_json
columns (
customerId,
nested products[*] columns (
productId,
unitPrice
) )
) t
)
Row/element
-
select order_id,
p.product_json.productName product,
unitPrice
from order_items oi
join products p
on oi.productId = p.product_id
where customerId = :cust_var
order by oi.order_id desc, p.product_id
-
Minimum Viable Product Complete!
Ship it!
-
Copyright © 2019 Oracle and/or its affiliates.
Soooo…How many
orders today?
Ryan McGuire / Gratisography
http://www.gratisography.com/
-
User Story #4
Sales must be able to view today's orders
-
{
"customerId" : 2,
"orderDatetime" : "2019-01-01T03:25:43",
"products" : [ {
"productId" : 1,
"unitPrice" : 74.95
}, {
"productId" : 10,
"unitPrice" : 35.97
}, …
]
}
-
select * from orders o
where o.order_json.orderDatetime >=
trunc ( sysdate );
ORA-01861: literal does
not match format string
-
select * from orders o
where json_value (
order_json,
'$.orderDatetime' returning date
) >= trunc ( sysdate )
ISO 8601 date
-
2019-01-01
ISO 8601 date
-
2019-01-01T03:25:43
ISO 8601 timestamp
-
select * from orders o
where json_value (
order_json,
'$.orderDatetime' returning date
) >= trunc ( sysdate )
{ "customerId": 1, … }
{ "customerId": 2, … }
…
-
User Story #4b
… and make it fast!
-
create index orders_date_i
on orders ( order_json );
ORA-02327: cannot create index on
expression with datatype LOB
-
create search index orders_json_i
on orders ( order_json )
for json
parameters ( 'sync (on commit)' );
12.2
-
select * from orders o
where json_value (
order_json,
'$.orderDatetime' returning date
) >= trunc ( sysdate )
{ "customerId": 1, … }
{ "customerId": 2, … }
…
-
-----------------------------------------------------
| Id | Operation | Name |
-----------------------------------------------------
| 0 | SELECT STATEMENT | |
|* 1 | TABLE ACCESS BY INDEX ROWID| ORDERS |
|* 2 | DOMAIN INDEX | ORDERS_JSON_I |
-----------------------------------------------------
-
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(JSON_VALUE("ORDER_JSON" FORMAT JSON ,
'$.orderDatetime' RETURNING TIMESTAMP NULL
ON ERROR) >= TIMESTAMP' 2019-01-15 00:00:00')
2 - access("CTXSYS"."CONTAINS"("O"."ORDER_JSON",
'sdatap(TMS_orderDatetime >=
"2019-01-15T00:00:00+00:00" /orderDatetime)')>0)
-
create index order_date_i
on orders (
json_value (
order_json,
'$.orderDatetime'
returning date
error on error
null on empty
)
);
-
create index order_date_i
on orders (
json_value (
order_json,
'$.orderDatetime'
returning date
error on error
null on empty
)
);
Data validation!
-
create index order_date_i
on orders (
json_value (
order_json,
'$.orderDatetime'
returning date
error on error
null on empty
)
);
Is it present?(12.2)
-
------------------------------------------------------------
| Id | Operation | Name |
------------------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | TABLE ACCESS BY INDEX ROWID BATCHED| ORDERS |
|* 2 | INDEX RANGE SCAN | ORDER_DATE_I |
------------------------------------------------------------
-
Search vs. Function-Based Indexes
JSON Search Index Function-based Index
Applicability Any* JSON query Matching function
Performance Slower Faster
Use Ad-hoc queries Application queries
-
0
5
10
15
20
25
-
We need to offer discounts!
Ryan McGuire / Gratisography
http://www.gratisography.com/
-
User Story #5
Customers may be able to enter a promotion code
-
{
…,
"promotion": {
"code": "20OFF",
"discountAmount": 20
}
}
-
Nothing to do in the database!
relax!
Ryan McGuire / Gratisography
http://www.gratisography.com/
-
0
20
40
60
80
100
120
-
Ryan McGuire / Gratisography
http://www.gratisography.com/
-
Where's the $$$?!
Ryan McGuire / Gratisography
http://www.gratisography.com/
-
0
5
10
15
20
25
-
-15
-10
-5
0
5
10
15
20
25
-
-60
-40
-20
0
20
40
60
-
-250
-200
-150
-100
-50
0
50
100
150
-
Finance need to view order profitability
-
User Story #6
Store unit cost for each brick
-
{ …,
"bricks": [ {
"colour": "red",
"shape": "cube",
"quantity": 13
}, {
"colour": "green",
"shape": "cuboid",
"quantity": 17
}, …
]
}
Add unitCost
-
We have a spreadsheet!
-
"bricks": [ {
"colour": "red",
"shape": "cube",
"quantity": 13
}, {
"colour": "green",
"shape": "cuboid",
"quantity": 17
}, …
] join on
colour, shape
-
Photo by Gus Ruballo on Unsplash
https://unsplash.com/@gusruballo?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyTexthttps://unsplash.com/s/photos/seatbelt?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText
-
select * from external ( (
colour varchar2(30),
shape varchar2(30),
unit_cost number
)
default directory tmp
location ( 'costs.csv' )
)
Inline external table 18c
-
select product_id, j.*
from products, json_table (
product_json columns (
nested bricks[*] columns (
pos for ordinality,
colour path '$.colour',
shape path '$.shape',
brick format json path '$'
)
)
) j
-
select product_id, j.*
from products, json_table (
product_json columns (
nested bricks[*] columns (
pos for ordinality,
colour path '$.colour',
shape path '$.shape',
brick format json path '$'
)
)
) j
-
select product_id, j.*
from products, json_table (
product_json columns (
nested bricks[*] columns (
pos for ordinality,
colour path '$.colour',
shape path '$.shape',
brick format json path '$'
)
)
) j
-
with costs as (
select * from external …
), bricks as (
select product_id, j.*
from products, json_table (
…
)
)
select …
from bricks join costs
on …
What goes here?
-
json_object
json_objectagg
json_array
json_arrayagg
12.2
-
select json_object (
'colour' value b.colour,
'shape' value b.shape,
'quantity' value b.quantity,
'unitCost' value c.cost
)
from bricks b
join costs c
on b.colour = c.colour
and b.shape = c.shape;
-
select json_mergepatch (
brick,
'{ "unitCost": ' || c.cost || '}'
)
from bricks b
join costs c
on b.colour = c.colour
and b.shape = c.shape;
Add/replace this…
…to this document
19c
-
{
"colour": "red",
"shape": "cube",
"quantity": 13,
"unitCost": 0.59
}
{
"colour": "green",
"shape": "cuboid",
"quantity": 17,
"unitCost": 0.39
}
-
json_arrayagg (
json_mergepatch (
brick,
'{ "unitCost": ' || cost || '}'
) order by pos
)
-
[ {
"colour": "red",
"shape": "cube",
"quantity": 13,
"unitCost": 0.59
}, {
"colour": "green",
"shape": "cuboid",
"quantity": 17,
"unitCost": 0.39
}, …
]
-
json_object (
'bricks' value
json_arrayagg (
json_mergepatch (
brick,
'{ "unitCost": ' || cost || '}'
) order by pos
)
)
-
"bricks": [ {
"colour": "red",
"shape": "cube",
"quantity": 13,
"unitCost": 0.59
}, {
"colour": "green",
"shape": "cuboid",
"quantity": 17,
"unitCost": 0.39
}, …
]
-
json_mergepatch (
product,
json_object (
'bricks' value
json_arrayagg (
json_mergepatch (
brick,
'{ "unitCost": ' || cost || '}'
) order by pos
)
)
)
-
{
"productName": "GEEKWAGON",
"descripion": "Ut commodo in …",
"unitPrice": 35.97,
"bricks": [ {
…, "unitCost": 0.59
}, {
…, "unitCost": 0.39
}, …
]
}
-
update products
set product_json = (
with costs as (
select * from external …
), bricks as (
select …
)
select json_mergepatch …
)
-
Ryan McGuire / Gratisography
http://www.gratisography.com/
-
User Story #7
Create report prices - discount – total cost
-
{ JSON }
Data Guide
-
dbms_json.add_virtual_columns (
'orders', 'order_json'
);
Must havejson search
index on12.2
-
desc orders
Name Null? Type
ORDER_ID NOT NULL NUMBER(38)
ORDER_JSON NOT NULL BLOB
ORDER_JSON$customerId NUMBER
ORDER_JSON$orderDatetime VARCHAR2(32)
ORDER_JSON$code VARCHAR2(8)
ORDER_JSON$discountAmount NUMBER
only scalar values
-
dbms_json.create_view_on_path (
'product_bricks', 'products',
'product_json', '$'
);
…using json_table
on this!
Create this view…
12.2
-
select product_id,
"PRODUCT_JSON$shape" shape,
"PRODUCT_JSON$colour" colour
from product_bricks
order by product_id, shape, colour
-
PRODUCT_ID SHAPE COLOUR
1 cube green
1 cube red
1 cylinder blue
1 cylinder blue
1 cylinder green
1 cylinder green
… … …
-
PRODUCT_ID SHAPE COLOUR
1 cube green
1 cube red
1 cylinder blue
1 cylinder blue
1 cylinder green
1 cylinder green
… … …
-
User Story #8
FIX ALL THE DATAZ!
-
{ ..., "bricks" : [
{
"colour" : "red",
"shape" : "cylinder",
"quantity" : 20,
"unitCost" : 0.39
}, {
"colour" : "red",
"shape" : "cylinder",
"quantity" : 20,
"unitCost" : 0.39
}
{ ..., "bricks" : [
{
"colour" : "red",
"shape" : "cylinder",
"quantity" : 8,
"unitCost" : 0.39
}, {
"colour" : "blue",
"shape" : "cylinder",
"quantity" : 10,
"unitCost" : 0.98
}
-
{ ..., "bricks" : [
{
"colour" : "red",
"shape" : "cylinder",
"quantity" : 20,
"unitCost" : 0.39
}, {
"colour" : "red",
"shape" : "cylinder",
"quantity" : 20,
"unitCost" : 0.39
}
{ ..., "bricks" : [
{
"colour" : "red",
"shape" : "cylinder",
"quantity" : 8,
"unitCost" : 0.39
}, {
"colour" : "blue",
"shape" : "cylinder",
"quantity" : 10,
"unitCost" : 0.98
}
-
{ ..., "bricks" : [
{
"colour" : "red",
"shape" : "cylinder",
"quantity" : 20,
"unitCost" : 0.39
}, {
"colour" : "red",
"shape" : "cylinder",
"quantity" : 20,
"unitCost" : 0.39
}
{ ..., "bricks" : [
{
"colour" : "red",
"shape" : "cylinder",
"quantity" : 8,
"unitCost" : 0.39
}, {
"colour" : "blue",
"shape" : "cylinder",
"quantity" : 10,
"unitCost" : 0.98
}
-
Wrong Data Model
PRODUCTS BRICKS
-
Fixed It!
PRODUCTS BRICKSPRODUCT_BRICKS
unique ( product_id,
brick_id
)
{ JSON } { JSON }{ JSON }
-
You still need to model { JSON } data!
-
Copyright © 2019 Oracle and/or its affiliates.
"The more I work with existing NoSQLdeployments however, the more I believe that their
schemaless nature has become an excuse for sloppiness and unwillingness to dwell
on a project’s data model beforehand"- Florents Tselai
https://tselai.com/modern-data-practice-and-the-sql-tradition.html
-
select distinct "PRODUCT_JSON$shape" shape,
"PRODUCT_JSON$colour" colour,
"PRODUCT_JSON$unitCost" unit_cost
from product_bricks
-
with vals as (
select distinct "PRODUCT_JSON$shape" shape,
"PRODUCT_JSON$colour" colour,
"PRODUCT_JSON$unitCost" unit_cost
from product_bricks
)
select rownum brick_id,
v.*
from vals v;
-
create table bricks as
with vals as (
select distinct "PRODUCT_JSON$shape" shape,
"PRODUCT_JSON$colour" colour,
"PRODUCT_JSON$unitCost" unit_cost
from product_bricks
)
select rownum brick_id,
v.*
from vals v;
-
create table bricks as
with vals as (
select distinct "PRODUCT_JSON$shape" "shape",
"PRODUCT_JSON$colour" "colour",
"PRODUCT_JSON$unitCost" "unitCost"
from product_bricks
)
select rownum brick_id,
json_object ( v.* ) brick_json
from vals v;
19c simplification
-
create table product_bricks as
select distinct product_id, brick_id
from product_bricks_vw
join bricks
on ...
-
json_mergepatch (
product_json,
'{ "bricks": null }'
)
-
When should I store { JSON }?
-
Storing JSON can be the right choice for…
JSON responses- 3rd party APIs- IoT devices
Schema extensions- flex fields- sparse columns
1 2
-
Further Reading
How to Store, Query, and Create JSON Documents in Oracle Database Blog Post
http://bit.ly/json-in-oracle-db
Presentation Live SQL Scriptshttp://bit.ly/agile-json-livesql
Presentation Slideshttp://bit.ly/agile-json-slides
Copyright © 2019 Oracle and/or its affiliates.
http://bit.ly/json-in-oracle-dbhttp://bit.ly/agile-json-livesqlhttp://bit.ly/agile-json-slides
-
Copyright © 2019 Oracle and/or its affiliates.
VS
-
Copyright © 2019 Oracle and/or its affiliates.
VS
-
Copyright © 2019 Oracle and/or its affiliates.
-
asktom.oracle.com
#MakeDataGreatAgainRyan McGuire / Gratisography
http://www.gratisography.com/
Related Documents
![Navigating State - php[architect] · Navigating State The Dev Lead Trenches: Issue Workflows for Teams ... The MySQL Document Store MySQL 5.7 introduced a native JSON ... been greatly](https://static.cupdf.com/doc/110x72/5ec5ec2e0ef944652c202fd2/navigating-state-phparchitect-navigating-state-the-dev-lead-trenches-issue.jpg)
