Modern SQL2018/05/10 · WITH RECURSIVE The Problem [0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties” SELECT

Modern SQL:Evolution of a dinosaur

Markus Winand

Kraków, 9-11 May 2018

Still using Windows 3.1? So why stick with

SQL-92?

@ModernSQL - https://modern-sql.com/ @MarkusWinand

SQL:1999

WITH (Common Table Expressions)

Understand

this first

WITH (non-recursive) The ProblemNested queries are hard to read:

SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)

Then this...



Then this...



Finally the first line makes sense



CTEs are statement-scoped views:

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)

Keyword

WITH (non-recursive) Since SQL:1999




c(…)

Name of CTE and (here optional) column names





c(…)

Definition





c(…)

Introduces another CTE

Don't repeat WITH





c(…)

May refer toprevious CTEs




c(…)AS(SELECT…FROM…)

SELECT…FROMbJOINcON(…)

Third CTE






No comma!






Main query







Read top down


‣ Literate SQL

Organize SQL code toimprove maintainability

‣ Assign column names

to tables produced by valuesor unnest.

‣ Overload tables (for testing)

with queries hide tablesof the same name.

Use-CasesWITH (non-recursive)

https://modern-sql.com/use-case/literate-sql

https://modern-sql.com/use-case/naming-unnamed-columns

https://modern-sql.com/use-case/unit-tests-on-transient-data

http://modern-sql.com/use-case/literate-sql

http://modern-sql.com/use-case/naming-unnamed-columns

http://modern-sql.com/use-case/unit-tests-on-transient-data

WITH are the "private methods" of SQL

WITH is a prefix to SELECT

WITH queries are only visible in the SELECT they precede

WITH in detail: https://modern-sql.com/feature/with

WITH (non-recursive) In a Nutshell

http://modern-sql.com/feature/with

AvailabilityWITH (non-recursive)19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 10.2 MariaDB8.0 MySQL

8.4 PostgreSQL3.8.3[0] SQLite

7.0 DB2 LUW9iR2 Oracle

2005[1] SQL Server[0]Only for top-level SELECT statements[1]Only allowed at the very begin of a statement. E.g. WITH...INSERT...SELECT.

WITHRECURSIVE (Common Table Expressions)

(This page is intentionally left blank)

WITHRECURSIVE The Problem

CREATETABLEt(idNUMERICNOTNULL,parent_idNUMERIC,…PRIMARYKEY(id))

Coping with hierarchies in the Adjacency List Model[0]


[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”

SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)



WHEREd0.id=?





WHEREd0.id=?



WITHRECURSIVE Since SQL:1999

WHEREd0.id=?

WITHRECURSIVEd(id,parent,…)AS(SELECTid,parent,…FROMtblWHEREid=?UNIONALLSELECTid,parent,…FROMdJOINtblON(tbl.parent=d.id))SELECT*FROMsubtree

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

Keyword

Since SQL:1999WITHRECURSIVE



Column list mandatory here




Executed first




Result sent there




Result visible twice




n---123(3rows)

Once it becomes

part of the final result




n---123(3rows)

Second leg of UNION is executed




n---123(3rows)

It's a loop!




n---123(3rows)

It's a loop!




n---123(3rows)

It's a loop!




n---123(3rows)

n=3 doesn't match




n---123(3rows)

n=3 doesn't matchLoop terminates


AvailabilityWITHRECURSIVE19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17


8.4 PostgreSQL3.8.3[0] SQLite

7.0 DB2 LUW11gR2 Oracle

2005 SQL Server[0]Only for top-level SELECT statements

SQL:2003

OVER and

PARTITIONBY

OVER (PARTITION BY) The ProblemTwo distinct concepts could not be used independently:

‣Merge rows with the same key properties

‣ GROUPBY to specify key properties

‣ DISTINCT to use full row as key

‣ Aggregate data from related rows ‣ Requires GROUPBY to segregate the rows

‣ COUNT, SUM, AVG, MIN, MAX to aggregate grouped rows

SELECTc1,SUM(c2)totFROMtGROUPBYc1

OVER (PARTITION BY) The Problem

Yes ⇠

Mer

ge ro

ws ⇢

No

No ⇠ Aggregate ⇢ Yes

SELECTc1,c2FROMt

SELECTDISTINCTc1,c2FROMt

SELECTc1,c2FROMt




Yes ⇠

Mer

ge ro

ws ⇢

No


SELECTc1,c2FROMt


SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)


,tot



Yes ⇠

Mer

ge ro

ws ⇢

No


SELECTc1,c2FROMt


SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)


,tot


OVER (PARTITION BY) Since SQL:2003

Yes ⇠

Mer

ge ro

ws ⇢

No


SELECTc1,c2FROMt


SELECTc1,c2FROMt

FROMt

,SUM(c2)OVER(PARTITIONBYc1)

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

OVER (PARTITION BY) How it works


dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000



dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000



dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000



dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000



dep salary ts1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000


)PARTITIONBYdep

OVER and

ORDERBY(Framing & Ranking)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

OVER (ORDER BY) The Problem

SELECTid,value,FROMtransactionst


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20


SELECTid,value,

(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)

FROMtransactionst


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20


SELECTid,value,

(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)

FROMtransactionst

Range segregation (<=)not possible with

GROUP BY orPARTITION BY

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20


SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYid


SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDING


SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW


SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20



SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20



SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20



SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20



SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20



SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10

22 2 +20

22 3 -10

333 4 +50

333 5 -30

333 6 -20



SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +20

22 3 -10 +10

333 4 +50 +50

333 5 -30 +20

333 6 -20 .0


PARTITIONBYacnt

OVER (ORDER BY) Since SQL:2003With OVER(ORDERBYn) a new type of functions make sense:

n ROW_NUMBER RANK DENSE_RANK PERCENT_RANK CUME_DIST1 1 1 1 0 0.252 2 2 2 0.33… 0.753 3 2 2 0.33… 0.754 4 4 3 1 1

‣ Aggregates without GROUPBY

‣ Running totals, moving averages

‣ Ranking‣ Top-N per Group

‣ Avoiding self-joins

[… many more …]

Use Cases

SELECT*FROM(SELECTROW_NUMBER()OVER(PARTITIONBY…ORDERBY…)rn,t.*FROMt)numbered_tWHERErn<=3

AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg

OVER (SQL:2003)

OVER may follow any aggregate function

OVER defines which rows are visible at each row

OVER() makes all rows visible at every row

OVER(PARTITIONBY …) segregates like GROUPBY

OVER(ORDERBY…BETWEEN) segregates using <, >

In a NutshellOVER (SQL:2003)

OVER (SQL:2003) Availability19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17


8.4 PostgreSQLSQLite

7.0 DB2 LUW8i Oracle

2005 SQL Server

Hive

ImpalaSpark

NuoDB

SQL:2006

XMLTABLE

SELECTid,c1,nFROMtbl,XMLTABLE('/d/e'PASSINGxCOLUMNSidINTPATH'@id',c1VARCHAR(255)PATH'c1',nFORORDINALITY)r

XMLTABLE Since SQL:2006Stored in tbl.x:

<d><eid="42"><c1>…</c1></e></d>

XPath* expression to identify rows

*Standard SQL allows XQuery



<d><eid="42"><c1>…</c1></e></d>




<d><eid="42"><c1>…</c1></e></d>


XPath* expressions to extract data



<d><eid="42"><c1>…</c1></e></d>


Row number (like for unnest)



<d><eid="42"><c1>…</c1></e></d>


Result id|c1|n----+----+---42|…|1

XMLTABLE Availability19

99

2001

2003

2005

2007

2009

2011

2013

2015

2017

MariaDBMySQL

10[0] PostgreSQLSQLite

9.7 DB2 LUW11gR1 Oracle

SQL Server[0]No XQuery (only XPath). No default namespace declaration.

SQL:2008

FETCHFIRST

SELECT*FROM(SELECT*,ROW_NUMBER()OVER(ORDERBYx)rnFROMdata)numbered_dataWHERErn<=10

FETCHFIRST The ProblemLimit the result to a number of rows. (LIMIT, TOP and ROWNUM are all proprietary)

SQL:2003 introduced ROW_NUMBER() to number rows.But this still requires wrapping to limit the result.

And how about databases not supporting ROW_NUMBER()?

SELECT*FROM(SELECT*,ROW_NUMBER()OVER(ORDERBYx)rnFROMdata)numbered_dataWHERErn<=10

FETCHFIRST The ProblemLimit the result to a number of rows. (LIMIT, TOP and ROWNUM are all proprietary)

SQL:2003 introduced ROW_NUMBER() to number rows.But this still requires wrapping to limit the result.

And how about databases not supporting ROW_NUMBER()?

Dammit! Let's takeLIMIT

SELECT*FROMdataORDERBYxFETCHFIRST10ROWSONLY

FETCHFIRST Since SQL:2008SQL:2008 introduced the FETCHFIRST…ROWSONLY clause:

FETCHFIRST Availability19

99

2001

2003

2005

2007

2009

2011

2013

2015

2017

5.1 MariaDB3.19.3[0] MySQL

6.5[1] 8.4 PostgreSQL2.1.0[1] SQLite

7.0 DB2 LUW12cR1 Oracle

7.0[2] 2012 SQL Server[0]Earliest mention of LIMIT. Probably inherited from mSQL[1]Functionality available using LIMIT[2]SELECTTOPn... SQL Server 2000 also supports expressions and bind parameters

SQL:2011

OFFSET

SELECT*FROM(SELECT*,ROW_NUMBER()OVER(ORDERBYx)rnFROMdata)numbered_dataWHERErn>10andrn<=20

OFFSET The ProblemHow to fetch the rows after a limit?

(pagination anybody?)

SELECT*FROMdataORDERBYxOFFSET10ROWSFETCHNEXT10ROWSONLY

OFFSET Since SQL:2011SQL:2011 introduced OFFSET, unfortunately!

SELECT*FROMdataORDERBYxOFFSET10ROWSFETCHNEXT10ROWSONLY

OFFSET Since SQL:2011SQL:2011 introduced OFFSET, unfortunately!

OFFSETGrab coasters & stickers!

https://use-the-index-luke.com/no-offset

OFFSET Since SQL:201119

99

2001

2003

2005

2007

2009

2011

2013

2015

5.1 MariaDB3.20.3[0] 4.0.6[1] MySQL

6.5 PostgreSQL2.1.0 SQLite

9.7[2] 11.1 DB2 LUW12c Oracle

2012 SQL Server[0]LIMIT[offset,]limit: "With this it's easy to do a poor man's next page/previous page WWW application."[1]The release notes say "Added PostgreSQL compatible LIMIT syntax"[2]Requires enabling the MySQL compatibility vector: db2setDB2_COMPATIBILITY_VECTOR=MYS

OVER

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt





currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt,ROW_NUMBER()OVER(ORDERBYx)rn





currbalance … rn

50 … 190 … 270 … 330 … 4






currbalance … rn

50 … 190 … 270 … 330 … 4


prevbalance … rn

50 … 190 … 270 … 330 … 4





currbalance … rn

50 … 190 … 270 … 330 … 4


prevbalance … rn

50 … 190 … 270 … 330 … 4





currbalance … rn

50 … 190 … 270 … 330 … 4


prevbalance … rn

50 … 190 … 270 … 330 … 4

+50+40-20-40

SELECT*,balance-COALESCE(LAG(balance)OVER(ORDERBYx),0)FROMt

Available functions:LEAD/LAGFIRST_VALUE/LAST_VALUENTH_VALUE(col,n)FROMFIRST/LASTRESPECT/IGNORENULLS

OVER (SQL:2011) Since SQL:2011SQL:2011 introduced LEAD, LAG, NTH_VALUE, … for that:

OVER (LEAD, LAG, …) Since SQL:201119

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 10.2[0] MariaDB8.0[0] MySQL

8.4[0] PostgreSQLSQLite

9.5[1] 11.1 DB2 LUW8i[1] 11gR2 Oracle

2012[1] SQL Server[0]No IGNORENULLS and FROMLAST[1]No NTH_VALUE

System Versioning (Time Traveling)

INSERTUPDATEDELETE

are DESTRUCTIVE

System Versioning The Problem

CREATETABLEt(...,start_tsTIMESTAMP(9)GENERATEDALWAYSASROWSTART,end_tsTIMESTAMP(9)GENERATEDALWAYSASROWEND,

PERIODFORSYSTEM_TIME(start_ts,end_ts))WITHSYSTEMVERSIONING

System Versioning Since SQL:2011Table can be system versioned, application versioned or both.

ID Data start_ts end_ts1 X 10:00:00

UPDATE...SETDATA='Y'...

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00

DELETE...WHEREID=1

INSERT...(ID,DATA)VALUES(1,'X')

System Versioning Since SQL:2011

ID Data start_ts end_ts1 X 10:00:00

UPDATE...SETDATA='Y'...

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00

DELETE...WHEREID=1

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00

INSERT...(ID,DATA)VALUES(1,'X')


Although multiple versions exist, only the “current” one is visible per default.

After 12:00:00, SELECT*FROMt doesn’t return anything anymore.




With FOR…ASOF you can query anything you like: SELECT*FROMtFORSYSTEM_TIMEASOFTIMESTAMP'2015-04-0210:30:00'

ID Data start_ts end_ts

1 X 10:00:00 11:00:00


1999

2001

2003

2005

2007

2009

2011

2013

2015

2017

5.1 MariaDB[0]

MySQLPostgreSQLSQLite

10.1[1] DB2 LUW10gR1[2] Oracle

2016 SQL Server[0]Available in MariaDB 10.3 beta.[1]Third column required (tx id), history table required.[2]Functionality available using Flashback


SQL:2016 (released: 2016-12-15)

LISTAGG

Since SQL:2016

grp val1 B1 A1 C2 X

grp val1 A, B, C2 X

SELECTgrp,LISTAGG(val,',')WITHINGROUP(ORDERBYval)FROMtGROUPBYgrp

LISTAGG(val,','ONOVERFLOWTRUNCATE'...'WITHCOUNT)➔'A,B,...(1)'

LISTAGG(val,','ONOVERFLOWERROR)

Default

LISTAGG

LISTAGG(val,','ONOVERFLOWTRUNCATE'...'WITHOUTCOUNT)➔'A,B,...'

Default

1999

2001

2003

2005

2007

2009

2011

2013

2015

5.1[0] MariaDB4.1[0] MySQL

7.4[1] 8.4[2]9.0[3] PostgreSQL3.5.4[4] SQLite

10.5[5] DB2 LUW11gR1 12cR2 Oracle

SQL Server[6]

[0]group_concat[1]array_to_string[2]array_agg[3]string_agg[4]group_concat without ORDER BY[5]No ON OVERFLOW clause[6]string_agg announced for vNext

LISTAGG Availability

[0] group_concat [1] array_to_string [2] array_agg [3] string_agg

[4] group_concat w/o ORDERBY [5] No ONOVERFLOW clause [6] string_agg announced for vNext

New in SQL:2016 JSON

LISTAGGhttps://modern-sql.com/feature/listagg

ROW PATTERN MATCHING https://www.slideshare.net/MarkusWinand/row-pattern-matching-in-sql2016

DATE FORMAT POLYMORPHIC TABLE FUNCTIONS

➔ https://modern-sql.com/blog/2017-06/whats-new-in-sql-2016

SQL has evolved beyond the relational idea.

Modern SQL? @MarkusWinand


If you are using SQL like 25 years ago,you are doing it wrong!



If you are using SQL like 25 years ago,you are doing it wrong!

A lot has happened since SQL-92.


https://www.flickr.com/photos/mfoubister/25367243054/

I have shown you a few features today


I have shown you a few features today


There are hundreds more to discover

@ModernSQL modern-sql.comMy other website:

https://use-the-index-luke.com

Training & co: https://winand.at/

Modern SQL2018/05/10 · WITH RECURSIVE The Problem [0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties” SELECT

Documents