Ranges, ranges everywhere (Oracle SQL)

Ranges, Ranges Everywhere!

Stew Ashton (stewashton.wordpress.com)UKOUG Tech 2016

Can you read the following line? If not, please move closer.

It's much better when you can read the code ;)

2

Agenda

• Defining ranges• Relating ranges: gaps, overlaps• Range DDL: sensible data• Ranges in one table• Ranges in two tables

3

Who am I?• 36 years in IT

– Developer, Technical Sales Engineer, Technical Architect– Aeronautics, IBM, Finance– Mainframe, client-server, Web apps

• 12 years using Oracle database– SQL performance analysis– Replace Java with SQL

• 4 years as in-house “Oracle Development Expert”• Conference speaker since 2014• Currently independent

4

Questions

5

What is a range?• Two values that can be compared

– Always use the same datatype – Comparable datatypes:

• integer, date (without time)• number, datetime, interval, (n)(var)char• rowid

• Range design questions:– Is the "end" value part of the range?– Are NULLs allowed?

6

Allen’sIntervalAlgebra

1 2 3 4A precedes B 1 2

B preceded by A 3 4A meets B 1 2 B met by A 2 3

A overlaps B 1 3B overlapped by A 2 4

A finished by B 1 3B finishes A 2 3A contains B 1 4B during A 2 3A starts B 1 2

B started by A 1 3A and B 1 2

are equal 1 2

Meet

Gap

"Overlap"

1 2 3 41 2 3 4A precedes B 1 2

B preceded by A 3 4

1 2 3 4A precedes B 1 2

B preceded by A 3 4A meets B 1 2 B met by A 2 3

7

End value: Inclusive or Exclusive• Design must allow ranges to "meet"• Discrete quantities can be inclusive– [1-3] meets [4-6] : no intermediate integer– [Jan. 1-31] meets [Feb. 1-28] : no intermediate date

• Continuous quantities require exclusive– Most ranges are continuous (including dates, really)

8

Votes for Exclusive end values• SQL:2013 and Oracle 12c Temporal Validity

– "Period": date/time range• [Closed-Open): includes start time but not end time

• WIDTH_BUCKET() function– Puts values in equiwidth histogram– Buckets must touch– [Closed-open): upper boundary value goes in higher bucket

• Me!– Exclusive end values work for every kind of range– Except: ROWID ranges must be inclusive

9

DDL: make sure data is sensible• Start_range < End_range• If date without time, CHECK( dte = trunc(dte))• If integer, say so• Is NULL allowed?– If so, what does it mean?– Ex. Temporal Validity :

NULL end value means "until the end of time"• Are overlaps allowed?

10

Overlaps avoided by unique constraintsUnique(start,end) Unique(start) Unique(end) 1 2 3 4

No constraint works A overlaps B 1 3B overlapped by A 2 4

Y A finished by B 1 3B finishes A 2 3

No constraint works A contains B 1 4B during A 2 3

Y A starts B 1 2 B started by A 1 3

Y Y Y A and B 1 2 are equal 1 2

11

Avoiding Overlaps: 3 solutions

1. Triggers– Hard to do right, not very scalable

2. "Refresh on commit" materialized views– Not scalable?

3. Virtual ranges

12

Virtual range: no gaps, no overlaps• One column: start value• End value is calculated:

= next row's start– Putting identical value in 2

rows is denormalization• Last row has unlimited end• Maybe OK for audit trails?

START_VALUE END_VALUE

16-11-15 08:30 16-11-15 09:30

16-11-15 09:30 16-11-15 18:30

16-11-15 18:30 (null)

START_VALUE

16-11-15 08:30

16-11-15 09:30

16-11-15 18:30

Physical (table)

Virtual (view)

13

Semi-Virtual range: no overlaps• Start column always used• End column optional:

– If null, use next row's start– If not null, use lesser of end

column and next row's start– Last row can have limited end

• Or: intermediate row with 'not exists' flag– ≅ Change Data Capture format

START_VALUE END_VALUE

16-11-15 08:30 16-11-15 09:30

16-11-15 18:30 (null)

START_VALUE D

16-11-15 08:30

16-11-15 09:30 D

16-11-15 18:30

14

Range-related SQL• Why hard?

– Can't use BETWEEN– Inequality joins impact performance– With overlaps, 1 value point can be in any number of rows– Joining 2 tables with overlaps -> row explosion– NULLs have special meanings

• Common problems– Find gaps– Intersect: find overlaps– Union: packing ranges between gaps– Joins

• Today, ends are exclusive, everything is NOT NULL (unless specified)

15

16

FROM_TM TO_TM07:00 08:0009:00 10:5010:00 10:4512:00 12:4518:00 23:00

select * from ( select max (to_tm) over(order by from_tm) as gap_from, lead(from_tm) over(order by from_tm) as gap_to from t) where gap_from < gap_to;

select to_tm as gap_from, lead(from_tm) over(order by from_tm) as gap_to from t

FROM_TM GAP_FROM GAP_TO07:00 08:00 09:0009:00 10:50 10:0010:00 10:45 12:0012:00 12:45 18:0018:00 23:00

GAP_FROM GAP_TO08:00 09:0010:50 12:0012:45 18:00

Gaps, ex. Free time in calendarFROM_TM GAP_FROM GAP_TO

07:00 08:00 09:0009:00 10:50 10:0010:00 10:50 12:0012:00 12:45 18:0018:00 23:00

17

Intersect: finding OverlapsTest case Start End01:precedes 1 201:precedes 3 402:meets 1 202:meets 2 303:overlaps 1 303:overlaps 2 404:finished by 1 304:finished by 2 305:contains 1 405:contains 2 306:starts 1 206:starts 1 307:equals 1 207:equals 1 2

select test_case, dte, col from t unpivot (dte for col in ( start_date as 1, end_date as -1))

A overlaps B 1 3B overlapped by A 2 4

1 2 2 3

3 4

18

select test_case, dte, col from t unpivot (dte for col in ( start_date as 1, end_date as -1))

select test_case, dte "Start", lead(dte,1,dte) over( partition by test_case order by dte, col desc ) "End", sum(col) over( partition by test_case order by dte, col desc ) "Rows" from t unpivot (dte for col in ( start_date as 1, end_date as -1))

Intersect: finding OverlapsTest case Dte Col01:precedes 1 101:precedes 2 -101:precedes 3 101:precedes 4 -102:meets 1 102:meets 2 -102:meets 2 102:meets 3 -103:overlaps 1 103:overlaps 3 -103:overlaps 2 103:overlaps 4 -1

19

select test_case, dte "Start", lead(dte,1,dte) over( partition by test_case order by dte, col desc ) "End", sum(col) over( partition by test_case order by dte, col desc ) "Rows" from t unpivot (dte for col in ( start_date as 1, end_date as -1))

select * from ( select test_case, dte "Start", lead(dte,1,dte) over( partition by test_case order by dte, col desc ) "End", sum(col) over( partition by test_case order by dte, col desc ) "Rows" from t unpivot (dte for col in ( start_date as 1, end_date as -1))) where "Start" < "End";

Intersect: finding OverlapsTest case Start End Rows01:precedes 1 2 101:precedes 2 3 001:precedes 3 4 101:precedes 4 4 002:meets 1 2 102:meets 2 2 202:meets 2 3 102:meets 3 3 003:overlaps 1 2 103:overlaps 2 3 203:overlaps 3 4 103:overlaps 4 4 0

✖

✖

✖

✖

20

select * from ( select test_case, dte "Start", lead(dte,1,dte) over( partition by test_case order by dte, col desc ) "End", sum(col) over( partition by test_case order by dte, col desc ) "Rows" from t unpivot (dte for col in ( start_date as 1, end_date as -1))) where "Start" < "End";

select * from ( select test_case, dte "Start", lead(dte,1,dte) over( partition by test_case order by dte, col desc ) "End", sum(col) over( partition by test_case order by dte, col desc ) "Rows" from t unpivot (dte for col in ( start_date as 1, end_date as -1))) where "Rows" > 1and "Start" < "End";

Intersect: finding OverlapsTest case Start End Rows01:precedes 1 2 101:precedes 2 3 001:precedes 3 4 102:meets 1 2 102:meets 2 3 103:overlaps 1 2 103:overlaps 2 3 203:overlaps 3 4 1

Test case Start End Rows03:overlaps 2 3 204:finished by 2 3 205:contains 2 3 206:starts 1 2 207:equals 1 2 2

21

Test case Start End01:precedes 1 201:precedes 3 402:meets 1 202:meets 2 303:overlaps 1 303:overlaps 2 404:finished by 1 304:finished by 2 305:contains 1 405:contains 2 306:starts 1 206:starts 1 307:equals 1 207:equals 1 2

Packing RangesTest case Start End01:precedes 1 201:precedes 3 402:meets 1 303:overlaps 1 404:finished by 1 305:contains 1 406:starts 1 307:equals 1 2

Test case Start End01:precedes 1 201:precedes 302:meets 103:overlaps 104:finished by 105:contains 106:starts 107:equals 1

select * from t match_recognize( partition by test_case order by end_date, start_date measures min(start_date) start_date, last(end_date) end_date pattern(a* b) define a as end_date >= next(start_date));

select * from t match_recognize( partition by test_case order by end_date, start_date measures min(start_date) start_date, last(end_date) end_date pattern(a* b) define a as end_date >= next(start_date) or end_date is null);

22

JOIN: range to range> create table A(start_n, end_n) asselect level, level+1 from dualconnect by level <= 10000;

> create table B as select start_n+9995 start_n, end_n+9996 end_nfrom A;

> select * from Ajoin B on (A.start_n <= B.start_n and B.start_n < A.end_n) or (B.start_n <= A.start_n and A.start_n < B.end_n);

Elapsed: 00:00:13.332

Exadata?

All data in buffer cache

Elapsed: 00:00:13.332

InMemory?Elapsed: 00:00:09.842

23

JOIN: range to range------------------------------------------------------------------------------------------| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:17.82 | 90 || 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:17.82 | 90 || 2 | CONCATENATION | | 1 | | 10 |00:00:00.01 | 90 || 3 | MERGE JOIN | | 1 | 55 | 10 |00:00:00.01 | 45 || 4 | SORT JOIN | | 1 | 10000 | 10000 |00:00:00.01 | 24 || 5 | TABLE ACCESS FULL | T_NEW | 1 | 10000 | 10000 |00:00:00.01 | 24 ||* 6 | FILTER | | 10000 | | 10 |00:00:00.01 | 21 ||* 7 | SORT JOIN | | 10000 | 10000 | 55 |00:00:00.01 | 21 || 8 | TABLE ACCESS FULL| T_OLD | 1 | 10000 | 10000 |00:00:00.02 | 21 || 9 | MERGE JOIN | | 1 | 55 | 0 |00:00:17.80 | 45 || 10 | SORT JOIN | | 1 | 10000 | 10000 |00:00:00.02 | 24 || 11 | TABLE ACCESS FULL | T_NEW | 1 | 10000 | 10000 |00:00:00.01 | 24 ||* 12 | FILTER | | 10000 | | 0 |00:00:17.78 | 21 ||* 13 | SORT JOIN | | 10000 | 10000 | 99M|00:01:21.50 | 21 || 14 | TABLE ACCESS FULL| T_OLD | 1 | 10000 | 10000 |00:00:00.01 | 21 |------------------------------------------------------------------------------------------

24

Join, or Sort and Match?

A 1 4B is equal 1 4

B started by A 1 5B during A 2 3B finishes A 3 4

B overlapped by A 3 4 5B met by A 4 5

B preceded by A 5 6another A 5 7

✔

✖

?

✔✔

✔✔

25

Join, or Sort and Match?

A 1 4B is equal 1 4

B started by A 1 5B during A 2 3B finishes A 3 4

B overlapped by A 3 4 5B met by A 4 5

B preceded by A 5 6another A 5 7

✖

?3

333

26

select A_start_n, A_end_n, B_start_n, B_end_n from ( select 'A' ttype, A.* from A union all select 'B' ttype, B.* from B) match_recognize ( order by start_n, end_n measures decode(f.ttype,'A',f.start_n, o.start_n) A_start_n, decode(f.ttype,'A',f.end_n, o.end_n) A_end_n, decode(f.ttype,'B',f.start_n, o.start_n) B_start_n, decode(f.ttype,'B',f.end_n, o.end_n) B_end_n all rows per match after match skip to next row pattern ( {-f-} (o|{-x-})+ ) define o as ttype != f.ttype and start_n < f.end_n, x as start_n < f.end_n);

Elapsed: 00:00:00.063

{- exclusion -}( grouping )+ at least oneAlternation A | B

✔✔

27

Child's

play

28

More!• Overlapping ranges with priority• Data warehouses with date ranges:– Trickle feed

• Impact on foreign keys• OLTP

• Take advantage of MATCH_RECOGNIZE ,