Dual-Pivot Quicksort - Asymmetries in Sorting

Dual-Pivot Quicksort — Asymmetries in Sorting

Sebastian Wild Markus E. Nebel[wild, nebel] @cs.uni-kl.de

Computer Science Department

Feb 2014Dagstuhl Seminar 14 091

Data Structures and Advanced Modelsof Computation on Big Data

Sebastian Wild Dual-Pivot Quicksort Feb. 2014 1 / 19

Sorting Algorithms in Practice

Many inventionsby algorithms comunity

vs. Few methodssuccessful in practice

C

C++

Java 6

Quicksort+Mergesort variant as stable sort

.NET

Haskell

Python Timsort

Sorting methods listed on Wikipedia Sorting methods of standard libraries for random access data


Sorting Algorithms in Practice

Many inventionsby algorithms comunity

vs. Few methodssuccessful in practice

C

C++

Java 6

Quicksort+Mergesort variant as stable sort

.NET

Haskell

Python Timsort

Sorting methods listed on Wikipedia Sorting methods of standard libraries for random access data


History of Quicksort in Practice

1961,62 Hoare: first publication, average case analysis

1969 Singleton: median-of-three & Insertionsort on small subarrays

1975-78 Sedgewick: detailled analysis of many optimizations

1993 Bentley, McIlroy: Engineering a Sort Function

1997 Musser: O(n logn) worst case by bounded recursion depth

Basic algorithm settled since 1961; latest tweaks from 1990’s.Since then: Almost identical in all programming libraries!

Until 2009: Java 7 switches to a new dual pivot Quicksort!

1961 1969 1975 ’78 1993 1997 today’62 ’77










1961 1969 1975 ’78 1993 1997 today’62 ’77










1961 1969 1975 ’78 1993 1997 today’62 ’77

2009


Discovery of Yaroslavskiy’s Algorithm

2009 Vladimir Yaroslavskiy (researcher in St. Petersburg)experiments with Quicksort with two pivots

11 Sep 2009 announcement on Java core library mailing list

29 Oct 2009 inclusion in development version of library

2009 – 2011 optimizations by Josh Bloch, Jon Bentley, many others

28 July 2011 public release of Java 7 with Yaroslavskiy’s Quicksort

1961 1969 1975 ’78 1993 1997’62 ’77

today2009


Running Time Experiments

Why switch to new, unknown algorithm?

Because it is faster!

0 0.5 1 1.5 2

·106

7

8

9

n

time

10−

6·n

lnn

Java 6 Library

Normalized Java runtimes (in ms).Average and standard deviation of1000 random permutations per size.

remains true for basic variants of algorithms: vs. !


Running Time Experiments

Why switch to new, unknown algorithm? Because it is faster!

0 0.5 1 1.5 2

·106

7

8

9

n

time

10−

6·n

lnn

Java 6 Library

Java 7 Library

Normalized Java runtimes (in ms).Average and standard deviation of1000 random permutations per size.

remains true for basic variants of algorithms: vs. !


Dual Pivot Quicksort

High Level Algorithm:

1 Partition array arround two pivots p 6 q.2 Sort 3 subarrays recursively.

How to do partitioning?

1 For each element x, determine its class

small for x < p

medium for p < x < q

large for q < x

by comparing x to p and/or q

2 Arrange elements according to classes p q


Dual Pivot Quicksort

High Level Algorithm:

1 Partition array arround two pivots p 6 q.2 Sort 3 subarrays recursively.

How to do partitioning?

1 For each element x, determine its class

small for x < p

medium for p < x < q

large for q < x

by comparing x to p and/or q

2 Arrange elements according to classes p q


Dual Pivot Quicksort — Previous Work

Using two pivots is not a new idea!

Robert Sedgewick, 1975

in-place dual pivot Quicksort implementation

more comparisons and swaps than classic Quicksort

Pascal Hennequin, 1991

list-based Quicksort with r pivots

r = 2 same #comparisons as classic Quicksort (∼ 2n lnn)

Using two pivots does not pay.


Dual Pivot Quicksort — Previous Work

Using two pivots is not a new idea!

Robert Sedgewick, 1975

in-place dual pivot Quicksort implementation

more comparisons and swaps than classic Quicksort

Pascal Hennequin, 1991

list-based Quicksort with r pivots

r = 2 same #comparisons as classic Quicksort (∼ 2n lnn)

Using two pivots does not pay.


Dual Pivot Quicksort — Comparison “Bound”

How many comparisons to determine classes ( small , medium or large ) ?

Assume compare with p first (Hennequin, 1991). small elements need 1, others 2 comparisons

On average:13 small elements

13 · 1+

23 · 2 =

53 comparisons per element

Any partitioning method requires53(n− 2) ∼ 20

12n comparisons, right?

No! (Stay tuned . . . )






13 · 1+

23 · 2 =










13 · 1+

23 · 2 =






Yaroslavskiy’s Algorithm

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

 q ?

< p ? skip

swap ` swap k

37

3 7

Cheap elements (1 cmp.):

small in k’s range ( s@K )

large in g’s range ( l@ G )

Key Observation: sizes of classes and ranges are correlated:

many small elements k’s range big

many large elements g’s range big

> 13 cheap elements

(on average)


Asymmetries in #Comparisons


 q ?

 13 cheap elements

(on average)


Comparisons per Partitioning Step

Can give full distribution of s@K and l@ G(hypergeometric conditional on p and q)

Expected #cmps:

cn ∼ 2n two cmps per element

− 312n small in k’s range

− 16n large in g’s range

= 1912 n

Recall: “lower bound” was 2012n.


Results — Basic Operations

Solving dual-pivot Quicksort recurrence gives:

Comparisons:

Yaroslavskiy needs only ∼ 1.9n lnn comparisons on average.

Classic Quicksort: ∼ 2n lnn

Swaps:

∼ 0.6n lnn swaps for Yaroslavskiy

∼ 0.3n lnn swaps for classic Quicksort

Which algorithm is faster?Result remains inconclusive!

Comparisons Swaps

0

0.5

1

1.5

2

−5%

+80%

Classic Quicksort

Yaroslavskiy


Results — Basic Operations

Solving dual-pivot Quicksort recurrence gives:

Comparisons:

Yaroslavskiy needs only ∼ 1.9n lnn comparisons on average.

Classic Quicksort: ∼ 2n lnn

Swaps:

∼ 0.6n lnn swaps for Yaroslavskiy

∼ 0.3n lnn swaps for classic Quicksort

Which algorithm is faster?Result remains inconclusive!

Comparisons Swaps

0

0.5

1

1.5

2

−5%

+80%

Classic Quicksort

Yaroslavskiy


Instruction Counts à la Knuth

How to force conclusive analysis?

More detailed cost model!

old days: Knuth’s MIX instructions

in the Java universe: Java Bytecode instructions

#Bytecodes highly correlated with running time


Instruction Counts à la Knuth

How to force conclusive analysis?

More detailed cost model!

old days: Knuth’s MIX instructions

in the Java universe: Java Bytecode instructions

#Bytecodes highly correlated with running time


Results — Primitive Instructions

Results:Yaroslavskiy’s algorithm: ∼ 21.7n lnn Bytecodes

Classic Quicksort: ∼ 18n lnn Bytecodes

Similar for Knuth’s new mythical machine MMIX

Yaroslavskiy: ∼ (13.1υ+2.8µ)n lnn

Classic Quicksort: ∼ (11υ+2.6µ)n lnn

υ (“oops”): CPU cycles; µ (“mems”): memory accesses

Classic Quicksort significantly better in all these measures . . .

→ These models ignore memory hierarchies; see Alex López-Ortiz’ talk!






















Asymmetries in #Comparisons — Revisited


 q ?

< p ? skip

swap ` swap k

37

3 7








Exploiting Asymmetries in #Comparisons

Motivation: medium elements always cause 2 cmps Should avoid medium elements if possible!

Thought Experiment:

Assume always τ1 · n / τ2 · n / τ3 · n elements small / medium / large.

 q ?

< p ? skip

swap ` swap k

37

3 7

< p p ≤ ◦ ≤ q ≥ qp q

Comparisons per paritioning step:

2n two per element− τ1 ( τ1 + τ2 )n small in k’s range

− τ3 τ3 n large in g’s range

Minimal for τ = (1, 0, 0) or τ = (0, 0, 1).




Thought Experiment:


 q ?

 q ?

 q ?

< p ? skip

swap ` swap k

37

3 7

< p p ≤ ◦ ≤ q ≥ qp q




Minimal for τ = (1, 0, 0) or τ = (0, 0, 1).



τ = (1, 0, 0) optimal w. r. t. one partitioning step

But: worst case for partitioning:

τ = (1, 0, 0) Symmetric case τ = ( 13, 13, 13)

Is τ = ( 12, 0, 1

2) better?

trade-off between partitioning costs and equality of split

What is the optimal τ ?






Is τ = ( 12, 0, 1

2) better?








Is τ = ( 12, 0, 1

2) better?




Results — Optimal τ

We could show: Overall ∼2− τ1 ( τ1 + τ2 ) − τ3 τ3

−∑3i=1 τi ln(τi)

n lnn cmps

optimal choiceτ ∗ ≈ (0.4288, 0.2688, 0.3024)

quite different from (13 ,13 ,13)

similar analysis for #Bytecodes:

τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

τ1

τ2

Optimal skew of pivots heavily depends on cost measure!





n lnn cmps

optimal choiceτ ∗ ≈ (0.4288, 0.2688, 0.3024)



τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

τ1

τ2






n lnn cmps

optimal choiceτ ∗ ≈ (0.4288, 0.2688, 0.3024)



τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

τ1

τ2






n lnn cmps

optimal choiceτ ∗ ≈ (0.4288, 0.2688, 0.3024)



τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

1.49311.5171

τ1

τ2






n lnn cmps

optimal choiceτ ∗ ≈ (0.4288, 0.2688, 0.3024)



τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

1.49311.5171

τ1

τ2






n lnn cmps

optimal choiceτ ∗ ≈ (0.4288, 0.2688, 0.3024)



τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

1.49311.5171

τ1

τ2



Conclusion

We’ve seen:Breaking symmetry in order of comparisons is beneficialin multi-pivot Quicksort

skewed pivots can amplify savings

trade-off between partitioning costs and balance in subproblem sizes

Open Questions:How about other input distributions?(up to now only random permutations)

Is Yaroslavskiy’s algorithm special ?Are there better asymmetric Quicksorts (for practice)?

→ Martin Aumüller’s talk


Conclusion

We’ve seen:Breaking symmetry in order of comparisons is beneficialin multi-pivot Quicksort

skewed pivots can amplify savings

trade-off between partitioning costs and balance in subproblem sizes

Open Questions:How about other input distributions?(up to now only random permutations)

Is Yaroslavskiy’s algorithm special ?Are there better asymmetric Quicksorts (for practice)?

→ Martin Aumüller’s talk


Choice of τ in Practice

Lesson for Practice?

exact quantiles too expensive use sampling!

sample size k controls deviation

limited to discrete quantiles

Java 7 library uses tertiles of five elements

E[τ2,4] =

19.298n lnn Bytecodes

Recall: Optimum (for #Bytecodes)

τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

Running time:

C++: 0.5% fasterJava: . . . depends on JIT

probably better choice:E[τ1,3] =









E[τ2,4] =



τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

Running time:











E[τ2,4] =



τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

Running time:











E[τ2,4] = 19.298n lnn Bytecodes


τ ∗BC ≈ (0.2068, 0.3485, 0.4447)

Running time:


probably better choice:E[τ1,3] = 18.791n lnn Bytecodes








E[τ2,4] = 19.298n lnn Bytecodes


τ ∗BC ≈ (0.2068, 0.3485, 0.4447)Running time:


probably better choice:E[τ1,3] = 18.791n lnn Bytecodes


Beating the “Lower Bound”

∼ 2012n comparisons only needed,

if there is one comparison location,then checks for x and y independent

But: Can have several comparison locations!

Here: Assume two locations C1 and C2 s. t.

C1 first compares with p.

C2 first compares with q.

C1 executed often, iff p is large.

C2 executed often, iff q is small.

C1 executed ofteniff many small elementsiff good chance that C1 needs only one comparison

(C2 similar)

less comparisons than 53 per elements on average


Yaroslavskiy’s Quicksort

DUALPIVOTQUICKSORTYAROSLAVSKIY(A, left, right)

1 if right − left > 12 p := A[left]; q := A[right]3 if p > q then Swap p and q end if4 ` := left + 1; g := right − 1; k := `5 while k 6 g6 if A[k] < p7 SwapA[k] andA[`] ; ` := `+ 18 else if A[k] > q9 while A[g] > q and k < g do g := g− 1 end while10 SwapA[k] andA[g] ; g := g− 111 if A[k] < p12 SwapA[k] andA[`] ; ` := `+ 113 end if14 end if15 k := k+ 116 end while17 ` := `− 1; g := g+ 118 SwapA[left] andA[`] ; SwapA[right] andA[g]19 DUALPIVOTQUICKSORTYAROSLAVSKIY(A, left , `− 1)20 DUALPIVOTQUICKSORTYAROSLAVSKIY(A, `+ 1,g− 1)21 DUALPIVOTQUICKSORTYAROSLAVSKIY(A,g+ 1, right )22 end if


Yaroslavskiy’s QuicksortDUALPIVOTQUICKSORTYAROSLAVSKIY(A, left, right)

1 if right − left > 12 p := A[left]; q := A[right]3 if p > q then Swap p and q end if4 ` := left + 1; g := right − 1; k := `5 while k 6 g6 Ck if A[k] < p7 SwapA[k] andA[`] ; ` := `+ 18 elseC′k if A[k] > q9 Cg while A[g] > q and k < g do g := g− 1 end while10 SwapA[k] andA[g] ; g := g− 111 C′g if A[k] < p12 SwapA[k] andA[`] ; ` := `+ 113 end if14 end if15 k := k+ 116 end while17 ` := `− 1; g := g+ 118 SwapA[left] andA[`] ; SwapA[right] andA[g]19 DUALPIVOTQUICKSORTYAROSLAVSKIY(A, left , `− 1)20 DUALPIVOTQUICKSORTYAROSLAVSKIY(A, `+ 1,g− 1)21 DUALPIVOTQUICKSORTYAROSLAVSKIY(A,g+ 1, right )22 end if

2 comparison locations

Ck handles pointer k

Cg handles pointer g

Ck first checks q

Cg first checks > q

C ′g if needed < p


Dual Pivot Quicksort Recurrence

Cn expected #comparisons to sort random permutation of {1, . . . , n}

Cn satisfies recurrence relation

Cn = Tn + 2n(n−1)

∑16p<q6n

(Cp−1 + Cq−p−1 + Cn−q

),

with Tn expected #comparisons in first partitioning step

recurrence solvable by standard methods

linear Tn ∼ a · n yields Cn ∼ 65a · n lnn.

It remains to compute Tn.


Analysis of Yaroslavskiy’s Algorithm (1)

first comparison for all elements (at Ck or Cg ) ∼ n comparisons

second comparison for some elements at C′k resp. C′g. . . but how often are C ′k resp. C ′g reached?

C ′k : all non- small elements reached by pointer k.

C ′g : all non- large elements reached by pointer g.

second comparison for medium elements not avoidable ∼ 1

3n comparisons in expectation

it remains to count:large elements reached by k andsmall elements reached by g.



Second comparisons for small and large elements?Depends on location!

C ′k number of large elements in k’s range: l@K

C ′g number of small elements in g’s range: s@G

p q

k’s range K g’s range G

l@K = 3 s@G = 2

l@K and s@G are random even for fixed p and q



Second comparisons for small and large elements?Depends on location!

C ′k number of large elements in k’s range: l@K

C ′g number of small elements in g’s range: s@G

p q

k’s range K g’s range G

l@K = 3 s@G = 2

l@K and s@G are random even for fixed p and q


Distribution of l@K and s@G (1)

Assume p and q are fixed.

1 Where do k and g cross?

Recall invariant: qg

←p 6 ◦ 6 q k

→?

k and g cross at (rank of) q|K| ∼ q, |G| ∼ n− q

2 How many small and large elements?

#small =∣∣{1, . . . , p− 1}∣∣ = p− 1

#large =∣∣{q+ 1, . . . , n}

∣∣ = n− q

|K|, |G|, #small and #large are constant (for given p and q).



Assume p and q are fixed.

1 Where do k and g cross?

Recall invariant: qg

←p 6 ◦ 6 q k

→?

k and g cross at (rank of) q|K| ∼ q, |G| ∼ n− q

2 How many small and large elements?

#small =∣∣{1, . . . , p− 1}∣∣ = p− 1

#large =∣∣{q+ 1, . . . , n}

∣∣ = n− q

|K|, |G|, #small and #large are constant (for given p and q).



3 Conditional Distribution of l@K

We draw positions of large elements at random.n− 2 positionsdraw #large positions without replacement|K| positions contribute to l@K

l@KD= Hypergeometric (#large, |K|, n− 2)

= Number of red balls obtainedwhen drawing #large balls without replacementform urn with n− 2 balls, exactly |K| of which are red.

E [l@K |p, q] =#large · |K|

n− 2∼

1 , 2

(n− q) · qn− 2

law of total expectation:

E [l@K] =∑

16p<q6n

1/(n2

)· (n− q) · q

n− 2∼ 1

6n



3 Conditional Distribution of l@K

We draw positions of large elements at random.n− 2 positionsdraw #large positions without replacement|K| positions contribute to l@K

l@KD= Hypergeometric (#large, |K|, n− 2)

= Number of red balls obtainedwhen drawing #large balls without replacementform urn with n− 2 balls, exactly |K| of which are red.

E [l@K |p, q] =#large · |K|

n− 2∼

1 , 2

(n− q) · qn− 2

law of total expectation:

E [l@K] =∑

16p<q6n

1/(n2

)· (n− q) · q

n− 2∼ 1

6n

Dual-Pivot Quicksort - Asymmetries in Sorting

Technology