10/13/20151 CS 3343: Analysis of Algorithms Lecture 9: Review for midterm 1 Analysis of quick sort.

04/19/23 1

CS 3343: Analysis of Algorithms

Lecture 9: Review for midterm 1

Analysis of quick sort

04/19/23 2

Exam (midterm 1)

• Closed book exam

• One cheat sheet allowed (limit to a single page of letter-size paper, double-sided)

• Tuesday, Feb 24, 10:00 – 11:25pm

• Basic calculator (no graphing) is allowed

04/19/23 3

Materials covered

• Up to Lecture 8 (Feb 6)• Comparing functions• O, Θ, Ω

– Definition, limit method, L’Hopital’s rule, sterling’s formula• Analyzing iterative algorithms

– Know how to count the number of basic operations, and express the running time as a sum of a series

– Know how to compute the sum of a series (geometric, arithmetic, or other frequently seen series)

• Analyzing recursive algorithms– Define recurrence– Solve recurrence using recursion tree / iteration method– Solve recurrence using master method– Prove using substitution method

04/19/23 4

Asymptotic notations

• O: <=

• o: <

• Ω: >=

• ω: >

• Θ: =

(in terms of growth rate)

04/19/23 5

Mathematical definitions

• O(g(n)) = {f(n): positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) n>n0}

• Ω(g(n)) = {f(n): positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n) n>n0}

• Θ(g(n)) = {f(n): positive constants c1, c2, and n0 such that 0 c1 g(n) f(n) c2 g(n) n n0}

04/19/23 6

Big-Oh

• Claim: f(n) = 3n2 + 10n + 5 O(n2)

• Proof by definition:

f(n) = 3n2 + 10n + 5 3n2 + 10n2 + 5 , n > 1 3n2 + 10n2 + 5n2, n > 1

18 n2, n > 1If we let c = 18 and n0 = 1, we have f(n) c n2, n > n0. Therefore by definition, f(n) = O(n2).

04/19/23 7

Use limits to compare orders of growth

0• lim f(n) / g(n) = c > 0

∞n→∞

f(n) o(g(n))

f(n) Θ (g(n))

f(n) ω (g(n))

f(n) O(g(n))

f(n) Ω(g(n))

lim f(n) / g(n) = lim f(n)’ / g(n)’n→∞ n→∞

Condition:

If both lim f(n) and lim g(n) = ∞ or 0

L’ Hopital’s rule

Stirling’s formula

!n nn en 2/1(constant)

04/19/23 8

Useful rules for logarithms

For all a > 0, b > 0, c > 0, the following rules hold• logba = logca / logcb = lg a / lg b

– So: log10n = log2n / log2 10

• logban = n logba– So: log 3n = n log3 = (n)

•b

logba = a

– So: 2log

2n = n

• log (ab) = log a + log b– So: log (3n) = log 3 + log n = (log n)

• log (a/b) = log (a) – log(b)– So: log (n/2) = log n – log 2 = (log n)

• logba = 1 / logab• logb1 = 0

04/19/23 9

Useful rules for exponentials

• For all a > 0, b > 0, c > 0, the following rules hold• a0 = 1 (00 = ?) Answer: does not exist• a1 = a• a-1 = 1/a• (am)n = amn

• (am)n = (an)m

– So: (3n)2 = 32n = (32)n =9n

• aman = am+n

– So: n2 n3 = n5

– 2n 22 = 2n+2 = 4 * 2n = (2n)

04/19/23 10

More advanced dominance ranking

1)(logloglogloglog/log

loglogloglog/

!log~log23!

)3(

23

123

nnnnn

nnnnnnnn

nnnnnnnn nnn

04/19/23 11

Sum of arithmetic series

If a1, a2, …, an is an arithmetic series, then

2

)( 1

1

nn

ii

aana

04/19/23 12

Sum of geometric series

if r < 1

1

)1/()1(

)1/()1(1

1

0 n

rr

rr

r n

n

n

i

i if r > 1

if r = 1

?2

1lim

?2

1lim

?2

1

0

0

n

iin

n

iin

n

i

i

04/19/23 13

Sum manipulation rules

n

xii

x

mii

n

mii

i ii i

i ii ii ii

aaa

acca

baba

1

)(

Example:

n

ii

n

ii

n

i

n

i

nin

i

i

nnn

nnii

11

1 1

1

1

2

1

2

22)1(224)24(

04/19/23 14

Analyzing non-recursive algorithms

• Decide parameter (input size)

• Identify most executed line (basic operation)

• worst-case = average-case?

• T(n) = i ti

• T(n) = Θ (f(n))

04/19/23 15

Statement cost time__InsertionSort(A, n) {

for j = 2 to n { c1 n

key = A[j] c2 (n-1)

i = j - 1; c3 (n-1)

while (i > 0) and (A[i] > key) { c4 S

A[i+1] = A[i] c5 (S-(n-1))

i = i - 1 c6 (S-(n-1))

} 0

A[i+1] = key c7 (n-1)

} 0

}

Analysis of insertion Sort

04/19/23 16

Best case

• Array already sorted• S = j=1..n tj

• tj = 1 for all j• S = n. T(n) = Θ (n)

1 i j

sorted Key

Inner loop stops when A[i] <= key, or i = 0

04/19/23 17

Worst case

• Array originally in reverse order sorted• S = j=1..n tj

• tj = j• S = j=1..n j = 1 + 2 + 3 + … + n = n (n+1) / 2 = Θ (n2)

1 i j

sorted

Inner loop stops when A[i] <= key

Key

04/19/23 18

Average case

• Array in random order• S = j=1..n tj

• tj = j / 2 in average• S = j=1..n j/2 = ½ j=1..n j = n (n+1) / 4 = Θ (n2)

1 i j

sorted

Inner loop stops when A[i] <= key

Key

04/19/23 19

Analyzing recursive algorithms

• Defining recurrence relation

• Solving recurrence relation– Recursion tree (iteration) method– Substitution method– Master method

04/19/23 20

Analyzing merge sort

MERGE-SORT A[1 . . n]1. If n = 1, done.2. Recursively sort A[ 1 . . n/2 ]

and A[ n/2+1 . . n ] .3. “Merge” the 2 sorted lists

T(n)Θ(1)2T(n/2)

f(n)

T(n) = 2 T(n/2) + Θ(n)

04/19/23 21

Recursive Insertion Sort

RecursiveInsertionSort(A[1..n])

1. if (n == 1) do nothing;

2.RecursiveInsertionSort(A[1..n-1]);

3.Find index i in A such that A[i] <= A[n] < A[i+1];

4. Insert A[n] after A[i];

)(1)( nnTnT

04/19/23 22

Binary Search

BinarySearch (A[1..N], value) { if (N == 0)

return -1; // not foundmid = (1+N)/2;

if (A[mid] == value) return mid; // found

else if (A[mid] > value) return BinarySearch (A[1..mid-1], value); else

return BinarySearch (A[mid+1, N], value) }

)1(2

)(

nTnT

04/19/23 23

Recursion tree

Solve T(n) = 2T(n/2) + n.

n

n/4 n/4 n/4 n/4

n/2 n/2

(1)

…

h = log n

n

n

n

#leaves = n (n)

Total(n log n)

…

04/19/23 24

• Recurrence: T(n) = 2T(n/2) + n.• Guess: T(n) = O(n log n). (eg. by recursion tree

method)• To prove, have to show T(n) ≤ c n log n for

some c > 0 and for all n > n0

• Proof by induction: assume it is true for T(n/2), prove that it is also true for T(n). This means:

• Fact: T(n) = 2T(n/2) + n• Assumption: T(n/2)≤ cn/2 log (n/2)• Need to Prove: T(n)≤ c n log (n)

Substitution method

04/19/23 25

Proof

• To prove T(n) = O(n log n), we need to show that T(n) cn logn for some positive c and all sufficiently

large n. • Let’s assume this inequality is true for T(n/2), which

means T(n/2) cn/2 log(n/2)

• Substitute T(n/2) in the recurrence by the r.h.s. of the above inequality, we haveT(n) = 2 T(n/2) + n

2 * cn/2 log (n/2) + n cn (log n – 1) + n cn log n – (cn – n) cn log n for c ≥ 1 and all n ≥ 0.

Therefore, by definition, T(n) = O(n log n).

04/19/23 26

Master theoremT(n) = a T(n/b) + f (n)

CASE 1: f (n) = O(nlogba – ) T(n) = (nlogba) .

CASE 2: f (n) = (nlogba) T(n) = (nlogba log n) .

CASE 3: f (n) = (nlogba + ) and a f (n/b) c f (n)T(n) = ( f (n)) .

Optional: extended case 2

Key: compare f(n) with nlogba

Regularity Condition

04/19/23 27

Analysis of Quick Sort

04/19/23 28

Quick sort

• Another divide and conquer sorting algorithm – like merge sort

• Anyone remember the basic idea?

• The worst-case and average-case running time?

• Learn some new algorithm analysis tricks

04/19/23 29

Quick sort

Quicksort an n-element array:

1. Divide: Partition the array into two subarrays around a pivot x such that elements in lower subarray x elements in upper subarray.

2. Conquer: Recursively sort the two subarrays.

3. Combine: Trivial.

x x xx ≥ x≥ x

Key: Linear-time partitioning subroutine.

04/19/23 30

Partition

• All the action takes place in the partition() function– Rearranges the subarray in place– End result: two subarrays

• All values in first subarray all values in second

– Returns the index of the “pivot” element separating the two subarrays

x x xx ≥ x≥ xp rq

04/19/23 31

Pseudocode for quicksort

QUICKSORT(A, p, r)if p < r

then q PARTITION(A, p, r) QUICKSORT(A, p, q–1)QUICKSORT(A, q+1, r)

Initial call: QUICKSORT(A, 1, n)

04/19/23 32

Idea of partition

• If we are allowed to use a second array, it would be easy

66 1010 55 88 1313 33 22 1111

66 55 33 22 1111 1313 88 1010

22 55 33 66 1111 1313 88 1010

04/19/23 33

Another idea

• Keep two iterators: one from head, one from tail

66 1010 55 88 1313 33 22 1111

66 22 55 33 1313 88 1010 1111

33 22 55 66 1313 88 1010 1111

04/19/23 34

In-place Partition

66 1010 55 88 1313 33 22 11112 3 8 103 6

04/19/23 35

Partition In Words

• Partition(A, p, r):– Select an element to act as the “pivot” (which?)

– Grow two regions, A[p..i] and A[j..r]• All elements in A[p..i] <= pivot

• All elements in A[j..r] >= pivot

– Increment i until A[i] > pivot

– Decrement j until A[j] < pivot

– Swap A[i] and A[j]

– Repeat until i >= j

– Swap A[j] and A[p]

– Return j

Note: different from book’s partition(), which uses two iterators that both move forward.

04/19/23 36

Partition CodePartition(A, p, r) x = A[p]; // pivot is the first element i = p; j = r + 1; while (TRUE) {

repeat i++; until A[i] > x or i >= j; repeat j--; until A[j] < x or j < i; if (i < j) Swap (A[i], A[j]); else break;

} swap (A[p], A[j]); return j;

What is the running time of partition()?

partition() runs in (n) time

04/19/23 37

i j66 1010 55 88 1313 33 22 1111x = 6

p r

i j66 1010 55 88 1313 33 22 1111

i j66 22 55 88 1313 33 1010 1111

i j66 22 55 88 1313 33 1010 1111

i j66 22 55 33 1313 88 1010 1111

ij66 22 55 33 1313 88 1010 1111

33 22 55 66 1313 88 1010 1111qp r

scan

scan

scan

swap

swap

final swap

Partition example

04/19/23 38

66 1010 55 88 1111 33 22 1313

33 22 55 66 1111 88 1010 1313

22 33 55 66 88 1010 1111 1313

22 33 55 66 1010 88 1111 1313

22 33 55 66 88 1010 1111 1313

Quick sortexample

04/19/23 39

Analysis of quicksort

• Assume all input elements are distinct.

• In practice, there are better partitioning algorithms for when duplicate input elements may exist.

• Let T(n) = worst-case running time on an array of n elements.

04/19/23 40

Worst-case of quicksort

• Input sorted or reverse sorted.• Partition around min or max element.• One side of partition always has no elements.

)(

)()1(

)()1()1(

)()1()0()(

2n

nnT

nnT

nnTTnT

(arithmetic series)

04/19/23 41

Worst-case recursion treeT(n) = T(0) + T(n–1) + n

04/19/23 42


T(n)

04/19/23 43

n

T(0) T(n–1)


04/19/23 44

n

T(0) (n–1)


T(0) T(n–2)

04/19/23 45

n

T(0) (n–1)


T(0) (n–2)

T(0)

T(0)

04/19/23 46

n

T(0) (n–1)


T(0) (n–2)

T(0)

T(0)

2

1

nkk

height

height = n

04/19/23 47

n

T(0) (n–1)


T(0) (n–2)

T(0)

T(0)

2

1

nkk

n

height = n

04/19/23 48

n

(n–1)


(n–2)

(1)

2

1

nkk

n

height = n

(1)

(1)

(1)T(n) = (n) + (n2)

= (n2)

04/19/23 49

Best-case analysis(For intuition only!)

If we’re lucky, PARTITION splits the array evenly:

T(n) = 2T(n/2) + (n)= (n log n) (same as merge sort)

What if the split is always 109

101 : ?

)()(109

101 nnTnTnT

What is the solution to this recurrence?

04/19/23 50

Analysis of “almost-best” case

)(nT

04/19/23 51


nT101 nT

109

n

04/19/23 52


n101 n10

9

nT100

1 nT100

9 nT100

9 nT10081

n

04/19/23 53


n1001

(1)

(1)

… …log10/9n

n

n

n

…O(n) leavesO(n) leaves

n101 n10

9

n1009 n100

9 n10081

n

04/19/23 54

log10n


n

(1)

(1)

… …log10/9n

n

n

n

T(n) n log10/9n + (n)

…

n log10n

O(n) leavesO(n) leaves

(n log n)

n1001

n101 n10

9

n1009 n100

9 n10081

04/19/23 55

Quicksort Runtimes

• Best-case runtime Tbest(n) (n log n)

• Worst-case runtime Tworst(n) (n2)

• Worse than mergesort? Why is it called quicksort then?

• Its average runtime Tavg(n) (n log n )

• Better even, the expected runtime of randomized quicksort is (n log n)

04/19/23 56

Randomized quicksort

• Randomly choose an element as pivot– Every time need to do a partition, throw a die to

decide which element to use as the pivot– Each element has 1/n probability to be selected

Rand-Partition(A, p, r) d = random(); // a random number between 0 and 1 index = p + floor((r-p+1) * d); // p<=index<=r swap(A[p], A[index]); Partition(A, p, r); // now do partition using A[p] as pivot

04/19/23 57

Running time of randomized quicksort

• The expected running time is an average of all cases

T(n) =

T(0) + T(n–1) + dn if 0 : n–1 split,T(1) + T(n–2) + dn if 1 : n–2 split,T(n–1) + T(0) + dn if n–1 : 0 split,

nknTkTn

nTn

k

1

0)1()(

1)(

Expectation

04/19/23 58

nknTkTn

nTn

k

1

0)1()(

1)(

nkTn

n

k

1

0)(

2

04/19/23 59

Solving recurrence

1. Recursion tree (iteration) method- Good for guessing an answer

2. Substitution method- Generic method, rigid, but may be hard

3. Master method- Easy to learn, useful in limited cases only

- Some tricks may help in other cases

04/19/23 60

Substitution method

1. Guess the form of the solution:(e.g. using recursion trees, or expansion)

2. Verify by induction (inductive step).

The most general method to solve a recurrence (prove O and separately):

04/19/23 61

Expected running time of Quicksort

• Guess

• We need to show that for some c and sufficiently large n

• Use T(n) instead of for convenience

nkTn

nTn

k

1

0)(

2)(

)log()( nnOnT ncnnT log)(

)(nT

04/19/23 62

• Fact:

• Need to show: T(n) ≤ c n log (n)

• Assume: T(k) ≤ ck log (k) for 0 ≤ k ≤ n-1

• Proof:

nkTn

nTn

k

1

0)(

2)(

nkkn

c n

k

1

0

log2

ncn

ncn 4

log

ncn log if c ≥ 4. Therefore, by defintion, T(n) = (nlogn)

using the fact that 8

log2

log221

0

nn

nkk

n

k

n

nn

n

n

c

8log

2

2 22

nkTn

nTn

k

1

0

)(2

)(

04/19/23 63

What are we doing here?



The lg k in the second term is bounded by lg n

Tightly Bounding The Key Summation

1

2

12

1

1

2

12

1

1

2

12

1

1

1

1

0

lglg

lglg

lglg

lglg

n

nk

n

k

n

nk

n

k

n

nk

n

k

n

k

n

k

knkk

nkkk

kkkk

kkkk

Move the lg n outside the summation

Split the summation for a tighter bound

0lglim0

xxx

04/19/23 64

The summation bound so far

Tightly BoundingThe Key Summation

1

2

12

1

1

2

12

1

1

2

12

1

1

2

12

1

1

1

lg1lg

lg1lg

lg2lg

lglglg

n

nk

n

k

n

nk

n

k

n

nk

n

k

n

nk

n

k

n

k

knkn

knnk

knnk

knkkkk

What are we doing here?The lg k in the first term is bounded by lg n/2

What are we doing here?lg n/2 = lg n - 1

What are we doing here?Move (lg n - 1) outside the summation

04/19/23 65


Tightly BoundingThe Key Summation

12

1

12

1

1

1

1

2

12

1

12

1

1

2

12

1

1

1

2

)(1lg

lg

lglg

lg1lglg

n

k

n

k

n

k

n

nk

n

k

n

k

n

nk

n

k

n

k

knn

n

kkn

knkkn

knknkk

What are we doing here?Distribute the (lg n - 1)

What are we doing here?The summations overlap in range; combine them

What are we doing here?The Guassian series

04/19/23 66



48

1lglg

2

1

1222

1lg1

2

1

lg12

1

lg2

)(1lg

22

12

1

12

1

1

1

nnnnnn

nnnnn

knnn

knnn

kk

n

k

n

k

n

k

What are we doing here?Rearrange first term, place upper bound on second

What are we doing?Guassian series

What are we doing?Multiply it all out

04/19/23 67


!!Done!

2when8

1lg

2

1

)lg24

(8

1lg

2

1

48

1lglg

2

1lg

22

22

221

1

nnnn

nnn

nnn

nnnnnnkk

n

k

10/13/20151 CS 3343: Analysis of Algorithms Lecture 9: Review for midterm 1 Analysis of quick sort.

Documents