A little bit of recurrences here.. From CLRS. Dr. M. Sakalli, Marmara University

A little bit of recurrences here.. From CLRS. A little bit of recurrences here.. From CLRS. Dr. M. Sakalli, Marmara UniversityDr. M. Sakalli, Marmara UniversityPicture May 5Picture May 5 thth 2006, RPI, NY 2006, RPI, NY

2-2M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes

Plan for Analysis of Recursive AlgorithmsPlan for Analysis of Recursive Algorithms

o Decide on a parameter indicating an Decide on a parameter indicating an input’sinput’s sizesize..

o Identify the Identify the algorithm’salgorithm’s basicbasic operation. operation.

o Check whether the number of times the basic op. is executed Check whether the number of times the basic op. is executed may vary on different inputs of the same size. (If it may, the may vary on different inputs of the same size. (If it may, the worst, averageworst, average, and , and bestbest cases must be investigated cases must be investigated separately.)separately.)

o Set up aSet up a recurrencerecurrence relation with an relation with an appropriate initial appropriate initial conditioncondition expressing the number of times the basic op. is expressing the number of times the basic op. is executed.executed.

o Solve the recurrenceSolve the recurrence (or, at the very least, establish its (or, at the very least, establish its solution’s order of growth) by backward substitutions or solution’s order of growth) by backward substitutions or another method.another method.


Three methods for solving recurrences -- Three methods for solving recurrences -- obtaining obtaining θθ or big-O or big-O bounds on T(n):bounds on T(n):

1.1. The The substitution methodsubstitution method: make a : make a guessguess and and then prove the guess is then prove the guess is correct by inductioncorrect by induction..

2.2. The recursion-tree method: convert the recurrence into The recursion-tree method: convert the recurrence into a tree of a tree of costs & sumcosts & sum them. them.

3.3. The "The "master methodmaster method": gives bounds for recurrences of the form: ": gives bounds for recurrences of the form: T(n) = aT(n/b)+f(n) where a T(n) = aT(n/b)+f(n) where a ≥≥ 1, b > 1, and f(n) is given positive 1, b > 1, and f(n) is given positive funtion.funtion.


Substitution method, two stepsSubstitution method, two steps-Make a guess for the solution-Make a guess for the solution

-Use inductive method to prove that the initial guess holds. -Use inductive method to prove that the initial guess holds.

-Check the valid initial conditions. -Check the valid initial conditions.

To Determine upper boundTo Determine upper bound

1- for a choice of 1- for a choice of cc>0, prove that >0, prove that

2- 2- ff((nn) ) ≤≤ c gc g((nn), for every ), for every nn ≥≥ nn00, T(, T(nn) ) ≤≤ c nc n ln( ln(nn)) (2)(2)

f(n) = O(g(n))f(n) = O(g(n))

We say that We say that “f(n) is big-O of g(n).”“f(n) is big-O of g(n).”

As n increases, f(n) grows no faster than g(n). As n increases, f(n) grows no faster than g(n).

In other words, g(n) is an In other words, g(n) is an asymptotic upper bound asymptotic upper bound on f(n) for on f(n) for c>0, and c>0, and nn ≥≥ nn00..


Big-Big- notation notation

To Determine lower boundTo Determine lower bound

1- for a choice of 1- for a choice of cc>0, prove that >0, prove that

2- 2- c c gg((nn) ) ≤ ≤ ff((nn), for all ), for all nn ≥≥ nn00, , c nc n ln( ln(nn) ) ≤≤ T( T(nn) ) (2)(2)

We say that “We say that “ff((nn) ) is omega of is omega of gg((nn))..””

As As n n increases, increases, ff((nn) ) grows no slower than grows no slower than gg((nn))..

In other words, In other words, gg((nn) ) is an is an asymptotic lower bound asymptotic lower bound on on ff((nn))..


Prove nProve n22 + + n n = = OO((nn33), or question would have been: ), or question would have been: Prove that f(n) is big-O of g(n)=nProve that f(n) is big-O of g(n)=n33..

f(n) f(n) ≤ ≤ c g(n), for all n>nc g(n), for all n>n00..

for nfor n00=1, c=1, c≥≥2, for all n>n2, for all n>n00, this gives , this gives nn22 + + n n ≤ ≤ 2 2 nn33, , end of proof. end of proof.

or we could have chosen, for nor we could have chosen, for n00=2, c=2, c≥≥1..1..


The same example: Determine upper bound of T(n) = 2T(The same example: Determine upper bound of T(n) = 2T(n/2n/2) + n ) + n (1)(1)

1- for a choice of 1- for a choice of cc, prove that , prove that

2- 2- ff((nn) ) ≤≤ c gc g((nn), for every ), for every nn ≥≥ nn00, T(, T(nn) ) ≤≤ c nc n ln( ln(nn)) (2)(2)

This is the guess made g(n)=This is the guess made g(n)=nn ln( ln(nn), ),

Hence start with by assuming that Hence start with by assuming that T(T(nn) ) ≤≤ cncn lg( lg(nn)) holds for f loor( holds for f loor(nn/2), i.e. that/2), i.e. that

T(T(n/2n/2) ) ≤≤ c c n/2n/2 lg( lg(n/2n/2))

Substitute this into the equation above: Substitute this into the equation above:

T(T(nn) ) ≤≤ 2[2[c c n/2n/2 lg( lg(n/2n/2)] + )] + n n

≤≤ cc nn lg( lg(nn/2) + /2) + n n = = c nc n lg( lg(nn) - cn lg(2) + n = cn lg(n) - cn + n) - cn lg(2) + n = cn lg(n) - cn + n

≤≤ c nc n lg( lg(nn), this will hold if ), this will hold if cc ≥≥ 1. 1. (2)(2)

-To complete the inductive proof, determine -To complete the inductive proof, determine cc and and nn00 .. And show that inequality holds for the boundary .. And show that inequality holds for the boundary

conditions. conditions.

-T(1)>0, but lg(1)=0, which seems doesn’t satisfy the equation. -T(1)>0, but lg(1)=0, which seems doesn’t satisfy the equation.

-Then work out between equations (1) and (2), -Then work out between equations (1) and (2),

obtain obtain nn=2, T(2) = 2T(1)+2 =2, T(2) = 2T(1)+2 ≤ ≤ c c 2 lg(2), T(3) 2 lg(2), T(3) =2T(1)+3=2T(1)+3 << cc 3 lg(3).. 3 lg(3).. mustmust

-Determining c requires establishing the base cases for T(1).. -Determining c requires establishing the base cases for T(1)..

- (2T(1)+2)/ (- (2T(1)+2)/ (2 lg(2)2 lg(2) ) )≤ ≤ cc T(1)+1 < T(1)+1 < cc, (, (2T(1)+3)2T(1)+3) /(3 lg(3)) /(3 lg(3))<< cc, ,

- Chose the maximum of {(T(1)+1), (- Chose the maximum of {(T(1)+1), (2T(1)+3)2T(1)+3) /(3 lg(3))} /(3 lg(3))} T(1)+1, therefore for T(1) =1, c=2. T(1)+1, therefore for T(1) =1, c=2.


Sometimes manipulation variables can make an unknown Sometimes manipulation variables can make an unknown recurrence look like equation. recurrence look like equation.

For example, consider the recurrence:For example, consider the recurrence:T(n) = 2T(T(n) = 2T(√√nn) + lg n) + lg n

Let m = lg(n), then n = 2Let m = lg(n), then n = 2mm::T(2T(2mm) = 2T(2) = 2T(2m/2m/2) + m) + mAnd now, let S(m') = T(2And now, let S(m') = T(2mm):):

S(m') = 2S(m'/2) + m'S(m') = 2S(m'/2) + m'Solution!! S(m') = O(m' lg m'). Changing back to T(n), Solution!! S(m') = O(m' lg m'). Changing back to T(n), T(n) = T(2T(n) = T(2mm) = S(m') = O(lg n lg lg n).) = S(m') = O(lg n lg lg n).


Example 1: Substitution method: Recursive evaluation of Example 1: Substitution method: Recursive evaluation of nn!!

Definition:Definition: n n ! = 1 ! = 1 2 2 … … ((n-n-1) 1) nn for for n n ≥≥ 1 and 0! = 11 and 0! = 1Recursive definition of Recursive definition of nn!: !: FF((nn) = ) = FF((n-n-1) 1) nn for for n n ≥≥ 1 and 1 and FF((0) = 10) = 1

Size: nSize: nBasic operation: Multiplication Basic operation: Multiplication Recurrence relation: Recurrence relation: M(n) = M(n-1) + 1, M(0) = 0

M(M(nn) = M() = M(nn-1) + 1, n>0, M(0) = 0-1) + 1, n>0, M(0) = 0

M(M(nn) = M() = M(nn-2) + 1 + 1-2) + 1 + 1……M(M(nn) = M() = M(nn--ii) + ) + ii, , ii==nn, , M(M(nn) =) =nn....


Example 3: Counting #bitsExample 3: Counting #bits

A(2A(2kk) = A(2) = A(2kk-1-1)+1,)+1,A(2A(200) = 0…) = 0…

A(2A(2kk) ) = A(2= A(2kk-1-1)+1,)+1, = [A(2= [A(2kk-2-2)+1]+1.)+1]+1.…….. = A(2= A(2kk--kk)+)+k k n n=2=2kk kk = log2( = log2(nn) ) (log((log(nn))))


The recursion-tree methodThe recursion-tree methodIn a recursion tree, each node represents the cost of a recursive call of the algorithm In a recursion tree, each node represents the cost of a recursive call of the algorithm

somewhere in its recursive invocation, but ignoring the cost of further recursive calls somewhere in its recursive invocation, but ignoring the cost of further recursive calls from within it, which are accounted for by "child" costs of that node. from within it, which are accounted for by "child" costs of that node.

The costs of the nodes are summed to get the total cost at each level and finally the sum of the The costs of the nodes are summed to get the total cost at each level and finally the sum of the per level costs... per level costs...

Recursion trees are especially useful for the T(Recursion trees are especially useful for the T(nn) of a divide-and-conquer algorithm, they are ) of a divide-and-conquer algorithm, they are used to generate good guesses, which can then be verified by the substitution method. used to generate good guesses, which can then be verified by the substitution method. Hence, a bit of "sloppiness" to be tolerated. Hence, a bit of "sloppiness" to be tolerated.

Find an upper bound for T(n) = 3T(Find an upper bound for T(n) = 3T(n/4n/4) + ) + θθ(n^2) (n^2) Ignore the floor function and non-integer values of n/4, so that Ignore the floor function and non-integer values of n/4, so that T(n) = 3T(n/4) + cn^2 for some c > 0.T(n) = 3T(n/4) + cn^2 for some c > 0. cn^2cn^2 / | \/ | \ / | \/ | \ / | \/ | \ T(n/4) T(n/4) T(n/4)T(n/4) T(n/4) T(n/4)Continue this process until reaching the boundary which is a subproblem of size 1 at depth i, Continue this process until reaching the boundary which is a subproblem of size 1 at depth i, n/4n/4ii = 1, = 1, ii = (lg(n))/2 =log = (lg(n))/2 =log44(n). Thus the tree has (lg(n)/2) + 1 levels: 0, 1, 2, ..., (lg(n))/2.(n). Thus the tree has (lg(n)/2) + 1 levels: 0, 1, 2, ..., (lg(n))/2.

Non-recursive cost at each level decreasing by a factor of 4 and three times as many of them Non-recursive cost at each level decreasing by a factor of 4 and three times as many of them as in the previous level.. So at each level the cost is c(n/4as in the previous level.. So at each level the cost is c(n/4 ii))22 and 3and 3ii of branches, c3 of branches, c3ii(n/4(n/4ii))22 ,= ,= (3/16)(3/16)i i cncn22 … …


At the last level of depth logAt the last level of depth log44(n), there are 3(n), there are 3lg(n)/2lg(n)/2 = 3 = 3loglog44(n)(n) = n = nloglog44(3)(3) nodes of cost T(1) for a total cost of nodes of cost T(1) for a total cost of θθ(n(nloglog44(3)(3)=n=nloglog44(3)(3) ). ).

Thus the total cost of all levels is: 4.2.3Thus the total cost of all levels is: 4.2.3 loglog44(n)-1(n)-1 inf infT(n) = T(n) = ΣΣ (3/16) (3/16)ii cn cn22 + + θθ(n(nloglog44(3)(3)) < ) < ΣΣ( (3/16)( (3/16)ii cn cn22) + ) + θθ(n(nloglog44(3)(3))) i=0i=0 i=0 i=0

= (1/(1 - 3/16)) cn= (1/(1 - 3/16)) cn22 + + θθ(n(nloglog44(3)(3)) = 16/13 cn) = 16/13 cn22 + + θθ(n(nloglog44(3)(3)) )

= O(n= O(n22))

Which gives the initial guess that would have been chosen at the fist hand. Which gives the initial guess that would have been chosen at the fist hand.

Now use the substitution method to verify the guess, T(n) = O(nNow use the substitution method to verify the guess, T(n) = O(n22) as an upper bound) as an upper bound T(n) = 3T(T(n) = 3T(n/4n/4) + ) + θθ(n(n22))

Need to show T(n) Need to show T(n) ≤≤ dn dn22 for some d > 0. Letting c be as before, we have: for some d > 0. Letting c be as before, we have:T(n) T(n) ≤≤ 3T( 3T(n/4n/4) + cn) + cn22

≤≤ 3d(3d(n/4n/4))22 + cn + cn22 by ind. hyp. by ind. hyp. ≤≤ 3d(n/4)3d(n/4)22 + cn + cn22

= (3/16)dn= (3/16)dn22 + cn + cn22

≤≤ dndn22

Choose d Choose d ≥≥ (16/13)c (16/13)c


• TT((nn) = ) = ((nnloglogbbaa)+ )+ ΣΣj=0j=0

loglogbbn-1n-1

aajjff((nn//bbjj))

j=0

logbn-1


The master theoremThe master theoremLet a Let a ≥≥ 1 and b > 1, 1 and b > 1, f(n) asymptotically positive!!!f(n) asymptotically positive!!!, and let T(n) be defined , and let T(n) be defined

by: by: T(n) = aT(n/b) + f(n) T(n) = aT(n/b) + f(n)

where n/b can be interpreted to be rounded either up or down. Then where n/b can be interpreted to be rounded either up or down. Then T(n) can be bounded asymptotically:T(n) can be bounded asymptotically:

1.1. If f(n) = O(If f(n) = O(nnloglogbb((aa)-)-ЄЄ ) for some ) for some ЄЄ> 0, then T(n) = > 0, then T(n) = θθ(n(nloglogbb((aa))))2.2. If f(n) = If f(n) = θθ(n(nloglogbb((aa))), k), k≥≥0, then T(n) = 0, then T(n) = θθ(n(nloglogbb((aa)) log logk+1k+1(n) )(n) )3.3. If f(n) = If f(n) = ΩΩ((nnloglogbb((aa)+)+ЄЄ ) for some ) for some ЄЄ> 0, and if regularity condition, > 0, and if regularity condition,

af(n/b) af(n/b) ≤≤ cf(n) cf(n) for some c < 1 and for some c < 1 and for all n > nfor all n > n00 for some for some nn00>0,>0, and then T(n) = and then T(n) = θθ(f(n)).(f(n)).

The regularity condition implies that if f(n), the driving cost is not a The regularity condition implies that if f(n), the driving cost is not a polynomial then a limited 4polynomial then a limited 4thth condition allows considering polylog condition allows considering polylog factorsfactors

To cover the gap between 1-2 and 2-3, corollary below.. To cover the gap between 1-2 and 2-3, corollary below.. Corollary if Corollary if ff(n) is (n) is θθ(n(nloglogbb((aa) ) lglgkk(n)), then T(n) = (n)), then T(n) = θθ(n(nloglogbb((aa)) lg lgk+1k+1(n) )(n) )For example T(n)= 2 T(n/2) + n log n. nFor example T(n)= 2 T(n/2) + n log n. n lg2lg2 =n vs (n lgn). Since f(n)=n lgn is =n vs (n lgn). Since f(n)=n lgn is

asymptotically less than nasymptotically less than n1+1+ for any for any . This which makes the case 2.). This which makes the case 2.)a=2, b=2, non-polynomial f(n), however f(n)a=2, b=2, non-polynomial f(n), however f(n)(n log n), k=1, therefore (n log n), k=1, therefore (n (n

loglog22 n)!! n)!!


If f(n)=nIf f(n)=nd d no need to check regularity, just compare both side. no need to check regularity, just compare both side.

Master TheoremMaster Theorem: If T(n) = aT(n/b) + O(n: If T(n) = aT(n/b) + O(ndd) for some ) for some constants a ≥ 1, b > 1, d ≥ 0, thenconstants a ≥ 1, b > 1, d ≥ 0, then

(n(nloglogbbaa))if d < logif d < logbba, (a > ba, (a > bdd) )

T(n) = T(n) = (n(nddlgn) lgn) if d = logif d = logbba, (a = a, (a =

bbdd) ) (n(ndd)) if d > logif d > logbba, (a < a, (a <

bbdd) )

Why? The proof uses a recursion tree argument. Why? The proof uses a recursion tree argument.

case 1: case 1: ff(n) is "polynomially smaller than (n) is "polynomially smaller than ((nnloglogbbaa))case 2: case 2: ff(n) is "asymptotically equal to (n) is "asymptotically equal to ((nnloglogbbaa))case 3: case 3: ff(n) is "polynomially larger than (n) is "polynomially larger than ((nnloglogbbaa))Corollary case4: if Corollary case4: if ff((nn)=)=((nnloglogbbaalglgkknn), then ), then TT((nn)=)=((nnloglogbbaalglgkk+1+1nn). ).

(as exercise)(as exercise)


Exp1: Exp1: T(n) = 9T(n/3) + nT(n) = 9T(n/3) + n• f(n) = n, nf(n) = n, nloglogbba a = n = n22

• compare f(n) = n with the cost of recursiveness.. compare f(n) = n with the cost of recursiveness..

n = O(nn = O(n2 -2 -) () ( =1, so f(n) is polynomially smaller than n =1, so f(n) is polynomially smaller than n loglogbbaa ), case 1 will ), case 1 will

apply here T(n) = apply here T(n) = (n(n22))Exp2: Exp2: T(n) = T(n/2) + 1T(n) = T(n/2) + 1• f(n) = 1, nf(n) = 1, nloglogbba a = n = n0 0 = 1= 1

1 = 1 = (n(n00) ) (driving cost f(n) is polynomial equal to the cost of recursiveness, n(driving cost f(n) is polynomial equal to the cost of recursiveness, n loglogbbaa ) )• case 2 applies: T(n) = case 2 applies: T(n) = (n(n00lgn) = lgn) = (lgn) (lgn) Exp3: Exp3: T(n) = T(n/2) + nT(n) = T(n/2) + n22

• D(n) = nD(n) = n22, n, nloglogbba a = n = nloglog221 1 = n= n0 0 = 1= 1

nn22 = = (n(n0+0+) (0<) (0<<=2, so D(n) is polynomially larger)<=2, so D(n) is polynomially larger)• Since D(n) is a polynomial in n, case 3 applies T(n) = Since D(n) is a polynomial in n, case 3 applies T(n) = (n(n22))

Exp4: Exp4: T(n) = 4T(n/2) + nT(n) = 4T(n/2) + n22

D(n) = nD(n) = n22, n, nloglogbbaa = n = nlg4lg4 = n = n22 compare both sides, D(n) with ncompare both sides, D(n) with n22 polynomially equal, case 2 holds, T(n) = polynomially equal, case 2 holds, T(n) = (n(n22lgn)lgn)


Exp5: Exp5: T(n) = 7T(n/3) + nT(n) = 7T(n/3) + n22

• f(n) = nf(n) = n22, n, nloglogbba a = n = nloglog337 7 = n= n1+ 1+

• compare f(n) = ncompare f(n) = n22 with n with n1+ 1+ , n, n22 = = (n(n1+1+) so f(n) is polynomially larger) so f(n) is polynomially larger• Since f(n) is a polynomial in n, case 3 holds and T(n) = Since f(n) is a polynomial in n, case 3 holds and T(n) = (n(n22))Exp6: Exp6: T(n) = 7T(n/2) + nT(n) = 7T(n/2) + n22

• nnloglogbba a = n = nloglog227 7 = n= n2+ 2+

• compare f(n) = ncompare f(n) = n22 with n with n2+ 2+ f(n) is polynomially smaller f(n) is polynomially smaller• Case 1 holds and T(n) = Case 1 holds and T(n) = (n(nloglog2277))Exp7: T(Exp7: T(nn) = 3T() = 3T(nn/4) + /4) + nn lg lgnn • f(n) = f(n) = nlgn, recursive side nnlgn, recursive side nloglog443 3 = = ((nn1- 1- ))

• comparing both sides, case 3 applies here, since f(n)=log(n), we need to check comparing both sides, case 3 applies here, since f(n)=log(n), we need to check regularity condition for sufficiently large n regularity condition for sufficiently large n • af(n/b) = 3(n/4)lg(n/4) af(n/b) = 3(n/4)lg(n/4) c n lg n = cf(n) for c=3/4 c n lg n = cf(n) for c=3/4 • T(n) = T(n) = (n lg n)(n lg n)

•Use recursion tree for a good guess and give a proof of your result by using Use recursion tree for a good guess and give a proof of your result by using substitution method for substitution method for •T(n)=3T(n/4) + cnT(n)=3T(n/4) + cn22

•T(n)=T(n/3) + T(2n/3) + cnT(n)=T(n/3) + T(2n/3) + cn


Proof of Master Theorem, CLRS, Better to Proof of Master Theorem, CLRS, Better to read from Chapter 4 of CLRS. read from Chapter 4 of CLRS.

o The proof for the exact powers, The proof for the exact powers, nn==bbkk for for kk1.1.o Lemma 4.2Lemma 4.2 (1) if (1) if nn=1=1

• forfor T T((nn) = ) = • aTaT((nn//bb)+)+ff((nn) if ) if nn==bbkk for for kk11

• where where aa 1, 1, bb>1, >1, ff((nn) be a nonnegative function,) be a nonnegative function,• Then Then • TT((nn) = ) = ((nnloglogbbaa)+ )+ aajjff((nn//bbjj))

o Proof: Proof: • By iterating the recurrenceBy iterating the recurrence• By recursion tree By recursion tree

j=0

logbn-1

2-19M, Sakalli, CS246 Design & Analysis of Algorithms, Lecture Notes19

Recursion tree for Recursion tree for TT((nn)=)=aTaT((nn//bb)+)+ff((nn) )


Proof of Master Theorem (cont.)Proof of Master Theorem (cont.)

Lemma 4.3:Lemma 4.3:• Let Let aa 1, 1, bb>1, >1, ff((nn) be a nonnegative function defined on exact power of ) be a nonnegative function defined on exact power of bb, ,

then then

• gg((nn)= )= aajjff((nn//bbjj) can be bounded for exact power of ) can be bounded for exact power of bb as: as:

1.1. If If ff((nn)=)=OO((nnloglogbbaa--) for some ) for some >0, then >0, then gg((nn)= )= OO((nnloglogbb

aa).).2.2. If If ff((nn)= )= ((nnloglogbb

aa), then ), then gg((nn)= )= ((nnloglogbbaa lg lg nn).).

3.3. If If ff((nn)= )= ((nnloglogbbaa++) for some ) for some >0 and if >0 and if afaf((nn//bb) ) cfcf((nn) for some ) for some cc<1 and all <1 and all

sufficiently large sufficiently large n n bb, then , then gg((nn)= )= ((ff((nn)).)).

j=0

logbn-1


Proof of Lemma 4.3Proof of Lemma 4.3

o For case 1: For case 1: ff((nn)=)=OO((nnloglogbbaa--) implies ) implies ff((nn//bbjj)=)=OO((((n n //bbjj))loglogbb

aa--), so), so

o gg((nn)= )= aajjff((nn//bbjj) =) =OO( ( aajj((n n //bbjj))loglogbbaa-- ))

o = = OO((nnloglogbbaa-- aajj/(/(bbloglogbb

aa--))jj ) = ) = OO((nnloglogbbaa-- aajj/(/(aajj((bb--))jj))))

o = = OO((nnloglogbbaa-- ((bb))jj ) = ) = OO((nnloglogbb

aa-- ((((((bb )) loglogbbnn-1)/(-1)/(bb-1) ) -1) )

o = = OO((nnloglogbbaa-- ((((((bbloglogbbnn)) -1)/(-1)/(bb-1)))=-1)))=OO((nnloglogbb

a a nn-- ((nn -1)/(-1)/(bb-1))-1))o = = OO((nnloglogbb

a a ))

j=0

logbn-1

j=0

logbn-1

j=0

logbn-1

j=0

logbn-1

j=0

logbn-1


Proof of Lemma 4.3(cont.)Proof of Lemma 4.3(cont.)

o For case 2: For case 2: ff((nn)= )= ((nnloglogbbaa) implies ) implies ff((nn//bbjj)= )= ((((n n //bbjj))loglogbb

aa), so), so

o gg((nn)= )= aajjff((nn//bbjj) = ) = ( ( aajj((n n //bbjj))loglogbbaa))

o = = ((nnloglogbbaa aajj/(/(bbloglogbb

aa))jj ) = ) = ((nnloglogbbaa 1)1)

o = = ((nnloglogbbaa log logbb

nn) = ) = ((nnloglogbbaalglg n n) )

j=0

logbn-1

j=0

logbn-1

j=0

logbn-1

j=0

logbn-1


Proof of Lemma 4.3(cont.)Proof of Lemma 4.3(cont.)

o For case 3:For case 3:• Since Since gg((nn) contains ) contains ff((nn), ), gg((nn) = ) = ((ff((nn)) ))

• Since Since afaf((nn//bb) ) cfcf((nn), ), aajjff((nn//bbjj) ) ccjjff((nn) , why???) , why???

• gg((nn)= )= aajjff((nn//bbjj) ) ccjjff((nn) ) ff((nn) ) ccjj

• ==ff((nn)(1/(1-)(1/(1-cc)) =O()) =O(ff((nn))))

• Thus, Thus, gg((nn)=)=((ff((nn))))

j=0

logbn-1

j=0

logbn-1

j=0


Proof of Master Theorem (cont.)Proof of Master Theorem (cont.)o Lemma 4.4:Lemma 4.4:

• forfor T T((nn) = ) = (1) if (1) if nn=1=1

• aTaT((nn//bb)+)+ff((nn) if ) if nn==bbkk for for kk11

• where where aa 1, 1, bb>1, >1, ff((nn) be a nonnegative function,) be a nonnegative function,

1.1. If If ff((nn)=)=OO((nnloglogbbaa--) for some ) for some >0, then >0, then TT((nn)= )= ((nnloglogbb

aa).).

2.2. If If ff((nn)= )= ((nnloglogbbaa), then ), then TT((nn)= )= ((nnloglogbb

aa lg lg nn).).

3.3. If If ff((nn)=)=((nnloglogbbaa++) for some ) for some >0, and if >0, and if afaf((nn//bb) ) cfcf((nn) for some ) for some cc<1 <1

and all sufficiently large and all sufficiently large nn, then , then TT((nn)= )= ((ff((nn)).)).


Proof of Lemma 4.4 (cont.)Proof of Lemma 4.4 (cont.)o Combine Lemma 4.2 and 4.3, Combine Lemma 4.2 and 4.3,

• For case 1: For case 1: – TT((nn)= )= ((nnloglogbb

aa)+)+OO((nnloglogbbaa)=)=((nnloglogbb

aa).).

• For case 2: For case 2: – TT((nn)= )= ((nnloglogbb

aa)+)+((nnloglogbbaa lg lg nn)=)=((nnloglogbb

aa lg lg nn).).

• For case 3:For case 3:– TT((nn)= )= ((nnloglogbb

aa)+)+((ff((nn))=))=((ff((nn)) because )) because ff((nn)= )= ((nnloglogbbaa++).).


Floors and Ceilings Floors and Ceilings o TT((nn)=)=aTaT((nn//bb)+)+ff((nn) and ) and TT((nn)=)=aTaT((nn//bb)+)+ff((nn))o Want to prove both equal to Want to prove both equal to TT((nn)=)=aTaT((nn//bb)+)+ff((nn))o Two results:Two results:

• Master theorem applied to all integers Master theorem applied to all integers nn. . • Floors and ceilings do not change the result.Floors and ceilings do not change the result.

– (Note: we proved this by domain transformation too).(Note: we proved this by domain transformation too).o Since Since nn//bbnn//bb, and , and nn//bb nn//bb, upper bound for floors and lower , upper bound for floors and lower

bound for ceiling is held.bound for ceiling is held.o So prove upper bound for ceilings (similar for lower bound for floors).So prove upper bound for ceilings (similar for lower bound for floors).


Upper bound of proof for Upper bound of proof for TT((nn)=)=aTaT((nn//bb)+)+ff((nn))

o consider sequence consider sequence nn, , nn//bb, , nn//bb//bb, , nn//bb //bb//bb, …, …

o Let us define Let us define nnjj as follows: as follows:

o nnj j = n = n if if jj=0=0

o = = nnjj-1-1//bb if if jj>0>0

o The sequence will be The sequence will be nn00, , nn11, …, , …, nnloglogbbnn

o Draw recursion tree:Draw recursion tree:



The proof of upper bound for ceilingThe proof of upper bound for ceiling

• TT((nn) = ) = ((nnloglogbbaa)+ )+ aajjff((nnjj))

• Thus similar to Lemma 4.3 and 4.4, the upper bound is proven. Thus similar to Lemma 4.3 and 4.4, the upper bound is proven.

j=0

logbn -1


Where Are the GapsWhere Are the Gaps

nnloglogbbaa ff((nn), ), case 2: within constant distancescase 2: within constant distancescc11

cc22

nn

ff((nn), case 1, at least polynomially smaller), case 1, at least polynomially smaller

Gap between case 1 and 2

nn Gap between case 3 and 2

ff((nn), case 3, at least polynomially larger), case 3, at least polynomially larger

Note: 1. for case 3, the Note: 1. for case 3, the regularity regularity also must hold.also must hold. 2. if 2. if ff((nn) is ) is lg lg nn smaller, then fall in gap in 1 and 2 smaller, then fall in gap in 1 and 2 3. if 3. if ff((nn) is ) is lg lg nn larger, then fall in gap in 3 and 2 larger, then fall in gap in 3 and 2 4. if 4. if ff((nn)=)=((nnloglogbbaalglgkknn), then ), then TT((nn)=)=((nnloglogbbaalglgkk+1+1nn). ). (as exercise)(as exercise)


o # of levels * cost of each level = O(# of levels * cost of each level = O(cncn log log3/23/2 nn)=O)=O(n(n lg lg nn))

o ComplicationComplication• If the tree is complete binary tree, # of leaves =If the tree is complete binary tree, # of leaves =

• But the tree is not completeBut the tree is not complete

• Go down from the root, more and more internal nodes are absentGo down from the root, more and more internal nodes are absent

o Verify OVerify O(n(n lg lg nn) is an upper bound by the substitution method ) is an upper bound by the substitution method

o Exmp1: T(n) = T(2n/3) + 1 (Case 2)Exmp1: T(n) = T(2n/3) + 1 (Case 2)• a=1, b=3/2, f(n)=1a=1, b=3/2, f(n)=1

• nnloglogbba a = n = nloglog3/23/21 1 =1, f(n)=1 ==1, f(n)=1 =(1)(1), , • T(n) = T(n) = (lg (lg nn))

A recursion tree for T(n)=T(n/3) + T(2n/3) + cnA recursion tree for T(n)=T(n/3) + T(2n/3) + cn

nx

n

n

x

x

)2/3(log

)2/3(

1)3

2(


Running time of the BS algorithm given belowRunning time of the BS algorithm given below

Algorithm recursive Binary-Search-Rec(A[1…n], k, l, r)Algorithm recursive Binary-Search-Rec(A[1…n], k, l, r) Input: a sorted array A of n comparable items, search key k, leftmost Input: a sorted array A of n comparable items, search key k, leftmost

and rightmost index positions in Aand rightmost index positions in A Output: Index of array’s element that is equal to k or -1 if k not foundOutput: Index of array’s element that is equal to k or -1 if k not found

functionfunction BSRBSR((XX : : namename; ; start, endstart, end: : integerinteger) ) ////l = start; r =end.l = start; r =end. beginbegin middlemiddle := := (start+end)/2(start+end)/2; ; ////while while l <= r l <= r do do

ifif names(middle)=xnames(middle)=x thenthen returnreturn numbers(middle)numbers(middle); ;

elseifelseif X&ltnames(middle)X&ltnames(middle) thenthen returnreturn BSRBSR(X,start,middle-1)(X,start,middle-1); ;

elseelse -- -- X&gtnames(middle)X&gtnames(middle) returnreturn BSRBSR(X,middle+1,end)(X,middle+1,end); ;

end ifend if; ; endend searchsearch; ;

You can write BS nonrecursive.. Simple.. Searcing up and down stream You can write BS nonrecursive.. Simple.. Searcing up and down stream of array.. of array..

T(n)=T(n/2)+1. T(n)=T(n/2)+1. worst case not ordered and ordered input cases O(n) and O(lg(n)), worst case not ordered and ordered input cases O(n) and O(lg(n)),

respectively. respectively. Best case. O(1) for both cases. Best case. O(1) for both cases. Recursive or iterative shouldn’t make any difference. Recursive or iterative shouldn’t make any difference.


algorithm stoogesort(array L, i = 0, j = length(L)-1)algorithm stoogesort(array L, i = 0, j = length(L)-1)

if L[j] < L[i] thenif L[j] < L[i] then

L[i] ↔ L[j]L[i] ↔ L[j]

if j - i > 1 thenif j - i > 1 then

t = (j - i + 1)/3t = (j - i + 1)/3

stoogesort(L, i , j-t)stoogesort(L, i , j-t)

stoogesort(L, i+t, j )stoogesort(L, i+t, j )

stoogesort(L, i , j-t)stoogesort(L, i , j-t)

return Lreturn L

Provided extra file.. Provided extra file..



Recurrence RelationsLiber Abaci, Fibonacci, the Florentian scholar from the Renaissance studied Liber Abaci, Fibonacci, the Florentian scholar from the Renaissance studied

simplified population growth functions. simplified population growth functions.

At some stage starts with a population of 1 pair of rabbits. After a month fertile At some stage starts with a population of 1 pair of rabbits. After a month fertile rabbits produce a newborn pair. Happy they are, they never die and reproduce rabbits produce a newborn pair. Happy they are, they never die and reproduce each month. each month.

f(n)=f(n-1)+f(n-2), start from f(0)=f(1)=1..f(n)=f(n-1)+f(n-2), start from f(0)=f(1)=1..

f(n) ≥ 2 f(n−2), faster than exponential growth of 2∗f(n) ≥ 2 f(n−2), faster than exponential growth of 2∗ kk..

The Fibonacci recurrence is an example of a linear homogeneous recurrence relation:The Fibonacci recurrence is an example of a linear homogeneous recurrence relation:

aann = C = C1 1 aan-1n-1 + C + C2 2 aan-2n-2 + . . . + C + . . . + Cr r aan-rn-r

A general solution of this, involves a linear combination of solutions, a(n) = A general solution of this, involves a linear combination of solutions, a(n) = nn and and there is no f(n). *** Substitute it.. there is no f(n). *** Substitute it..

nn = C = C1 1 n-1n-1 + C + C2 2 n-2n-2 + . . . + C + . . . + Cr r n-rn-r

Characteristic equation is Characteristic equation is rr = C = C1 1 r-1r-1 + C + C2 2 r-2r-2 + . . . + C + . . . + Cr r

Assume no complex roots, solutions Assume no complex roots, solutions 11, , 22, , 33, …, , …, rr. then any linear combination is . then any linear combination is also a general solution. also a general solution.

nn = A = A1 1 11nn + A + A2 2 22

nn + . . . + A + . . . + Ar r rrnn


Fibonacci: Fibonacci:

ffnn = f = fn-1n-1 + f + fn-2,n-2,, f, f00 = f = f11 =1. =1.

*** Substitute *** Substitute kk .. ..

nn = = n-1n-1 + + n-2n-2 22 - - - 1 = 0; - 1 = 0;

Roots of this characteristic equation are Roots of this characteristic equation are

1,2 1,2 = (1 + - sqrt(5))/2= (1 + - sqrt(5))/2

AAnn = = b b ((1+sqrt(5))/2)((1+sqrt(5))/2)nn + + dd ((1 - sqrt(5))/2)((1 - sqrt(5))/2)nn. .

Initial values, 1 and 1. Initial values, 1 and 1.

b b rr11nn + + dd rr22

nn = b + d = 1, n=0 = b + d = 1, n=0

AAnn = = b b ((1+sqrt(5))/2)((1+sqrt(5))/2)nn + + dd ((1 - sqrt(5))/2)((1 - sqrt(5))/2)nn = 1; n=1 = 1; n=1

b b = (1/sqrt(5))((1+sqrt(5))/2) = (1/sqrt(5))((1+sqrt(5))/2) d d = (-1/sqrt(5))((1-sqrt(5))/2) = (-1/sqrt(5))((1-sqrt(5))/2) Substitute Substitute bb, , dd back to the equation to get specific equation.. back to the equation to get specific equation..

ffnn = (1/sqrt(5)) = (1/sqrt(5)) [((1+sqrt(5))/2)[((1+sqrt(5))/2)n+1n+1 - - ((1 - sqrt(5))/2)((1 - sqrt(5))/2)n+1n+1],],


HomogenousHomogenous

First Order First Order Linear Homogeneous Recurrence RelationsLinear Homogeneous Recurrence Relations

aann = C = C1 1 aan-1n-1 rewriting with the roots rewriting with the roots

AAnn = C = C1 1 n-1n-1 and and = C = C1, 1, AAnn = = 00 (C (C11))nn

Example: Compound interest Example: Compound interest

AAnn = A = An-1n-1 + .3 + .3 AAn-1n-1 A Ann = = AA00 (1.3) (1.3)nn


AAnn = 7A = 7An-1n-1; A; A00 = 5, hence = 5, hence = 7, a = 7, ann = = 5(7)5(7)nn

Second Order Second Order Linear Homogeneous Recurrence RelationsLinear Homogeneous Recurrence Relations

aann = C = C1 1 aan-1n-1 + C + C2 2 aan-2n-2 , quadratic, solve two roots, as usual, r , quadratic, solve two roots, as usual, r11, r, r22. .

AAnn = = bb rr11nn + + dd rr22

nn. Needs initial values such as . Needs initial values such as bb and and dd..


aann = C = C1 1 aan-1n-1 rewriting with the roots rewriting with the roots

AAnn = C = C1 1 n-1n-1 and and = C = C1, 1, AAnn = = 00 (C (C11))nn


AAnn = A = An-1n-1 + .3 + .3 AAn-1n-1 A Ann = = AA00 (1.3) (1.3)nn


AAnn = 7A = 7An-1n-1; A; A00 = 5, hence = 5, hence = 7, a = 7, ann = = 5(7)5(7)nn

Second Order Second Order Linear Homogeneous Recurrence RelationsLinear Homogeneous Recurrence Relations

aann = C = C1 1 aan-1n-1 + C + C2 2 aan-2n-2 , quadratic, solve two roots, as usual, r , quadratic, solve two roots, as usual, r11, r, r22. .

AAnn = = bb rr11nn + + dd rr22

nn. Needs initial values such as . Needs initial values such as bb and and dd..


Inhomogeneous Recurrence. Inhomogeneous Recurrence.

aann = ca = can-1n-1 + f(n) + f(n)

Solution for homogeneous side in this case, aSolution for homogeneous side in this case, a00 c cnn

Suppose particular solution for inhomogeneous form is Suppose particular solution for inhomogeneous form is **

((nn** = c = cnn + + **nn + f(n)); + f(n));

aann = bc + = bc + **nn; is a general solution. ; is a general solution.

Roots of this characteristic equation are Roots of this characteristic equation are

1,2 1,2 = (1 + - sqrt(5))/2= (1 + - sqrt(5))/2

AAnn = = b b ((1+sqrt(5))/2)((1+sqrt(5))/2)nn + + dd ((1 - sqrt(5))/2)((1 - sqrt(5))/2)nn. .

Initial values, 1 and 1. Initial values, 1 and 1.

b b rr11nn + + dd rr22

nn = b + d = 1, n=0 = b + d = 1, n=0

AAnn = = b b ((1+sqrt(5))/2)((1+sqrt(5))/2)nn + + dd ((1 - sqrt(5))/2)((1 - sqrt(5))/2)nn = 1; n=1 = 1; n=1

b b = (1/sqrt(5))((1+sqrt(5))/2) = (1/sqrt(5))((1+sqrt(5))/2)

d d = (-1/sqrt(5))((1-sqrt(5))/2) = (-1/sqrt(5))((1-sqrt(5))/2)

Substitute Substitute bb, , dd back to the equation to get specific equation.. back to the equation to get specific equation..

ffnn = (1/sqrt(5)) = (1/sqrt(5)) [((1+sqrt(5))/2)[((1+sqrt(5))/2)n+1n+1 - - ((1 - sqrt(5))/2)((1 - sqrt(5))/2)n+1n+1] ]


Quiz Evaluate T(n) of functions given below. Quiz Evaluate T(n) of functions given below. 1)1) T(n) = T(n/3) + T(2n/3) +c n, Deterministic select algorithm using median. ST(n) = T(n/3) + T(2n/3) +c n, Deterministic select algorithm using median. Solve the recurrence for the

running time of the algorithm by drawing a recurrence tree. Answer: The height of the tree is at least log3 n, and is at most log3/2 n and the sum of the costs in each level is n. Hence T(n)=(nlgn)

2)2) T(n) = 2T(n/2) + O(1) T(n) = 2T(n/2) + O(1) Answer: Case 1: Answer: Case 1: (n), epsilon=1. (n), epsilon=1.

3)3) T(n) = 4T(n/2) + nT(n) = 4T(n/2) + n22 Answer: Case 2: T(n) = (nAnswer: Case 2: T(n) = (n22 log n) log n)

4)4) What is the upper complexity of T(n)=kWhat is the upper complexity of T(n)=knn + log + log88(n(n2n2n) + (10n)) + (10n)k.k. Answer: if c<=1, O(nlogAnswer: if c<=1, O(nlog88(n)), otherwise O(k(n)), otherwise O(knn).).

5)5) T(n) = 3T(2ceil(n/3)) + 1000 stooge sort.. T(n) = 3T(2ceil(n/3)) + 1000 stooge sort.. lglg3/23/2(3) = lg3/lg(3/2)=lg3/(lg3-1) = ~2.72(3) = lg3/lg(3/2)=lg3/(lg3-1) = ~2.72

lglg3/23/2(2*1.5)=lg(2*1.5)=lg3/23/2(2)+lg(2)+lg3/23/2(1.5)=lg(1.5)=lg3/23/2(2)+1=~l.7+1..(2)+1=~l.7+1..Case 1. what is the meaning of this.. Case 1. what is the meaning of this..

5)5) M(M(nn) = M() = M(nn-1) + 1, n>0, M(0) = 0, substation method, sums to -1) + 1, n>0, M(0) = 0, substation method, sums to (n) (n) 6)6) A(2A(2kk) = A(2) = A(2k-1k-1)+1, case 2, yields )+1, case 2, yields (lgn)(lgn)6)6) Recursion in d-dimensional mesh (recursion on d-dimensional mesh, d>0)Recursion in d-dimensional mesh (recursion on d-dimensional mesh, d>0)

T(n) = T(n/(2T(n) = T(n/(2dd)) + 47d)) + 47d22nn1/d1/d

h.s =1, T(n)=theta(47dh.s =1, T(n)=theta(47d22nn1/d1/d), T(n)=theta(47d), T(n)=theta(47d22nn1/d1/d), , case 3. No need to check regularity. ), , case 3. No need to check regularity. 7)7) T(n) = T(n/2) + log n (PRAM mergesort)T(n) = T(n/2) + log n (PRAM mergesort)

h.s =1, for sufficiently large n, (log n)>1, appealing to T(n)=log n, regularity check af(n/b)h.s =1, for sufficiently large n, (log n)>1, appealing to T(n)=log n, regularity check af(n/b)cf(n), cf(n), log(n/2)log(n/2)clog(n), (1-c)lgnclog(n), (1-c)lgn1, not possible always to satisfy this relation for large n,, and 0<c<1, (for the 1, not possible always to satisfy this relation for large n,, and 0<c<1, (for the regularity condition 0<cregularity condition 0<c<1)<1). Therefore one needs to use other methods, simply check the equation given . Therefore one needs to use other methods, simply check the equation given below (the final equation obtained at the bottom of iterative tree method), there the first part is 1, and the below (the final equation obtained at the bottom of iterative tree method), there the first part is 1, and the second part is logsecond part is log22n. Therefore the result is theta(logn. Therefore the result is theta(log22n). n).

4)4) TT((nn) = ) = ((nnloglogbbaa)+ )+ ΣΣi=0i=0

loglogbbn-1n-1

aajjff((nn//bbjj) = ) = (1)+ (1)+ ΣΣi=0i=0

lglgn-1 n-1 log(log(nn//bbjj)= )=

5)5) =logn =logn ΣΣi=0i=0

lglgn-1 n-1 log(log(nn/2/2jj), lgn-j, lgn-1, lgn-2, lgn-3, .., lgn-lgn+1 = lgnlgn-), lgn-j, lgn-1, lgn-2, lgn-3, .., lgn-lgn+1 = lgnlgn-ΣΣi=0i=0

lglgn-1 n-1 j, j,

(n(n+1)/2), (lg(n)-1)(lgn)/2(n(n+1)/2), (lg(n)-1)(lgn)/2 ((loglog22nn))


wikipediawikipedia

Inadmissible equationsInadmissible equations The following equations cannot be solved using the master theorem:The following equations cannot be solved using the master theorem: [2][2]

*** *** aa is not a constant, n^lg2^n=n^n, the number of children is not a constant, n^lg2^n=n^n, the number of children increasing with exp trendincreasing with exp trend

non-polynomial difference between f(n) and n^logba=n non-polynomial difference between f(n) and n^logba=n

aa<1 cannot have less than one sub problem<1 cannot have less than one sub problem

f(n) is not positivef(n) is not positive

case 3 but regularity violation.case 3 but regularity violation.

A little bit of recurrences here.. From CLRS. Dr. M. Sakalli, Marmara University

Documents