Lecture3 - Analysis and Recurrence Relationships

8/14/2019 Lecture3 - Analysis and Recurrence Relationships

1/58

COMP2230 Introduction toAlgorithmics

Lecture 3

A/Prof Ljiljana Brankovic


2/58

Lecture Overview

Analysis of Algorithms - Text, Section 2.3


3/58

Analysis of Algorithms

Analysis of algorithms refers to estimating the timeand space required to execute the algorithm.

We shall be mostly concerned with time complexity.

Problems for which there exist polynomial timealgorithms are considered to have an efficient

solution.


4/58


Problems can be: Unsolvable

These problems are so hard that there does not exist

an algorithm that can solve them. Example is thefamous Halting problem.

IntractableThese problems can be solved, but for some instances

the time required is exponential and thus theseproblems are not always solvable in practice, evenfor small sizes of input.


5/58


Problems can be (continued): Problems of unknown complexity

These are, for example, NP-complete problems; for

these problems neither there is a known polynomialtime algorithm, nor they have been shown to beintractable.

Feasible (tractable) problemsFor these problems there is a known polynomial time

algorithm.


6/58

Analysis of AlgorithmsWe shall not try to calculate the exact time needed to

execute an algorithm; this would be very difficult to do,and it would depend on the implementation, platform, etc.

We shall only be estimatingthe time needed for algorithmexecution, as a function of the size of the input.

We shall estimate the running time by counting somedominant instructions.

A barometerinstruction is an instruction that is executedat least as often as any other instruction in thealgorithm.


7/58


Worst-casetime is the maximumtime needed toexecute the algorithm, taken over all inputs of size n.

Average-casetime is the averagetime needed toexecute the algorithm, taken over all inputs of size n.

Best-casetime is the minimumtime needed to executethe algorithm, taken over all inputs of size n.


8/58

Analysis of AlgorithmsIt is usually harder to analyse the average-casebehaviour of the algorithm than the best and the

worst case.

Behaviour of the algorithm that is the most importantdepends on the application. For example, foralgorithm that is controlling a nuclear reactor, theworst case analysis is clearly very important. On the

other hand, for an algorithm that is used in a non-critical application and runs on a variety ofdifferent inputs, the average case may be moreappropriate.


9/58

Example 2.3.1 Finding the Maximum

Value in an Array Using a While LoopThis algorithm finds the largest number in the array s[1], s[2], ... , s[n].

Input Parameter: s

Output Parameters: Nonearray_max_ver1(s){

large= s[1]i= 2while (i s.last){

if (s[i] > large)large= s[i] // larger value found

i= i+ 1}

return large}


10/58

The input is an array of size n.

What can be used as a barometer?

The number of iterations of while loop appears to bea reasonable estimate of execution time - The loop isalways executed n-1 times.

Well use the comparison i s.last as a barometer.

The worst-case and the average-case (as well as thebest case!) times are the same and equal to n.


11/58

Suppose that the worst-case time of an algorithmis (for input of size n)

t(n) = 60n2+ 5n + 1

For large n, t(n) grows as 60n2:

n T(n)=60n2+ 5n + 1 60n2

10 6,051 6,000

100 600,501 600,0001000 60,005,001 60,000,000

10,000 6,000,050,001 6,000,000,000


12/58

The constant 60 is actually not important as itdoes not affect the growth of t(n) with n. Wesay that t(n) is of order n2, and we write it as

t(n) = (n2)

(t(n) is theta of n2)


13/58

Let f and g be nonnegative functions on the positive integers.

Asymptotic upper bound: f(n) =O(g(n)) if there exist constants C1> 0 and N1such that f(n) C1g(n) for all n N1.

We read f(n) =O(g(n)) as f(n) is big oh of g(n)) or

f(n) is of order at most g(n).

nN1

f(n)

C1g(n)


14/58

Asymptotic lower bound: f(n) =(g(n)) if there exist constants C2> 0 and N2such that f(n) C2g(n) for all n N2.

We read f(n) =(g(n)) as f(n) is of order at least g(n) or f(n) isomega of g(n)).

nN2

f(n)

C2g(n)


15/58

Asymptotic tight bound: f(n) =(g(n)) if there exist constantsC1,C2> 0 and N such that C2 g(n) f(n) C1g(n) for all n N.

Equivalently, f(n) =(g(n)) if f(n) =O(g(n)) and f(n) =(g(n)) ).

We read f(n) =(g(n)) as f(n) is of order g(n) or f(n) is theta ofg(n))

Note that n=O(2n) but n (2n).

nN

f(n)

C1g(n)

C2g(n)


16/58

Example

Let us now formally show that if t(n) = 60n2+ 5n + 1 thent(n) = (n2).

Since t(n) = 60n2+ 5n + 1 60n2+ 5n2+ n2 = 66n2 for all n 1

it follows that t(n) = 60n2+ 5n + 1 = O(n2).

Since t(n) = 60n2+ 5n + 1 60n2for all n 1it follows that t(n) = 60n2+ 5n + 1 = (n2).

Since t(n) = 60n2

+ 5n + 1 = O(n2

) and t(n) = 60n2

+ 5n + 1 =

(n2

)it follows that t(n) = 60n2+ 5n + 1 = (n2).


17/58


18/58

Common Asymptotic GrowthFunctions

Theta form Name(1) Constant

(log log n) Log log

(log n) Log

(nc), 0 < c < 1 Sublinear

(n) Linear

(n log n) n log n

(n2) Quadratic

(n3) Cubic

(nk), k 1 Polynomial

(cn), c > 1 Exponential

(n!) Factorial


19/58

The running time of different algorithms on a processor performing one million high levelinstructions per second (from Algorithm Design, by J. Kleinberg and E Tardos).

n n nlogn n2

n3

1.5n

2n

n!10 < 1 sec < 1 sec < 1 sec < 1 sec < 1 sec < 1 sec 4 sec

30 < 1 sec < 1 sec < 1 sec < 1 sec < 1 sec 18 min 1025y.

50 < 1 sec < 1 sec < 1 sec < 1 sec 11 min 36 years >1025y.

100 < 1 sec < 1 sec < 1 sec 1 sec 12,892 y. 1017y. >1025y.

1,000 < 1 sec < 1 sec 1 sec 18 min >1025y. >1025y. >1025y.

10,000 < 1 sec < 1 sec 2 min 12 days >1025y. >1025y. >1025y.

100,000 < 1 sec 2 sec 3 hours 32 years >1025y. >1025y. >1025y.

1,000,000 1 sec 20 sec 12 days 31,710 y. >1025

y. >1025

y. >1025

y.


20/58

Properties of Big Oh

If f(x) = O(g(x)) and h(x) = O(g(x))

then f(x)+h(x) = O( max { g(x),g(x) } )and f(x) h(x) = O( g(x) g(x) )

If f(x) = O(g(x)) and g(x) = O(h(x))

then f(x) = O( h(x) )


21/58

Examples

1. 5 n21000 n + 8 = O(n2) ?

2. 5 n21000 n + 8 = O(n3) ?

3. 5 n2+ 1000 n + 8 = O(n) ?

4. nn= O(2n) ?

5. If f(n) = (g(n)), then 2f(n) =(2g(n)) ?

6. If f(n) = O(g(n)), then g(n) = (f(n)) ?


22/58

Examples

1. 5 n21000 n + 8 = O(n2) ? True

2. 5 n21000 n + 8 = O(n3) ? True

3. 5 n2+ 1000 n + 8 = O(n) ?False n2grows faster than n

4. nn= O(2n) ?False nngrows faster than 2n

5. If f(n) = (g(n)), then 2f(n) =(2g(n)) ?

Falsehere constants matter, as they become exponents !!!E.g., f(n) = 3n, g(n) = n, 23n(2n)

6. If f(n) = O(g(n)), then g(n) = (f(n)) ? True if f(n) Cg(n) for n N,then g(n) f(n)/C for n N.


23/58

Examples

7. If f(n) = (g(n)), and g(n) = (h(n)), then f(n) + g(n)=(h(n)) ?

8. If f(n) = O(g(n)), then g(n) = O(f(n)) ?

9. If f(n) =

(g(n)), then g(n) =

(f(n)) ?

10. f(n) + g(n) = (h(n)) where h(n) = max {f(n), g(n)} ?

11. f(n) + g(n) = (h(n)) where h(n) = min {f(n), g(n)} ?


24/58

Examples

7. If f(n) = (g(n)), and g(n) = (h(n)), then f(n) + g(n)=(h(n)) ?

True

8. If f(n) = O(g(n)), then g(n) = O(f(n)) ? False e.g., f(n) = n, g(n)= n2

9. If f(n) = (g(n)), then g(n) = (f(n)) ? True

10. f(n) + g(n) = (h(n)) where h(n) = max {f(n), g(n)} ? True

11. f(n) + g(n) =

(h(n)) where h(n) = min {f(n), g(n)} ?False f(n) = n, g(n) = n2, h(n) n


25/58

Examples find theta notationfor the following

12. 6n+1

13. 2 lg n + 4n + 3n lg n

14. (n2+ lg n)(n + 1) / (n + n2)

15. for i=1to 2n

for j=1to n

x=x+1

16. i=n

while (i1) {

x=x+1

i=i/2}

17. j=n

while (j1) {

for i=1 to jx=x+1

j=j/2

}


26/58

Examples find theta notationfor the following

12. 6n+1 = (n)

13. 2 lg n + 4n + 3n lg n = (n lg n)

14. (n2+ lg n)(n + 1) / (n + n2) = (n)

15. for i=1to 2n

for j=1to n

x=x+1= (n2)

16. i=n

while (i1) {

x=x+1

i=i/2}

= (lg n)

17. j=n

while (j

1) {for i=1 to j

x=x+1

j=j/2

}

= (n)


27/58

Example: Towers of HanoiAfter creating the world, God set on Earth 3 diamond rods and 64

golden rings, all of different size. All the rings were initially onthe first rod, in order of size, the smallest at the top. God alsocreated a monastery nearby where monks task in life is totransfer all the rings onto the second rod; they are only allowedto move a single ring from one rod to another at the time, and a

ring can never be placed on top of another smaller ring.According to the legend, when monks have finished their task,the world will come to an end.

If monks move one ring per second and never stop, it will take

them more that 500,000 million years to finish the job (morethan 25 times the estimated age of the universe!).


28/58

Example 2

When is f(2n) = (f(n))?

f(n) = n2

Big O:(2n)24 n2, for all n > 0Big :(2n)2n2,for all n > 0

f(n) = 2n

Big O:22nC12n, for all n > n0 ???2n 2n C12n ???NO!!! There is no constant C1such that 2nC1forall n > n0.


29/58

Example: Towers of HanoiThe following is a way to move 3 rings from rod 1 to rod

2.


30/58

Example: Towers of Hanoi

The following is an algorithm that moves m rings fromrod i to rod j.

Hanoi (m,i,j){\\Moves the msmallest rings from rod ito rod j

if m > 0then {Hanoi(m-1,i,6-i-j)

write i j

Hanoi(m-1,6-i-j,j)}

}


31/58

Example: Towers of Hanoi

Recurrence relation:

0 ifm=0

t(m) =

2t(m-1) + 1 otherwise

Or, equivalently,t0= 0tn= 2tn-1+ 1, n > 0


32/58

Example 1

Show that for any real aand b, b > 0(n+a)b= (nb)

Big O:Show that there are constants C1 and n0suchthat (n+a)bC1nb, for all n > n0

(n+a)b(2n)b= 2bnbfor all n > a

Big

:Show that there are constants C2 and n0suchthat (n+a)bC2nb, for all n > n0

(n+a)b(n/2)b= (1/2)bnbfor all n > 2|a|


33/58

Solving Recurrence RelationsIteration (substitution): Example from first weeks

lecture

C(n) = n + C(n/2), n >1C(1) = 0

Solution for n=2k, for some k.

C(2k) = 2k+ C(2k-1)= 2k+ 2k-1+ C(2k-2)= 2k+ 2k-1+ + 21+ C(20)= 2k+ 2k-1+ + 21+ C(1)= 2k+ 2k-1+ + 21+ 0 (as C(1) = 0)

= 2k+12 = 2n-2 = (n).


34/58

Solving Recurrence RelationsSolving the same recurrence for any n:

C(n) = n + C(n/2), n >1C(1) = 0

Solution for 2k-1 n < 2k, for somek:

Since C(n)is an increasing function (prove it!):C (2k-1) C(n) < C (2k)

C(2k) = 2k+1-2, C(2k-1) = 2k-2

C(n) < C(2k)= 2k+1-2 < 4n,thus C(n) = O(n)

C(n) C(2k-1) = 2k2 > n-2 n/2, thusC(n) = (n)

Thus C(n)= (n).


35/58

Main Recurrence Theorem(also known as Master Theorem)

Let a, band kbe integers satisfying a 1, b 2and k 0. In the following, n/bdenotes either n/bor n/b.

In the case of the floor function the initial conditionT(0) =uis given; and in the case of the ceilingfunction, the initial condition T(1) = uis given.


36/58

Recurrence Theorem(also known as Master Theorem)

Upper Bound

If T(n) aT(n/b) + f(n)and f(n) =O(n k)then

O(n k) if a < b k

T(n) = O(n k log n) if a = b k

O(n logba) if a > b k

The same theorem holds for and notationsas well.


37/58


Lower Bound

If T(n) aT(n/b) + f(n)and f(n) =(n k)then

(n k) if a < b k

T(n) = (n k log n) if a = b k

(n logba) if a > b k


38/58


Tight Bound

If T(n) = aT(n/b) + f(n)and f(n) = (n k)then

(n k) if a < b k

T(n) = (n k log n) if a = b k



39/58

Example 1

Solvecn= n + cn/2, n > 1; c1= 0

We have a = 1, b = 2, f(n) = nand k=1.

Since a < bk

, cn=

(nk

) and since k = 1 , cn=

(n).


(nk

) if a < bk

T(n) = (n k log n) ifa = b k(n logba) if a > b k


40/58

Example 2

Solve cn= n + 2cn/2, n > 1; c1= 0


Since a = b k, cn= (nklog n)

and since k = 1 , cn= (n log n).


(n k) if a < b kT(n) = (n k log n) ifa = b k



41/58

Example 3Solve cn= n + 7cn/4


Since a > b k, cn= (nlog47)


(n k) if a < b k



42/58

Example 4Solve cnn 2+ cn/2 + cn/2

If cnis nondecreasing then cn/2cn/2and we have cnn 2+ 2cn/2

We have a = 2, b = 2, f(n) = n2and k=2.

Since a < b k, cn= O(nk) and since k=2, cn= O(n2)


(n k) if a < b k



43/58

Examples

Solve the following recurrences:

1. T(n) = T(n-1) + nr

2. T(n) = 2 T(n-1) + nr

(Note: The solution will depend on the valueof the parameter r andthe initial conditions, which both need to be specified)


44/58

Example

0 if n = 0

T(n)=

2 + 4T(n/2) otherwise

Restrict nto be a power of 2: n = 2kLook at the expansion

n T(n)

20 2

21 2 + 4 222 2 + 4 2 + 42 2


45/58

ExamplePattern seems to suggest:

T(n) = T(2k) = i=0k2 4i = 2 (4k+1- 1)/(4-1) = (8n2- 2)/3 = O(n2)

Proof by mathematical induction:

Base case: k = 0, n=20T(1) = (8 - 2)/3 = 2basis holds.

Assumption: True for k.

Proof that it is true for k+1:T(2k+1) = 2 + 4T(2k) = 2 + 4 (822k2)/3 = (6 + 8 22k+2-8)/3

= (8 22k+2- 2)/3


46/58

We assumed n is a power of 2 This gives a conditional bound

T(n)is in O(n2| n is a power of 2)

Can we remove this condition?

We can use smoothness(FoA p.89) basically show that well-behavedfunctions which have

bounds at certain points, have the same bounds at all

other points


47/58

Smoothness

A function fis smooth if: fis non-decreasing over the range [N, +)

eventually non-decreasing

For every integer b 2

f(bn)is O(f(n))

Let f(n)be smooth

Let t(n)be eventually non-decreasing

t(n)is (f(n))for na power of bimpliest(n)in (f(n))


48/58

SmoothnessFormally, a function fdefined on positive integers is

smooth if for any positive integer b 2, there arepositive constants Cand N, depending on b, suchthat

f(bn) Cf(n)

and

f(n) f(n+1)

for all n N.


49/58

Smoothness

Lemma 1:

If t is a nondecreasing function, fis a smooth

function, and t(n) = (f(n))for na power of b,then t(n) = (f(n)).


50/58

Smoothness

Which functions are smooth? log and polynomial functions are smooth

eg. log(n), log2(n), n3- n, n log(n)

superpolynomial functions are not

eg. 2n, n!

Using this, we can use guess-and-check a bit

more loosely: prove for easier cases, and usesmoothnessfor the rest

P f f th M i R


51/58

Proof of the Main RecurrenceTheorem

Let a, band kbe integers satisfying a 1, b 2and k 0. In the following, n/bdenotes either n/bor n/b.

In the case of the floor function the initial conditionT(0) =uis given; and in the case of the ceilingfunction, the initial condition T(1) = uis given.

We first need to show for both floor and ceiling T(n) iswell defined for all n. We use mathematical inductionto prove that (to be done in tutorials).

P f f th M i R


52/58


Next we need to show that

We use mathematical induction to show this.

Base Case: m=0

x0y0= (x1y1)/(x-y), that is, 1 = 1

Inductive Assumption: m=k

xk-iyi= (xk+1yk+1)/(x-y)

yxyx

yxyx

mm

im

i

im

,11

0

i=0

k

P f f th M i R


53/58


Inductive Step: m=k+1

xk+1-iyi= (xk+2yk+2)/(x-y)

xk+1-iyi= xk+1-iyi +x0yk+1 = xxk-iyi +yk+1 =

=x(xk+1yk+1)/(x-y) +yk+1 = (xk+2xyk+1+xyk+1yk+2)/(x-y) =

= (xk+2yk+2)/(x-y)

i=0

k+1

i=0

k+1

i=0

k

i=0

k

P f f th M i R


54/58


We are now ready to formulate a lemma that we shall latter use toprove the Main Recurrence Theorem.

Lemma 2:

If n>1 is a power of b and a

bk

the solution of recurrence relationT(n) = aT(n/b) + cnk isT(n) = C1nlogba+ C2nk

for some constants C1and C2, where C2>0 for a1 is a power of b and a = bkthe solution of recurrence relation

T(n) = aT(n/b) + cnk isT(n) = C3nk+ C4nklogbn

for some constants C3and C4>0.

P f f th M in R n


55/58


Proof:Suppose that n = bm.

Then m = logbn, nk= (bm)k=(bk)mand am= (blogba)m= nlogba

Then we have:

T(n) = aT(n/b) + cnk

T(n) = T(bm) = aT(bm-1) +c(bk)m

= a[aT(bm-2) +c(bk)m-1] + c(bk)m

= a2T(bm-2) +c[a(bk)m-1+(bk)m]

= a2[aT(bm-3) +c(bk)m-2] +c[a(bk)m-1+(bk)m]

= a3

T(bm-3

) +c[a2

(bk

)m-2

+a(bk

)m-1

+(bk

)m

]. . .

=amT(b0)+c am-i(bk)i

i=1

m

Proof of the Main Recurrence Theorem


56/58

Proof of the Main Recurrence TheoremProof (contd):

Thus T(n) = am

T(b0

)+c am-i

(bk

)i

If a bk, using we get

T(n) = amT(1) + c[((bk)m+1-am+1)/(bk-a)-am]= amT(1) + c(bk)m+1/(bk-a) + c[(-am+1)/(bk-a)-am]= amT(1) + c(bk)m+1/(bk-a) amcbk/(bk-a)= [T(1)- cbk/(bk-a)] am + [cbk/(bk-a)] (bk)m

= [T(1)- cbk/(bk-a)] nlogb a + [cbk/(bk-a)] (bk)m

= C1 nlogb a + C2(bk)m

where C1 =T(1)- cbk/(bk-a) and C2= cbk/(bk-a).

Note that if a < bkthen C2> 0 and if if a > bkthen C2< 0.

i=1

m

yxyx

yxyx

mm

im

i

im

,11

0



57/58

Proof of the Main Recurrence TheoremProof (contd):

Recall T(n) = am

T(b0

)+c am-i

(bk

)i

If a = bk, and we get

T(n) = amT(1) + c am

= amT(1) + cmam

= C3nk+ C4nklogbn

where C3 =T(1) and C4= c>0.Note that if a < bkthen C2> 0 and if if a > bkthen C2< 0.

i=1

m

i=1

m



58/58


Lemma 2 proves the main Recurrence Theorem for casewhere n is a power of b. We then apply Lemma 1(smooth lemma) to generalize the proof for any n(see text page 63 and 64 for details).

Lecture3 - Analysis and Recurrence Relationships

Documents