Algorithm Design and Analysis (ADA)

242-535 ADA: 2. Running Time

• Objectiveo to introduce the Big-Oh notation for estimating

the worst case running time of programs

242-535, Semester 1 2017-2018

2. Running Time of

Programs

1. Running Time: T()2. Big-Oh and Approximate Running Time3. Calculating Big-Oh Directly4. Analyzing Function Calls5. Analyzing Recursive Functions6. Towers of Hanoi7. Other Asymptotic Notations

Overview

• What is the running time of this program?

void main() int i, n; scanf("%d", &n); for(i=0; i<n; i++) printf("%d"\n", i);

1. Running Time: T()

continued

• Assume 1 instruction takes 1 ms

Counting Instructions

n value no. of loops Time (ms)1 1 31,000 1,000 3,0001,000,000 1,000,000 3,000,000

• There is no single answer!o the running time depends on the size of the n value

• Instead of a time answer in seconds, we want a time answer which is related to the size of the input.

continued

• For example:o running time T(n) = constant * no this means that as n gets bigger, so does the

program timeo running time is linearly related to the input size

size of n

runningtime

constant * n

• A simple way of writing the running time is:T(n) = 3n

n value no. of loops Time (ms)1 1 31,000 1,000 3,0001,000,000 1,000,000 3,000,000

• A program/algorithm has a running time T(n)o n is some measure of the input size

• T(n) is the largest amount of time the program takes on any input of size n

• T(n) is the worst running timeo not always accurate, but easy to calculate

• Time units are left unspecified.

Running Time Theory

Worst-case: (we use this usually)• T(n) = maximum time of algorithm on any input of size

n.- one possible value

Average-case: (we sometimes use this)• T(n) = expected time of algorithm over all inputs of size

n.- this approach needs info about the statistical distribution (probability) of the inputs.- e.g. uniform spread of data (i.e. all data is equally likely)

Best-case: (don't use this, it's misleading)• e.g. write a slow algorithm that works fast on specially

selected input.

1.1. Kinds of Running Time

• Loop fragment for finding the product of all the positive numbers in the A[] array of size n:

(2) int prod = 1;(3) for(j = 0; j < n; j++)(4) if (A[j] > 0)(5) prod = prod * A[j];

• Count each assignment and test as 1 “time unit”.

1.2. T(n) Example

• The while-loop is easier to count (and equivalent to the for-loop):

int prod = 1; // 1int j = 0; // 1while (j < n) // 1 for the test if (A[j] > 0) // 1 prod = prod*A[j]; // 1 + 1 j++; // 1

Convert 'for' to 'while'

We assume that 1 instructiontakes 1 "time unit"

What aboutcounting theloop?

• The for loop executes n timeso each loop carries out (in the worse case) 5 ops

• test of j < n, if test, multiply, assign, j incremento total loop time = 5no plus 3 ops at start and end

• small assign (line 2), init of j (line 3), final j < n test

• Total time T(n) = 5n + 3o running time is linear with the size of the array

Calculation

• If input size < 50, program B takes less time.• But for large n’s, which are more common in

real code, program B is worse and worse (slower).

1.3. Comparing Different T()’s

input size n

TA(n) = 100n

TB(n) = 2n2

• Formula (n = input size) Namen linearn2 quadraticn3 cubicnm polynomial, e.g. n10

mn ( m >= 2) exponential, e.g. 5n

n! factorial1 constantlog n logarithmicn log nlog log n

1.4. Common Growth Formulae &

Growth Examples

1 3 5 7 9 11 13 15 17 19

f(n) = n

f(n) = log(n)

f(n) = n log(n)f(n) = n 2

f(n) = n 3

f(n) = 2 n

n log nnlog n

• 3 9 50 100 1000 106

n 3 9 50 100 1ms 1secn2 9 81 2.5ms 10ms 1sec 12 daysn3 27 729 125ms 1sec 16.7 min 31,710yr2n 8 512 36yr 4*1016 yr 3*10287 yr3*10301016yrlog n 2 3 6 7 10 20

1.5. Execution Times

n (no. of instructions)

if n is 50, you willwait 36 years for an answer!

Assume 1 instruction takes 1 microsec (10-6 secs) to execute.How long will n instructions take?

• Logarithmic running times are best.

• Polynomial running times are acceptable, if the power isn’t too bigo e.g. n2 is ok, n100 is terrible

• Exponential times mean sloooooooow code.o some size problems may take longer to finish than

the lifetime of the universe!

• T() can guide our choice of which algorithm to implement, or program to useo e.g. selection sort or merge sort?

• T() helps us look for better algorithms in our own code, without expensive implementation, testing, and measurement.

1.6. Why use T(n)?

• Algorithms often perform much better on average than the worst case used in T()o quicksort is n log n on a “random” array, but n2 in the

worse case

o but for most algorithms, the worst case is a good predictor of its running time

o average case analyses can be done, but they are harder mathematically

(Wrong) Arguments against T(n)

continued

• Some people say:o “Who cares about running time? In a few years,

machines will be so fast that even bad algorithms will be fast.”

o History shows this argument to be wrong. As machines get faster, problem sizes get bigger.

o Most interesting problems (e.g. computer vision, natural language processing) always require more resources

• fast algorithms will always be needed

continued

• Some people say:o "Benchmarking (running programs on a standard set

of test cases) is easier."

o This is sometimes true, but the benchmarks only give numbers for that particular machine, OS, compiler, computer language.

• We want T() to be the same for an algorithm independent of the machine where it is running.

• This is not true since different machines (and OSes) execute instructions at different speeds.

• Consider the loop example (slide 11)o on machine A, every instruction takes 1 "time unit"o the result is TA(n) = 5n + 3

T() is too Machine Dependent

• On machine B, every instruction takes 1 "time unit" except for multiplication, which takes 5 "time units".

• The for loop executes n timeso each loop carries out (in the worse case) 5 ops

• test of j < n, if test, multiply, assign, j incremento total loop time = 9no plus 3 ops at start and end

• small assign (line 2), init of j (line 3), final j < n test

• Total time TB(n) = 9n + 3o running time is linear with the size of the array

• TA() = 5n + 3 and TB() = 9n +3

• These are both linear equations (which is good), but the constants are different (which is bad)

• We want a T() notation that is independent of machines.

• Big-Oh notation for T(n) ignores constant factors which depend on compiler/machine behaviouro that's good

• Big-Oh simplifies the process of estimating the running time of programso we don't need to count every code lineo that's also good

2. Big-Oh and Running Time

• The Big-Oh value specifies running time independent of:o machine architecture

• e.g. don’t consider the running speed of individual machine operations

o machine load (usage)• e.g. time delays due to other users

o compiler design effects • e.g. gcc versus Visual C

• When we counted instructions for the loop example, we found:o TA() = 5n + 3 o TB() = 9n + 3

• The Big-Oh equation, O(), is based on the T(n) equation but ignores constants (which vary from machine to machine). This means for both machine A and B:

T(n) is O(n)we say "T(n) is order n"

Example

• T(n) value Big Oh : O()o10n2+ 50n+100 O(n2)o (n+1)2 O(n2)o n10 O(2n)o5n3 + 1 O(n3)

• These simplifications have a mathematical reason, which is explained in section 2.2.

More Examples

hard tounderstand

• O() ignores constant factors, which means it is a more general measure of running time for algorithms across different platforms/compilers.

• It can be compared with Big-Oh values for other algorithms.o i.e. linear is better than polynomial and exponential, but

worse than logarithmico i.e. O(log n) < O(n) < O(n2) < O(2n)

2.1. Is Big-Oh Useful?

• The connection between T() and O() is:o the T() = f(n) can be written as T(n) is O( g(n) )o this means that g(n) is the most important thing

in T()'s f(n) function when n is large• Example 1:

o T(n) = 5n + 3 // the f() function is 5n + 3o write as T(n) is O(n) // the g() function is n

• Example 2:o T(n) = 9n + 3 // the f() function is 9n + 3o write as T(n) is O(n) // the g() function is n

2.2. Definition of Big-Oh

continued

• n0 and c are called witnesses to the relationship:T(n) = f(n) and T(n) is O(g(n) )

• In some textbooks this is written as:o f(n) is O(g(n)) // leave out the

More Formally

We write T(n) = f(n) as T(n) is O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 f(n) c*g(n) for all n n0.

• g(n) is a simple function of n without constants and additional smaller terms

• e.g. n, n2, 2n, n log n

• Not 5n2, 2n + 4, n3 + n2

• Use n2, 2n , n3

What is g(n)?

• O-notation gives an upper bound for a function to within a constant factor. We write T(n)=f(n) as T(n) is O(g(n)) if there are positive constants n0 and c such that at and to the right of n0, the value of f(n) always lies on or below c*g(n).

O-notation as a Graph

abovesimple to calculateand analyse andbigger

complex

• The fancy name for calculating big-Oh (and the other bounds mentioned later) is asymptotic analysis.

• Asymptotic means "a curve whose distance to another curve tends to zero as the curves travel off to infinity"o this is seen by the limit of f(n)/c*g(n) curve, as n ∞:o 0 ≤ f(n) ≤ c*g(n) for all n ≥ n0o so 0 ≤ f(n)/c*g(n) ≤ 1, such that f(n)/c*g(n) a constant

Asymptotic Analysis

• = 1 if d small relative to ho so c g(n) is close to f(n),

or 'tight'; we want this

o or 0 if d is big relative to h; no good, we need a closer g(n)

o or a constant between 0 and 1; this is okay (closer to 1 is better)

Limit of f(n)/c g(n)alwaysbigger

• T(n) = 10n2 + 50n + 100o can be written as T(n) is O(n2)

• Why?o f(n) = 10n2 + 50n + 100; g(n) = n2

o Witnesses: n0 = 1, c = 160o then f(n) <= c*g(n), n >= 1

so 10n2 + 50n + 100 <= 160 n2

since10n2 + 50n + 100 <=10n2 + 50n2 + 100n2 <= 160 n2

Example 1Informally, the n2 part is the most

important thing inthe T() function

T() and O() Graphed

T(n) = 10n2 + 50n + 10( f(n) == 10n2 + 50n + 10)

T(n) is O(n2)(c g(n) == 160n2)

n0 == 1

http://dlippman.imathas.com/ graphcalc/graphcalc.html

f(n)/c g(n) constant

(10n2 + 50n + 10) / (160n2)

ny = 1/16 = 0.0625

as n ∞ the equationwill become a constant

• T(n) = (n+1)2

o can be wriiten as T(n) is O(n2)

• Why?o f(n) = (n+1)2; g(n) = n2

so (n+1)2 <= 4n2

sincen2 + 2n + 1 <=n2 + 2n2 + n2 <= 4n2

Example 2

T() and O() Graphed

T(n) = (n+1)2

(f(n) == (n+1)2

T(n) is O(n2)(c g(n) == 4n2)

n0 == 1

• T(n) = n10

o can be written as T(n) is O(2n)• Why?

o f(n) = n10 ; g(n) = 2n

so n10 <= 2n

since 10*log2 n <= n (by taking logs of both sides) which is true when n >= 64

(10*log2 64 == 10*6; 60 <= 64)

Example 3

n10 and 2n Graphed

(58.770, 4.915E17)

T(n) = n10

f(n) == n10

T(n) is O(2n)(c g(n) == 2n

• Is a running tgime of n10 slow?o not sure

• But the maths shows that T(n) = n10 is about the same as 2n

• We know 2n is very slow, and that means that n10 is very slow also

Is n10 slow?

• When choosing an O() approximation to T(), remember that:o constant factors do not matter

• e.g. T(n) = (n+1)2 is O(n2)

o low-order terms do not matter• e.g. T(n) = 10n2 + 50n + 100 is O(n2)

o there are many possible witnesses because there are usually many O() graphs that are above the T() equation

2.4. Some Observations about O()

• Inside an O() expressions, always drop constant factors and low-order terms.

• For example:o T(n) = 3n5+ 10n4 + no T(n) is O(3n5)o but, T(n) is O(n5) is simpler and tighter

• this means that the O() is closer to the T() curve

2.5. Simplifying O() Expressions

• Up to now, I have calculated T(n) = f(n) by counting instructions (e.g. see the loop example), and then simplified T(n) to becomes T(n) is O(g(n))

• We can calculate the big-oh function, g(), directly, without counting instructionso easier and faster

3. Calculating Big-Oh Directly

• First decide on a size measure for the data in the program. This will become the n.

• Data Type Possible Size Measureinteger its valuestring its lengtharray its length

3. Big-Oh for Programs

• The Big-Oh value for a program is built up in stages by:o 1) Calculate the Big-Oh’s for all the simple statements in

the program• e.g. assignment, arithmetic

o 2) Then use those value to obtain the Big-Oh’s for the complex statements

• e.g. blocks, for loops, if-statements

3.1. Building a Big-Oh Result

• We assume that simple statements always take a constant amount of time to executeo written as O(1)o this is not a time unit (not 1 ms, not 1 microsec)o O(1) means a running time independent of the input size

• Kinds of simple statements:o assignment, break, continue, return, all library functions

(e.g. putchar(),scanf()), arithmetic, boolean tests, array indexing

Simple Statements (in C)

• The Big-Oh value for a complex statement is a combination of the Big-Oh values of its component simple statements.

• Kinds of complex statements:o blocks ... o conditionals: if-then-else, switcho loops: for, while, do-while

Complex Statements

continued

• The easiest way to see how complex statement timings are based on simple statements (and other complex statements) is by drawing a structure tree for the program.

3.2. Structure Trees

void main() int i;

(1) scanf(“%d”, &i);(2) while (i > 0) (3) putchar(‘0’ + i%2);(4) i = i/2;

(5) putchar(‘\n’);

Example: binary conversion

Structure Tree for Example

block1-5

1 5while2-4

block3-4

the time for thisis the time for

(3) + (4)

• Blocks: Running time bound = summation of the bounds of its parts.

• The summation rule means that only the largest Big-Oh value is considered.

3.3. Details for Complex Statements

"summation" means 'add'

Block Calculation Graphically

O( f1(n) )

O( f2(n) )

O( fk(n) )

O( f1(n) + f2(n) + ... + fk(n))

In other words:

O( largest fi(n) )

summation rule

The cost of a block is the cost of the biggest statement.

• First block's time T1(n) = O(n2)• Second block's time T2(n) = O(n)

• Total running time = O(n2 + n)= O(n2)the largest part

Block Summation Rule Example

• Conditionals: Running time bound = the cost of the if-test + larger of the bounds for the

if- and else- parts

• When the if-test is a simple statement (a boolean test), it is O(1).

Conditionalse.g. if statements, switches

Conditional Graphically

ElsePart

IfPart

O( max( f1(n), f2(n)) +1 )which is the same asO( max( f1(n), f2(n)) )

O( f1(n) ) O( f2(n) )

• Code fragment:if (x < y) // O(1) foo(x); // O(n)else bar(y); // O(n2)

• Total running time = O( max(n,n2) + 1)= O(n2 + 1)= O(n2)

If Example

Usually the cost ofan if-statement is the cost of the biggestbranch.

• Loops: Running time bound is usually =the max. number of times round the loop *the time to execute the loop body once

• But we must include O(1) for the increment and test each time around the loop.

• Must also include the initialization and final test costs (both O(1)).

While Graphically

O( f(n) )

At mostlp(n) times

aroundO( lp(n)*f(n) )

Altogether this is:O( lp(n)*(f(n)+1) + 1 )

which can be simplified to:O( lp(n)*f(n) )

• Code fragment:x = 0;while (x < n) // O(1) for test foo(x, n); // O(n2) x++; // O(1)

• Total running time of loop:= O( n*( 1 + n2 + 1) + 1 )= O(n3 + 2n + 1) = O(n3)

While Loop Example

Usually the cost ofa loop is the cost of the body * the number of loops.lp(n) = n f(n) = 1 + n2 + 1

Do-While Graphically

O( f(n) )

At mostlp(n) times

around

O( lp(n)*f(n) )

Altogether this is:O( lp(n)*(f(n)+1) + 1 )

which can be simplified to:O( g(n)*f(n) )

Usually the cost ofa loop is the cost of the body * the number of loops.

For-loop Graphically

Increment

Initialize

O( f(n) )At most

lp(n) timesaround

O( lp(n)*(f(n)+1+1) + 1)which can be simplified to:

O( lp(n)*f(n) )

O(1)Usually the cost ofa loop is the cost of the body * the number of loops.

• Code Fragment:for (i=0; i < n; i++) foo(i, n); // O(n2)

• It helps to rewrite this as a while loop:

i=0; // O(1)while (i < n) // O(1) for test foo(i, n); // O(n2) i++; // O(1)

For Loop Example

continued

• Running time for the for loop:= O( 1 + n*( 1 + n2 + 1) + 1 )= O( 2 + n3 + 2n )= O(n3)

Usually the cost ofa loop is the cost of the body * the number of loops.lp(n) = n f(n) = 1 + n2 + 1

(1) for(i=0; i < n; i++)_ (2) for (j = 0; j < n; j++)(3) A[i][j] = 0;

• line (3) is a simple op - takes O(1)• line (2) is a loop carried out n times

o takes O(n *1) = O(n)• line (1) is a loop carried out n times

o takes O(n * n) = O(n2)

3.4.1. Example: nested loops

(1) if (A[0][0] == 0) (2) for(i=0; i < n; i++)_ (3) for (j = 0; j < n; j++)(4) A[i][j] = 0;

(5) else (6) for (i=0; i < n; i++)(7) A[i][i] = 1;

3.4.2. Example: if statement

continued

• The if-test takes O(1);the if block takes O(n2);the else block takes O(n).

• Total running time:= O(1) + O( max(n2, n) )

= O(1) + O(n2)

= O(n2) // using the summation rule

void main() int i;

(1) scanf(“%d”, &i);(2) while (i > 0) (3) putchar(‘0’ + i%2);(4) i = i/2;

(5) putchar(‘\n’);

3.4.3. Time for a Binary Conversion

continued

• Lines 1, 2, 3, 4, 5: each O(1)• Block of 3-4 is O(1) + O(1) = O(1)• While of 2-4 loops at most (log2 i)+1 times

o total running time = O(1 * log2 i+1) = O(log2 i)• Block of 1-5:

= O(1) + O(log2 i) + O(1)= O(log2 i)

• Assume i = 2k

• Start 1st iteration, i = 2k

Start 2nd iteration, i = 2k-1

Start 3rd iteration, i = 2k-2

Start kth iteration, i = 2k-(k-1) = 21 = 2Start k+1th iteration, i = 2k-k = 20 = 1o the while will terminate after this iteration

• Since 2k = i, so k = log2 i• So k+1, the no. of iterations, = (log2 i)+1

Why (log2 i)+1 ?

Product rule

Quotient rule

Power rule

Change of base

Cancelation

Identities

Using a Structure Treeblock1-5

1 5while2-4

block3-4

O(1)O(1)

O(log2 i)

void selectionSort(int A[], int n) int i, j, small, temp;(1) for (i=0; i < n-1; i++) //outer loop(2) current = i;(3) for(j=i+1; j < n; j++) //inner loop(4) if (A[j] < A[small]) (5) current= j;(6) temp = A[current]; // exchange(7) A[current] = A[i];(8) A[i] = temp;

3.4.4. Selection Sort Code

Execution Example

OuterLoop #1

OuterLoop #2

OuterLoop #3

OuterLoop #4

inner loops

when n-1 elemsare sortedwe can stop!

sorted

Selection Sort Structure Tree

for1-8

block2-8

for3-52

• Lines 2, 5, 6, 7, 8: each is O(1)• If of 4-5 is O(max(1,0)+1) = O(1)• For of 3-5 is O( (n-(i+1))*1) = O(n-i-1)

= O(n), simplified• Block of 2-8

= O(1) + O(n) + O(1) + O(1) + O(1) = O(n)• For of 1-8 is:

= O( (n-1) * n) = O(n2 - n)= O(n2), simplified

if part else part

• In this section, we assume that the functions are not recursiveo we add recursion in section (5)

• Size measures for all the functions must be similar, so they can be combined to give the program’s Big-Oh value.

4. Analyzing Function calls

#include <stdio.h>

int bar(int x, int n); int foo(int x, int n):

void main() int a, n;(1) scanf(“%d”, &n);(2) a = foo(0, n);(3) printf(“%d\n”, bar(a,n));

Example Program

continued

int bar(int x, int n) int i;(4) for(i = 1; i <= n; i++)(5) x += i;(6) return x;

int foo(int x, int n) int i;(7) for(i = 1; i <= n; i++)(8) x += bar(i, n);(9) return x;

Calling Graph

• 1. Calculate times for Group 0 functionso those that call no other user functions

• 2. Calculate times for Group 1 functionso those that call Group 0 functions only

• 3. Calculate times for Group 2 functionso those that call Group 0 and Group 1 functions only

• 4. Continue until the time for main() is obtained.

Calculating Times with a Calling Graph

• Group 0: bar() is O(n)

• Group 1: foo() is O( n * n) = O(n2)

• Group 2: main() is= O(1) + O(n2) + O(1) + O(n)

= O(n2)

Example Program Analysis

bar() in body

• Recursive functions call themselves with a “smaller size” argument, and terminate by calling a base case.

int factorial(int n) if (n <= 1) return 1; else return n*factorial(n-1);

5. Analyzing Recursive Functions

• 1. Develop basis and inductive statements for the running time.

• 2. Solve the corresponding recurrence relation.o this usually requires the Big-Oh notation to be

rewritten as constants and multiples of no e.g. O(1) becomes a, O(n) becomes b*n,

O(n2) becomes c*n2, etc.

Running Time for a Recursive Function

continued

this is sometimes called the Iteration Method

• 3. Translate the solved relation back into Big-Oh notationo rewrite the remaining constants back into Big-Oh

formo e.g. a becomes O(1), b*n becomes O(n)

• Step 1.o Basis: T(1) = O(1)o Induction: T(n) = O(1) + T(n-1), for n > 1

• Step 2.o Simplify the relation by replacing the O() notation

with constants.o Basis: T(1) = ao Induction: T(n) = b + T(n-1), for n > 1

5.1. Factorial Running Time

• The simplest way to solve T(n) is to calculate it for some values of n, and then guess the general expression.

T(1) = aT(2) = b + T(1) = b + aT(3) = b + T(2) = 2b + aT(4) = b + T(3) = 3b + a

• “Obviously”, the general form is:T(n) = ((n-1)*b) + a

= bn + (a-b)

continued

• Step 3. Translate back:T(n) = bn + (a-b)

• Replace constants by Big-Oh notation:T(n) = O(n) + O(1)

= O(n)

• The running time for recursive factorial is O(n). That is fast.

void rSSort(int A[], int n) int imax, i; if (n == 1) return; else imax = 0; /* A[0] is biggest */ for (i=1; i < n; i++) if (A[i] > A[imax]) imax = i; swap(A, n-1, imax); rSSort(A, n-1);

5.2. Recursive Selection Sort

• Step 1.o Basis: T(1) = O(1)o Induction: T(n) = O(n-1) + T(n-1), for n > 1

• Step 2.o Basis: T(1) = ao Induction: T(n) = b(n-1) + T(n-1), for n > 1

Running Time

continued

multiple of n-1

theloop call to

rSSort()

Assume swap() is O(1), so ignoren == the size of the array

• Solve the relation:o T(1) = a

T(2) = b + T(1) = b + aT(3) = 2b + T(2) = 2b + b + aT(4) = 3b + T(3) = 3b + 2b + b + a

• General Form:o T(n) = (n-1)b + ... + b + a

= a + b(n-1)n/2

continued

• Step 3. Translate back:T(n) = a + b(n-1)n/2

• Replace constants by Big-Oh notation:T(n) = O(1) + O(n2) + O(n)

= O(n2)

• The running time for recursive selection sort is O(n2). That is slow for large arrays.

int binSrch(char A[], int i,int j, char key) int k; if (i > j) /* key not found */ return -1; k = (i+j)/2; if (key == A[k]) /* key found */ return k; if (key < A[k]) j = k-1; /* search left half */ else i = k+1; /* search right half */ return binSrch(A, i, j, key);

5.3. Binary Search

/* find 'h' in A[] */binSrch(A, 0, 7, ‘h’);

Execution Example0 1 2 3 4 5 6 7

A a d f g h w x y

you use binary searchto search for a numberin a phone book

• // find '6' in B[]binSrch(B, 0, 6, '6')

Another Example0 1 2 3 4 5 6

• Step 1.o Basis: T(1) = O(1)o Induction: T(n) = O(1) + T(< n/2 >), for n > 1

• Step 2.o Basis: T(1) = ao Induction: T(n) = b + T(< n/2 >), for n > 1

Running Timen == the range of the array

being looked at

• This time the relation is harder to solve since the precise value of < n/2 > depends on n.

• This affects how many calls it takes to get to T(1).

• Assume the simple case:o n starts as a power of 2: 2k

Solve the Relation

continued

• Evaluate T(n), where n is 2k:T(2k) = b + T(2k-1)

= b + b + T(2k-2)= b + b + b + T(2k-3)= b + b + b + .... + b + T(1)

• General Form:T(n) = bk +a

continued

k of these

e.g. T(32) = b + b + b + b + b + 1

Notice we are notstarting at T(1) in thisexample; that's ok.

• We assumed that 2k = n, so k = log2 n• This means that:

T(n) = b*log2 n + a

• Step 3. Replace constants by Big-Oh notation:T(n) = O(log2 n) + O(1)

= O(log2 n)

• Running time for binary search is O(log2 n).

6. Towers of HanoiPole A

Pole BPole C

How long to move 50 disksfrom A to B? 36 years

• Problem: transfer all the disks from pole A to pole B.o 1. In each step, only 1 disk can be moved from one

pole to another.

o 2. A disk may never be placed on top of a smaller disk.

o 3. Pole C may be used.

• In the following, X --> Y means “move the top disk from Pole X to Pole Y”.

• Problem: move 3 disks from A to B

• Solution:A --> B, A --> C, B --> C,

A --> B,C --> A, C --> B, A --> B

Solution for 3 Disks

void move(int n, Pole x, Pole y, Pole z)/* move n disks from x to y, use z */ if (n == 1) print “x --> y”; else move( n-1, x, z, y); // x to z, use y print “x --> y”; move( n-1, z, y, x); // z to y, use x

Pseudocode

• move( n-1, x, z, y);

print “x --> y”;

move( n-1, z, y, x);

Recursive Part Graphically

x y zmove(n, x, y, z)

Execution for 3 Disksmove 3 from A to B

using C

move 2 from A to Cusing B

move 2 from C to Busing A

move 1 from A to Busing C

move 1 from B to Cusing A

A --> B

A --> C

A --> B B --> C

start stop

. . . .

continued

nextslide

move 2 from C to Busing A

move 1 from A to Busing C

move 1 from C to Ausing B C --> B

C --> A A --> B

. . . .

• We will calculate the running time by using the number of disk moves as the size measure.

• So T(n) = number of disk moves to solve the n-disk problem.

Size for the Running Time

• Step 1.o Basis: T(1) = O(1)o Induction: T(n) = 2*T(n-1) + O(1), for n > 1

• Step 2.o Basis: T(1) = ao Induction: T(n) = 2*T(n-1) + b, for n > 1

Running Time

continued

• Solve the relation:T(n) = 2T(n-1) + b

= 2( 2T(n-2) + b ) + b= 4T(n-2) + 2b + b= 4( 2T(n-3) + b ) + 2b + b= 8T(n-3) + 4b + 2b + b

:= 2n-1T(1) + (2n-2 + 2n-3 + ... + 2 + 1)b= 2n-1a + (2n-2 + 2n-3 + ... + 2 + 1)b~= (2n-1 + 2n-2 + 2n-3 + ... + 2 + 1)b= (2n - 1)b

continued

• Step 3. Translate back:T(n) = (2n - 1)b

• Replace constants by Big-Oh notation:T(n) = O(2n)

• The running time for “Towers of Hanoi” is O(2n)o exponential growth!

continued

• This exponential growth means that a practical limit is quickly reached to the number of disks that can be moved.

• Look at slide 16 to see approximately how long it would take to move 1000 disks if a single disk move takes 1 microsec.

• “Towers of Hanoi” written in every language (C, C++, Java, JavaScript, Fortran, Ada, ML, etc):http://www.pangea.ca/kolar/javascript/Hanoi/HTonWebE.html

• Some history about the problem, animations, a Windows game:http://www.lhs.berkeley.edu/Java/Tower/

More Information

• Most of the time we want to calculate the worst case running time, which means big-Oh (O()).

• There are several other asymptotic notations, which are quite useful, especially big-theta.

7. Other Asymptotic Notations

Big-Oh Definition Again

continued

• n0 and c are called witnesses to the relationship:T(n) = f(n) and T(n) is O(g(n) )

• We can write this as:o f(n) is O(g(n)) // leave out the

We write T(n) = f(n) as T(n) is O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 f(n) c*g(n) for all n n0.

• O-notation gives an upper bound for a function to within a constant factor. We write f(n) is O(g(n)) if there are positive constants n0 and c such that at and to the right of n0, the value of f(n) always lies on or below c*g(n).

O-notation as a Graph

• Called big-omega.

Ω Notation

Ω(g(n)) = f(n) : there exist constants c > 0, n0 >0 such that 0 c*g(n) f(n) for all n n0

• f(n) is in a set of functions that are greater than or equal to g(n)• f(n) is bounded below by g(n)

EXAMPLE: √n is Ω(log n)√n is greater than or equal to log n

• Ω-notation gives a lower bound for a function to within a constant factor. We write f(n) is Ω(g(n)) if there are positive constants n0 and c such that at and to the right of n0, the value of f(n) always lies on or above c*g(n).

Ω-notation as a Graph

T() and omega() Graphed

T(n) = n0.5

Ω(n) = log n

√n is Ω(log n)

-notation (tight bounds)

(g(n)) = O(g(n)) (g(n))

EXAMPLE: )( is 2 2221 nnn

Less than and greater than or equal to.This make polynomials easy to simplify: ad nd + ad–1nd–1 + + a1n + a0 = (nd)

big-theta

• -notation bounds a function to within constant factors. We write f(n) is (g(n)) if there exist positive constants n0, c1, and c2 such that at and to the right of n0, the value of f(n) always lies between c1*g(n) and c2*g(n) inclusive.

-Notation as a Graph

T() and () GraphedT(n) = 0.5 n2 – 2n

Ω(n) = 0.2 n2

above)( is 2 22

21 nnn O(n) = n2

C1 = 0.2and C2 = 0.5

The O() and Ω(n) curvescould easily be tighter(i.e. closer to T(n)).How?

• The -notation means that:o c1*n2 ≤ 0.5*n2 – 2*n ≤ c2*n2 , for all n >=

• Dividing by n2 gives:o c1 ≤ 0.5 - 2/n ≤ c2

Consider c2• When n ≥ 0, then the r.h.s of the inequality is

o 0.5 - 2/n (or smaller subtraction) ≤ c2• When n == ∞

o 0.5 ≤ c2

Calculating c1 and c2

Consider c1• When n ≥ 5, then the l.h.s of the inequality is

o c1 ≤ 0.5 - 2/n (or a smaller subtraction)• When n == 5

o c1 ≤ 1/2 - 2/5 = 5/10 - 4/10o c1 ≤ 1/10

• Thus we can choose c1 = 1/10 c2 = 1/2, n0 = 5

T() and () Graphed Again

T(n) = 0.5 n2 – 2n

Ω(n) = 0.1* n2

)( is 2 2221 nnn O(n) = 0.5*n2

C1 = 0.1and C2 = 0.5

n0 == 5

o-notation and -notation

o(g(n)) = f(n) : for any constant c > 0, there is a constant n0 > 0 such that 0 f(n) < c*g(n) for all n n0

EXAMPLE: (n0 = 2/c)

O-notation and -notation are like and .o-notation and -notation are like < and >.

2n2 is o(n3)

little-oh little-omega

)(log is nn

o-notation and -notation

(g(n)) = f(n) : for any constant c > 0, there is a constant n0 > 0 such that 0 c*g(n) < f(n) for all n n0

EXAMPLE: (n0 = 1+1/c)

O-notation and -notation are like and .o-notation and -notation are like < and >.

little-oh little-omega

• Intuitively,

Asymptotic Summary

o() is like < O() is like

() is like > () is like

() is like =

the two important ones

Algorithm Design and Analysis (ADA)

time of programs242

n scanf

n fori

analysis ada242

worst case

web pages objectiveto

Documents

242-535 ADA: 1. Induction1 Objective o to introduce...

Introduction to Algorithm Analysis - Nanjing...

Design and Analysis of Algorithm Greedy Algorithm

MELJUN CORTES ALGORITHM Analysis of Algorithm Efficiency

CSC401 – Analysis of Algorithms Chapter 1 Algorithm...

Asymptotic Algorithm Analysis

Analysis Recursive Algorithm

Algorithm Analysis

Fundamentals of Algorithm Analysis Algorithm : Design &...

Image analysis algorithm

Algorithm Analysis 1

Analysis of Algorithm

Emotion recognition system through thermal facial...

242-535 ADA: 0. Preliminaries1 Objective o to give some...

Algorithm analysis basics

algorithm Chapter 2 Algorithm Analysis