Analysis & Design of Algorithms (CSCE 321)

Prof. Amr Goneid, AUC 1

Analysis & Design of Analysis & Design of AlgorithmsAlgorithms(CSCE 321)(CSCE 321)

Prof. Amr GoneidDepartment of Computer Science, AUC

Part 2. Types of Complexities

Prof. Amr Goneid, AUC 2

Types of ComplexitiesTypes of Complexities

Prof. Amr Goneid, AUC 3

Types of ComplexitiesTypes of Complexities

Rules for Upper Bound Comparing Complexities Types of Complexities

Prof. Amr Goneid, AUC 4

1. Rules for Upper Bound1. Rules for Upper Bound

If (k) is a constant, then: O(k) O(n) O(k f(n)) = O(f(n))

T(n)

n

O(n)

O(k)

O(kn)

Prof. Amr Goneid, AUC 5

Rules for Big ORules for Big O

O(f(n)) + O(g(n)) = max(O(f(n)) ,O(g(n))) e.g. f(n) = 2n = O(n), g(n) = 0.1 n3 = O(n3)

T(n) = max(O(n) , O(n3)) = O(n3)

2n

0.1 n3

O(n3)

Prof. Amr Goneid, AUC 6

Rules for Big ORules for Big O

O(f(n)) * O(g(n)) = O(f(n) * g(n))

e.g. f(n) = n, g(n) = n2

T(n) = n * n2 = O(n3)

O(n2)

Repeat n timesO(n3)

Prof. Amr Goneid, AUC 7

Rules for Big ORules for Big O

Logarithmic Complexity < Linear Complexity O(log n) O(n)Claim:for all n ≥ 1, log n ≤ n. Prove by induction on n.

Prof. Amr Goneid, AUC 8

Rules for Big ORules for Big O

For a polynomial of degree m,

Prove! O(nm-1) O(nm) follows from above

)n(Onnn)n(T.g.e

)n(O)n(TThen

na...nanaa)n(P)n(Tm

m

mm

332

2

210

2342

Prof. Amr Goneid, AUC 9

Summary of Rules for Big-OSummary of Rules for Big-O

Rule ExampleFor constant k, O(k) < O(n) O(7) = O(1) < O(n)

For constant k, O(kf) = O(f) O(2n) = O(n)

O(|f|+|g|) = O(|f|) + O(|g|) =

Max (O(|f|) , O(|g|)

O(6n2+n)=O(6n2)+O(n) = O(n2)

Nesting of loop O(g) within a loop O(f) gives O(f*g)

O(n4*n2)=O(n6)

O(nm-1) < O(nm) O(n2) < O(n3)

O((log n)k) O(n) O(log n) < O(n)

Prof. Amr Goneid, AUC 10

2. Comparing Complexities2. Comparing Complexities

Dominance:

If lim(n->) f(n)/g(n) =

then f(n) dominates (i.e. grows faster), but if

lim(n->) f(n)/g(n) = 0

then g(n) dominates.

In the latter case, we say that f(n) = o(g(n))

little oh

Prof. Amr Goneid, AUC 11

Comparing ComplexitiesComparing Complexities

Examples: if a > b then na dominates nb

Why? n2 dominates (3n+2)

Why? n2 dominates (n log n)

Why?

Prof. Amr Goneid, AUC 12

Using l’Hopital’s Rule (1696)Using l’Hopital’s Rule (1696)

Example: Show that

f(n) = n(k+α) + nk (log n)2 and

g(n) = k n(k+α)

grow at the same rate.

dn

d means prime thewhere

then

and If

)n(g

)n(flim

)n(g

)n(flim

)n(glim)n(flim

'

'

nn

nn

Prof. Amr Goneid, AUC 13

Comparing ComplexitiesComparing Complexities

Which grows faster:

f(n) = n3 or g(n) = n log n

f(n) = n 0.001 or g(n) = log n

f(n) = 2n+1 or g(n) = 2n

f(n) = 2n or g(n) = 22n

Prof. Amr Goneid, AUC 14

ExercisesExercises

Which function has smaller complexity ? f = 100 n4 g = n5

f = log(log n3) g = log n f = n2 g = n log n f = 50 n5 + n2 + n g = n5

f = en g = n!

Prof. Amr Goneid, AUC 15

3. Types of Complexities3. Types of Complexities

Constant Complexity T(n) = constant independent of (n) Runs in constant amount of time O(1) Example: cout << a[0][0]

Prof. Amr Goneid, AUC 16

Types of ComplexitiesTypes of Complexities

Logarithmic Complexity Log2n=m is equivalent to n=2m

Reduces the problem to half O(log2n) Example: Binary Search

T(n) = O(log2n)

Much faster than Linear Search which has T(n) = O(n)

Prof. Amr Goneid, AUC 17

Linear vs Logarithmic Complexities

0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10 11 12 13 14

x

x

logx

n

T(n)

O(log2n)

O(n)

Prof. Amr Goneid, AUC 18

Types of ComplexitiesTypes of Complexities

Polynomial Complexity T(n)=amnm+…+ a2n2 + a1n1 + a0

If m=1, then O(a1n+a0) O(n) If m > 1, then O(nm) as nm dominates

Prof. Amr Goneid, AUC 19

Polynomials Complexities

O(n)

O(n2)

O(n3)

n

Log

T(n

)

Prof. Amr Goneid, AUC 20

Types of ComplexitiesTypes of Complexities

Exponential Example: List all the subsets of a set of n

elements {a,b,c}

{a,b,c}, {a,b},{a,c},{b,c},{a},{b},{c},{} Number of operations T(n) = O(2n) Exponential expansion of the problem

O(an) where a is a constant greater than 1

Prof. Amr Goneid, AUC 21

Exponential Vs Polynomial

Log

T(n

)

n

O(n3)

O(n)

O(2n)

Prof. Amr Goneid, AUC 22

Types of ComplexitiesTypes of Complexities

Factorial time Algorithms Example: Traveling salesperson problem (TSP):

Find the best route to take in visiting n cities away from home. What are the number of possible routes? For 3 cities: (A,B,C)

Prof. Amr Goneid, AUC 23

Possible routes in a TSPPossible routes in a TSP

M on trea l

N Y

N Y

M on trea l

A m s terd am

M on trea l

A m s terd am

A m s terd am

M on trea l

N Y

N Y

A m s terd am

A m s terd am

N Y

M on trea l

H om e

–Number of operations = 3!=6, Hence T(n) = n!

–Expansion of the problem O(n!)

Prof. Amr Goneid, AUC 24

Exponential Vs Factorial

Log

T(n

)

n

O(2n)

O(n!)

O(nn)

Prof. Amr Goneid, AUC 25

Execution Time ExampleExecution Time Example

Example: For the exponential algorithm of listing all

subsets of a given set, assume the set size to be of 1024 elements

Number of operations is 21024 about 1.8*10308

If we can list a subset every nanosecond the process will take 5.7 * 10291 yr!!!

Prof. Amr Goneid, AUC 26

PP and and NP – NP – TimesTimes

P (Polynomial) Times:

O(1), O(log n), O(log n)2, O(n), O(n log n),

O(n2), O(n3), ….

NP (Non-Polynomial) Times:

O(2n) , O(en) , O(n!) , O(nn) , …..

Prof. Amr Goneid, AUC 27

PP and and NP – NP – TimesTimes

Polynomial time is

GOODTry to reduce the polynomial

power NP (e.g. Exponential) Time is

BAD