A simple example

A simple example• finding the maximum of a set S of n numbers

• Time complexity:

• Calculation of T(n): Assume n = 2k,

T(n) = 2T(n/2)+1= 2(2T(n/4)+1)+1= 4T(n/4)+2+1:=2k-1T(2)+2k-2+…+4+2+1=2k-1+2k-2+…+4+2+1=2k-1 = n-1

T(n)= 2T(n/2)+1 1 , n>2 , n2

Time complexity

A general divide-and-conquer algorithm

Step 1: If the problem size is small, solve this problem directly; otherwise, split the original problem into 2 sub-problems with equal sizes.

Step 2: Recursively solve these 2 sub-problems by applying this algorithm.

Step 3: Merge the solutions of the 2 sub- problems into a solution of the original problem.

Time complexity of the general algorithm

• Time complexity:

where S(n) : time for splitting M(n) : time for merging b : a constantc : a constant

• e.g. Binary search• e.g. quick sort• e.g. merge sort

T(n)= 2T(n/2)+S(n)+M(n) b

, n c , n < c

Divide and Conquer

• Divide-and-conquer method for algorithm design:– Divide: if the input size is too large to deal with in a

straightforward manner, divide the problem into two or more disjoint subproblems

– Conquer: use divide and conquer recursively to solve the subproblems

– Combine: take the solutions to the subproblems and “merge” these solutions into a solution for the original problem

Recurrences

• Running times of algorithms with Recursive calls can be described using recurrences

• A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs

• Example: Merge Sort

(1) if 1( )

2 ( / 2) ( ) if 1n

T nT n n n

solving_trivial_problem if 1( )

num_pieces ( / subproblem_size_factor) dividing combining if 1n

T nT n n

Solving Recurrences• Repeated substitution method

– Expanding the recurrence by substitution and noticing patterns

• Substitution method– guessing the solutions– verifying the solution by the mathematical induction

• Recursion-trees• Master method

– templates for different classes of recurrences

Repeated Substitution Method• Let’s find the running time of merge sort (let’s

assume that n=2b, for some b).1 if 1

( )2 ( / 2) if 1

nT n

T n n n

2

2

3

lg

( ) 2 / 2 substitute2 2 / 4 / 2 expand

2 ( / 4) 2 substitute2 (2 ( /8) / 4) 2 expand

2 ( /8) 3 observe the pattern

( ) 2 ( / 2 )2 ( / ) lg lg

i i

n

T n T n nT n n n

T n nT n n n

T n n

T n T n inT n n n n n n n

Repeated Substitution Method• The procedure is straightforward:

– Substitute– Expand– Substitute – Expand– …– Observe a pattern and write how your expression looks

after the i-th substitution– Find out what the value of i (e.g., lgn) should be to get the

base case of the recurrence (say T(1))– Insert the value of T(1) and the expression of i into your

expression

Integer Multiplication

• Algorithm: Multiply two n-bit integers I and J.– Divide step: Split I and J into high-order and low-order bits

– We can then define I*J by multiplying the parts and adding:

– So, T(n) = 4T(n/2) + n, which implies T(n) is O(n2).– But that is no better than the algorithm we learned in grade

school.

ln

h

ln

h

JJJ

III

2/

2/

2

2

lln

hln

lhn

hh

ln

hln

h

JIJIJIJI

JJIIJI

2/2/

2/2/

222

)2(*)2(*

An Improved Integer Multiplication Algorithm

• Algorithm: Multiply two n-bit integers I and J.– Divide step: Split I and J into high-order and low-order bits

– Observe that there is a different way to multiply parts:

– So, T(n) = 3T(n/2) + n, which implies T(n) is O(nlog2

3), by the Master Theorem.– Thus, T(n) is O(n1.585).

ln

h

ln

h

JJJ

III

2/

2/

2

2

lln

hllhn

hh

lln

llhhhlhhlllhn

hh

lln

llhhhllhn

hh

JIJIJIJI

JIJIJIJIJIJIJIJI

JIJIJIJJIIJIJI

2/

2/

2/

2)(2

2])[(2

2]))([(2*

Matrix multiplication

• Let A, B and C be n n matrices C = AB C(i, j) = A(i, k)B(k, j)

• The straightforward method to perform a matrix multiplication requires O(n3) time.

1 k n

Divide-and-conquer approach• C = AB

C11 = A11 B11 + A12 B21 C12 = A11B12 + A12 B22 C21 = A21 B11 + A22 B21 C22 = A21 B12 + A22 B22 • Time complexity: (# of additions : n2)

We get T(n) = O(n3)

C11 C12 = A11 A12 B11 B12 C21 C22 = A21 A22 B21 B22

T(n) = b 8T(n/2)+cn2 , n 2 , n > 2

• P = (A11 + A22)(B11 + B22)Q = (A21 + A22)B11

R = A11(B12 - B22)S = A22(B21 - B11)T = (A11 + A12)B22

U = (A21 - A11)(B11 + B12)V = (A12 - A22)(B21 + B22).

• C11 = P + S - T + VC12 = R + TC21 = Q + SC22 = P + R - Q + U

Strassen’s matrix multiplicaiton

Time complexity• 7 multiplications and 18 additions or subtractions• Time complexity:

T(n) = an2 + 7T(n/2)= an2 + 7(a(n/2)2 + 7T(n/4))= an2 + (7/4)an2 + 72T(n/4)= … := an2(1 + 7/4 + (7/4)2+…+(7/4)k-1+7kT(1)) cn2(7/4)log2n+7log2n, c is a constant= cnlog24+log27-log24 +nlog27 = O(nlog27) O(n2.81)

T(n) = b 7T(n/2)+an2 , n 2 , n > 2

Median Finding

• Given a set of "n" unordered numbers we want to find the "k th" smallest number. (k is an integer between 1 and n).

A Simple Solution• A simple sorting algorithm like heapsort will take Order of

O(nlg2n) time.

Step Running Time Sort n elements using heapsort O(nlog2n)

Return the kth smallest element O(1)Total running time O(nlog2n)

Linear Time selection algorithm

• Also called Median Finding Algorithm.• Find k th smallest element in O (n) time in

worst case.• Uses Divide and Conquer strategy.• Uses elimination in order to cut down the

running time substantially.

Steps to solve the problem

• Step 1: If n is small, for example n<6, just sort and return the kth smallest number in constant time i.e; O(1) time.

• Step 2: Group the given number in subsets of 5 in O(n) time.

• Step3: Sort each of the group in O (n) time. Find median of each group.

• Given a set (……..2,5,9,19,24,54,5,87,9,10,44,32,21,13,24,18,26,16,19,25,39,47,56,71,91,61,44,28………) having n elements.

5

9

19

24

2

10

9

87

5

54

2

13

32

44

19

16

26

18

4

71

56

47

39

25………………..

………………..

………………..

………………..

………………..

………………..

………………..

………………..

………………..

………………..

21

Arrange the numbers in groups of five

Find median of N/5 groups

5

9

19

24

2

87

54

10

9

5

44

32

13

2

26

19

18

16

4

71

56

47

39

25………………..

………………..

………………..

………………..

………………..

………………..

………………..

………………..

………………..

………………..

21

Median of each group

5

9

19

24

2

87

54

10

9

5

44

32

13

2

26

19

18

16

4

71

56

47

39

25………………..

………………..

………………..

………………..

………………..

………………..

………………..

………………..

………………..

………………..

Find the Median of each group

Find m ,the median of medians

21

3.n/10

Find the sets L and R

• Compare the n-1 elements with the median m and find two sets L and R such that every element in L is smaller than m and every element in R is greater than m.

m

L R

3n/10<L<7n/10 3n/10<R<7n/10

Description of the Algorithm step • If n is small, for example n<6, just sort and return the k the smallest

number.(time- 7)• If n>5, then partition the numbers into groups of 5.(time n)• Sort the numbers within each group. Select the middle elements (the

medians). (time- 7n/5)• Call your "Selection" routine recursively to find the median of n/5

medians and call it m. (time-Tn/5)• Compare all n-1 elements with the median of medians m and determine

the sets L and R, where L contains all elements <m, and R contains all elements >m. Clearly, the rank of m is r=|L|+1 (|L| is the size of L) (time- n)

Contd….

• If k=r, then return m• If k<r, then return k th smallest of the set L .(time T7n/10)

• If k>r, then return k-r th smallest of the set R.

Recursive formula

• T (n)=O(n) + T(n/5) +T(7n/10)We will solve this equation in order to get the complexity.

We assume that T (n)< C*n

T (n)= a*n + T (n/5) + T (7n/10)

<= C*n/5+ C*7*n/10 + a*n

<= C*n if 9*C/10 +a <= C

Which implies C >= 10*a

Hence 10*a*n satisfies the recurrence and so T(n) is O(n)

2-D ranking finding• Def: Let A = (a1,a2), B = (b1,b2). A dominates B iff a1> b1 and

a2 > b2

• Def: Given a set S of n points, the rank of a point x is the number of points dominated by x.

B

A C

D

E

rank(A)= 0 rank(B) = 1 rank(C) = 1

rank(D) = 3 rank(E) = 0

L

1+3=4

More efficient algorithm (divide-and-conquer)

A B

0+1=1

0+2=2

2+3=5

2+4=68 1

1

0 0

Straightforward algorithm:

compare all pairs of points : O(n2)

Input: A set S of planar points P1,P2,…,Pn

Output: The rank of every point in SStep 1: (Split the points along the median line L into A and B.) a. If S contains only one point, return its rank as 0. b. Else, choose a cut line L perpendicular to the x-axis such that n/2

points of S have x-values L (call this set of points A) and the remainder points have x-values L(call this set B). Note that L is a median x-value of this set.

Step 2: Find ranks of points in A and ranks of points in B, recursively.Step 3: Sort points in A and B according to their y-values. Scan these points sequentially and determine, for each point in B, the

number of points in A whose y-values are less than its y-value. The rank of this point is equal to the rank of this point among points in B, plus the number of points in A whose y-values are less than its y-value.

Divide-and-conquer 2-D ranking finding

time complexity : step 1 : O(n) (finding median) step 3 : O(n log n) (sorting) total time complexity :

T(n) 2T( n2

) + c1 n log n + c2 n

2T( n2

) + c n log n

4T( n4

) + c n log n2

+ c n log n

nT(1) + c(n log n + n log n2

+n log n4

+…+n log 2)

nT(1) + ncn 2log

2-D maxima finding problem• Def : A point (x1, y1) dominates (x2, y2) if x1 > x2 and

y1 > y2. A point is called a maxima if no other point dominates it

• Straightforward method : Compare every pair of points.

Time complexity: O(n2)

Divide-and-conquer for maxima finding

The maximal points of SL and SR

The algorithm:• Input: A set of n planar points.• Output: The maximal points of S.Step 1: If S contains only one point, return it as the

maxima. Otherwise, find a line L perpendicular to the X-axis which separates the set of points into two subsets SLand SR , each of which consisting of n/2 points.

Step 2: Recursively find the maximal points of SL and SR .Step 3: Find the largest y-value of SR. Project the

maximal points of SL onto L. Discard each of the maximal points of SL if its y-value is less than the largest y-value of SR .

• Time complexity: T(n) Step 1: O(n) Step 2: 2T(n/2) Step 3: O(n)

Assume n = 2k

T(n) = O(n log n)

T(n)= 2T(n/2)+O(n)+O(n) 1

, n > 1 , n = 1

The closest pair problem

• Given a set S of n points, find a pair of points which are closest together.

• 1-D version :Solved by sortingTime complexity :

O(n log n)

2-D version

• at most 6 points in area A:

The algorithm:• Input: A set of n planar points.• Output: The distance between two closest points. Step 1: Sort points in S according to their y-values and x-

values.Step 2: If S contains only two points, return infinity as

their distance.Step 3: Find a median line L perpendicular to the X-axis to

divide S into two subsets, with equal sizes, SL and SR.Step 4: Recursively apply Step 2 and Step 3 to solve the

closest pair problems of SL and SR. Let dL(dR) denote the distance between the closest pair in SL (SR). Let d = min(dL, dR).

Step 5: For a point P in the half-slab bounded by L-d and L, let its y-value by denoted as yP . For each such P, find all points in the half-slab bounded by L and L+d whose y-value fall within yP +d and yP -d. If the distance d between P and a point in the other half-slab is less than d, let d=d . The final value of d is the answer.

• Time complexity: O(n log n) Step 1: O(n log n) Steps 2~5:

T(n) = O(n log n)

T(n)= 2T(n/2)+O(n)+O(n) 1

, n > 1 , n = 1

The convex hull problem

• The convex hull of a set of planar points is the smallest convex polygon containing all of the points.

concave polygon: convex polygon:

• The divide-and-conquer strategy to solve the problem:

• The merging procedure: 1. Select an interior point p.2. There are 3 sequences of points which have

increasing polar angles with respect to p.(1) g, h, i, j, k(2) a, b, c, d(3) f, e

3. Merge these 3 sequences into 1 sequence:g, h, a, b, f, c, e, d, i, j, k.

4. Apply Graham scan to examine the points one by one and eliminate the points which cause reflexive angles.

• e.g. points b and f need to be deleted.

Final result:

Divide-and-conquer for convex hull

• Input : A set S of planar points• Output : A convex hull for SStep 1: If S contains no more than five points, use

exhaustive searching to find the convex hull and return.Step 2: Find a median line perpendicular to the X-axis

which divides S into SL and SR ; SL lies to the left of SR .Step 3: Recursively construct convex hulls for SL and SR.

Denote these convex hulls by Hull(SL) and Hull(SR) respectively.

• Step 4: Apply the merging procedure to merge Hull(SL) and Hull(SR) together to form a convex hull.

• Time complexity: T(n) = 2T(n/2) + O(n) = O(n log n)

The Fast Fourier Transform0 1110987654321 1512 1413

0 1110987654321 1512 1413

Outline

• Polynomial Multiplication Problem • Primitive Roots of Unity• The Discrete Fourier Transform• The FFT Algorithm

Polynomials• Polynomial:

• In general,

432 43825)( xxxxxp

11

2210

1

0

)(

or

)(

nn

n

i

ii

xaxaxaaxp

xaxp

Polynomial Evaluation• Horner’s Rule:

– Given coefficients (a0,a1,a2,…,an-1), defining polynomial

– Given x, we can evaluate p(x) in O(n) time using the equation

• Eval(A,x): [Where A=(a0,a1,a2,…,an-1)]– If n=1, then return a0

– Else, • Let A’=(a1,a2,…,an-1) [assume this can be done in constant time]• return a0+x*Eval(A’,x)

1

0

)(n

i

ii xaxp

)))((()( 12210 nn xaaxaxaxaxp

Polynomial Multiplication• Given coefficients (a0,a1,a2,…,an-1) and (b0,b1,b2,…,bn-1) defining two

polynomials, p() and q(), and number x, compute p(x)q(x).

• Horner’s rule doesn’t help, since

where

• A straightforward evaluation would take O(n2) time. The “magical” FFT will do it in O(n log n) time.

1

0

)()(n

i

ii xcxqxp

i

jjiji bac

0

Polynomial Interpolation & Polynomial Multiplication

• Given a set of n points in the plane with distinct x-coordinates, there is exactly one (n-1)-degree polynomial going through all these points.

• Alternate approach to computing p(x)q(x):– Calculate p() on 2n x-values, x0,x1,…,x2n-1.– Calculate q() on the same 2n x values.– Find the (2n-1)-degree polynomial that goes through the points

{(x0,p(x0)q(x0)), (x1,p(x1)q(x1)), …, (x2n-1,p(x2n-1)q(x2n-1))}.

• Unfortunately, a straightforward evaluation would still take O(n2) time, as we would need to apply an O(n)-time Horner’s Rule evaluation to 2n different points.

• The “magical” FFT will do it in O(n log n) time, by picking 2n points that are easy to evaluate…

Primitive Roots of Unity• A number w is a primitive n-th root of unity, for n>1, if

– wn = 1– The numbers 1, w, w2, …, wn-1 are all distinct

• Example 1:– Z*

11:

– 2, 6, 7, 8 are 10-th roots of unity in Z*11

– 22=4, 62=3, 72=5, 82=9 are 5-th roots of unity in Z*11

– 2-1=6, 3-1=4, 4-1=3, 5-1=9, 6-1=2, 7-1=8, 8-1=7, 9-1=5• Example 2: The complex number e2pi/n is a primitive n-th root of unity,

where

x x^2 x^3 x^4 x^5 x^6 x^7 x^8 x^9 x^101 1 1 1 1 1 1 1 1 12 4 8 5 10 9 7 3 6 13 9 5 4 1 3 9 5 4 14 5 9 3 1 4 5 9 3 15 3 4 9 1 5 3 4 9 16 3 7 9 10 5 8 4 2 17 5 2 3 10 4 6 9 8 18 9 6 4 10 3 2 5 7 19 4 3 5 1 9 4 3 5 110 1 10 1 10 1 10 1 10 1

1i

Properties of Primitive Roots of Unity• Inverse Property: If w is a primitive root of unity, then w -1=wn-1

– Proof: wwn-1=wn=1• Cancellation Property: For non-zero -n<k<n,

– Proof:

• Reduction Property: If w is a primitve (2n)-th root of unity, then w2 is a primitive n-th root of unity.

– Proof: If 1,w,w2,…,w2n-1 are all distinct, so are 1,w2,(w2)2,…,(w2)n-1

• Reflective Property: If n is even, then wn/2 = -1.– Proof: By the cancellation property, for k=n/2:

– Corollary: wk+n/2= -wk.

01

0

n

j

kjw

01

1111)1(

11)(

11)(1

0

kk

k

k

kn

k

nkn

j

kj

wwww

www

)1)(2/(0 2/2/02/02/01

0

)2/( nnnnn

j

jn n wwwwwwww

The Discrete Fourier Transform• Given coefficients (a0,a1,a2,…,an-1) for an (n-1)-degree polynomial p(x)• The Discrete Fourier Transform is to evaluate p at the values

– 1,w,w2,…,wn-1

– We produce (y0,y1,y2,…,yn-1), where yj=p(wj)– That is,

– Matrix form: y=Fa, where F[i,j]=wij.

• The Inverse Discrete Fourier Transform recovers the coefficients of an (n-1)-degree polynomial given its values at 1,w,w2,…,wn-1

– Matrix form: a=F -1y, where F -1[i,j]=w-ij/n.

1

0

n

i

ijij ay w

Correctness of the inverse DFT• The DFT and inverse DFT really are inverse operations• Proof: Let A=F -1F. We want to show that A=I, where

• If i=j, then

• If i and j are different, then

1

0

1],[n

k

kjki

njiA ww

Property)on Cancellati(by 01],[1

0

)(

n

k

kij

njiA w

1111],[1

0

01

0

nnnn

iiAn

k

n

k

kiki www

Convolution• The DFT and the inverse

DFT can be used to multiply two polynomials

• So we can get the coefficients of the product polynomial quickly if we can compute the DFT (and its inverse) quickly…

Pad with n 0's Pad with n 0's

[a0,a1,a2,...,an-1] [b0,b1,b2,...,bn-1]

DFT DFT

[a0,a1,a2,...,an-1,0,0,...,0] [b0,b1,b2,...,bn-1,0,0,...,0]

[y0,y1,y2,...,y2n-1] [z0,z1,z2,...,z2n-1]

ComponentMultiply

inverse DFT

[y0z0,y1z1,...,y2n-1z2n-1]

[c0,c1,c2,...,c2n-1](Convolution)

The Fast Fourier Transform• The FFT is an efficient algorithm for computing the DFT• The FFT is based on the divide-and-conquer paradigm:

– If n is even, we can divide a polynomial

into two polynomials

and we can write

The FFT Algorithm

The running time is O(n log n). [inverse FFT is similar]