Instructor: Shengyu Zhang 1. Example 1: Merge sort 2.

1

Instructor: Shengyu Zhang

2

Example 1: Merge sort

3

Starting example

Sorting: We have a list of numbers . We want to sort them in the increasing order.

4

An algorithm: merge sort

Merge sort: Cut it into two halves with equal size.

Suppose divides for simplicity. Suppose the two halves are sorted: Merge them.

Use two pointers, one for each half, to scan them, during which course do the appropriate merge.

How to sort each of the two halves? Recursively.

5

Complexity?

Suppose this algorithm takes time for an input with numbers.

Thus each of the two halves takes time. The merging?

Scanning elements, an time operation needed for each.

Total amount of time: .

6

How to solve/bound this recurrence relation?

7

A general method for designing algorithm: Divide and conquer Breaking the problem into subproblems

that are themselves smaller instances of the same type of problem

Recursively solving these subproblems

Appropriately combining their answers

8

Complexity

Running time on an input of size : Break problem into subproblems, each of

the same size . In general, is not necessarily equal to .

Time to recursively solve each subproblem: Time for breaking problem (into

subproblems) and combining the answers:

9

Master theorem

, , and are all constants. Then

Proof in textbook. Not required. But you need to know how to apply it.

10

Merge sort: . . So . By the master theorem: .

11

Example 2: Selection

12

Selection

Problem: Given a list of numbers, find the -th smallest.

We can sort the list, which needs . Can we do better, say, linear time? After all, sorting gives a lot more information

than we requested. Not always a waste: consider dynamic

programming where solutions to subproblems are also produced.

13

Idea of divide and conquer

Divide the numbers into 3 parts

, , Depending on the size of each part, we know

which part the -th element lies in.

Then search in that part.

Question: Which to choose?

14

Pivot

Suppose we use a number in the given list as a pivot.

As said, we divide the list into three parts. : Those numbers smaller than : Those numbers equal to : Those numbers larger than

15

After the partition

The division is simple: just scan the list and put elements into the corresponding part. time.

To select the -th smallest value, it becomes

Complexity?

16

Divide and conquer

Note: though there are two subproblems (of sizes and ), we need to solve only one of them. Compare: in quicksort, we need to sort both

substrings! Complexity:

A new issue: and are not determined Depends on the pivot.

17

If the pivot is the median:

Thus finally , better than by sorting.

18

If the pivot is at one end (say, the smallest)

What’s the complexity?

Complexity:

19

The similarity to quicksort tells us: a random pivot performs well It’s away from either end by with const. prob.

To be more precise, it’s in with probability . And in this case, the recursion becomes

20

Recall:

So

21

Thus we can use the following simple strategy: Pick a random pivot, do the recursion

Each random pivot falls in w/ prob. . . It is enough to get good pivots to make the

problem size to drop to 1. Thus .

22

Example 3: Matrix multiplication

23

Matrix multiplication

Recall: the product of two matrices is another matrix.

Question: how fast can we multiply two matrices?

Recall: : the entry in the matrix . Similar for ,

24

This takes multiplications (of numbers).

For a long time, people thought this was the best possible.

Until Straussen came up with the following.

25

If we break the matrix into blocks

Then the product is just block multiplication

8 matrix multiplications of dimension Plus additions.

26

Thus the recurrence is

This gives exactly the same , not interesting.

27

However, Straussen observed that we can actually use only 7 (instead of 8) multiplications of matrices with dimension .

28

God knows how he came up with it. And here is how:

where

29

Thus the recurrence becomes

And solving this gives

30

Best in theory? .

Conjecture: !

In practice? People still use the algorithm most of the time. (For example, in matlab.) It’s simple and robust.

31

Fast Fourier Transform (FFT)

32

Multiplication of polynomials

Many applications need to multiply two polynomials. e.g. signal processing.

A degree- polynomial is

The degree is at most . (The degree is exactly if .) The summation of two polynomials is easy

---------------------------------------------------------------- which still has degree at most .

33

Multiplication

Multiplication of two polynomials: a different story.

) --------------------------------------------------------------- where (convolution)

Try an example: The multiplication can have degree . If we directly use this formula to multiply two

polynomials, it takes time.

34

Better? YES!

By FFT: We can do it in time . FFT: Fast Fourier Transform

35

What determines/represents a polynomial? Let’s switch to degree-() for later simplicity. We already used coefficients to represent a

polynomial. Coefficient representation

Other ways? Yes: By giving values on (distinct) points.

Point-value representation. [Fact] A degree-() polynomial is uniquely determined

by values on distinct points.

36

Reason

We have coefficients to determine

We know values on distinct points.

How to get the coefficients? Just solve the system of equations:

37

In matrix form:

The matrix is Vandermonde matrix, which is invertible. Thus it has a unique solution for the coefficients .

unknowns

38

Advantage of point-value representation? It makes the multiplication extremely easy:

Though is hard For any fixed point , is simply multiplication of two

numbers. So given

and , it’s really easy to get

. time.

Thus we have the following interesting idea for polynomial multiplication…

39

Go to an easy world and come back HK used to have many industries. Later found mainland has less expensive labor. So:

moved the companies to mainland, did the work there, and then shipped the products back to HK to sell

This is worth doing if: it’s cheaper in mainland, and the traveling/shipping is not expensive either. which turned out to be true.

40

In our case

We need to investigate: the cost of traveling. Both way.

41

Traveling

From coefficient representation to point-value representation: Evaluate two polynomials both on points.

--- evaluation. From point-value representation to coefficient

representation: Compute one polynomial back from point values.

--- interpolation. Both can be done in time.

42

Evaluation

One point?

Directly by the above: multiplications Horner’s rule:

--- multiplications.

43

Number of points

How many points we need to evaluate on? Since we later want to reconstruct the

product polynomial , which has degree . So (point,value) pairs are enough to

recover . We’ll evaluate and on points , and get .

points are enough. We use for convenience. Then recover from .

44

Now evaluation on one point needs time, so evaluations on points need time.

Too bad. We want . Important fact: cost(evaluations on points) can be

cheaper than cost(evaluation on point). If we choose the points carefully.

And it turns out that the chosen points also make the (later) interpolation easier.

This powerful tool is called Fourier Transform.

Complex roots of unity

1 has a complex root , satisfing . Discrete Fourier Transform (DFT):

where Try now.

Note: evaluates polynomial on .

So our evaluation task is just DFT.

46

All the essences are here…

We want to evaluate and on , , …, . Suppose for simplicity that is a power of . For , define two new polynomials:

Then Divide and Conquer.

So we can evaluate by evaluating and on .

47

Distinct points

Note:

are not all distinct.

Only values, each repeating twice!

48

So we evaluate only points (instead of ). Recursion:

: To compute for . Rewriting recursion:

. Applying master theorem: . Since , the cost of evaluating is , as claimed.

49

Interpolation

How about interpolation? i.e., to get the coefficients by point values.

Almost the same process!

.

50

DFT matrix: . What’s the inverse of the matrix ?

Pretty much the same matrixreplace with .

…and then divide the whole matrix by for normalization.

51

Why? You can directly check by multiplying the two

matrices and get (the identity matrix). e.g.

52

Or

DFT matrix is unitary. Namely

: transpose. : complex conjugate DFT is symmetric: . So taking complex conjugate () gives

inverse.

53

Summary

Divide and conquer is a general method to design algorithms.

Master theorem to compute the complexity. Several examples.

Merge sort Selection Matrix multiplication FFT