More on Intractability Knapsack Problem Wednesday, August 5 th 1.

1

More on Intractability

Knapsack Problem

Wednesday, August 5th

2

Outline For Today

1. Knapsack Pseudo-Poly-time DP 1


3. Knapsack Greedy ½-Approximation Alg

4. Knapsack Fully Poly-time Approx. Scheme (FPTAS)

Recap: Classes P, NP

3

Given a computational problem C

P: C is ∈ P (polynomial-time solvable) if ∃an

algorithm solving C with O(nk) run-time, for some

constant k.

where n is the input length in bits

NP (or brute-force solvable): C ∈ NP if:

1. Correct solutions have polynomial length.

2. Claimed solutions are verifiable in poly-time.

Recap: NP-completeness

4

NP

C1

C*: NP-complete

C3 C4C5

C6

Ck

C2

C* is as hard as any NP problem!

Recap: History of NP-completeness

5

NP

C2

SAT

C3 C4C5

Ck

C6

C1

K1

K2… K20

K21

1971: Cook-Levin1972: Karp

Recap: History of NP-completeness

6

NP

C2

SAT

C3 C4C5

Ck

C6

C1

K1

K2… K20

K21

NP-complete

Since 1972

Recap: Showing C** is NP-complete

7

NP

C2

C*: NP-complete

C3 C4C5

Ck

C6

C*

*

If we can solve C** efficiently => we solve C*

efficiently

=> we solve all NP problems efficiently

C** is NP-

complete!

Recap: Two Very Important Skills

8

1. Recognizing NP-complete problems.

2. Learning The Right Methods of Approaching

NP-complete Problems

Recap: Approaching NP-complete Problems

9

Option 1: Focus to special-case inputs.

Option 2: Find an approximate answer.

Option 3: Be exponential time but better than

brute-force search.

Option 4: Heuristics: fast algorithms that are not

always correct (or even approximate)

Option 5: Mix some of these options

10

Outline For Today





11

Knapsack (Sec 6.4, 11.8)

Input:

n items

values for items v1, …, vn ≥ 0

sizes for items w1, …, wn ≥ 0

knapsack capacity W ≥ 0

Output: subset S ⊆ 1, 2, …, n items s.t.Fact: Knapsack is NP-

complete.3SAT≤p KNAPSACK

SUBSET-SUM ≤p

KNAPSACK

12

Knapsack Examplev1 = 2.2

w1 = 1.5

v2 = 4w2 = 3

v4 = 3w4 = 4.6

W=7.8

v3 = 2

w3 = 3

13

Knapsack Examplev1 = 2.2

w1 = 1.5

v2 = 4w2 = 3

v4 = 3w4 = 4.6

W=7.8

v3 = 2

w3 = 3

v2 = 4w2 = 3

v3 = 2w3 = 3

v1 = 2.2

w1 = 1.5

OPT = 4+2+2.2=8.2

14

Restrict wi and W to be Integers

Input:

n items


sizes for items w1, …, wn ≥ 0 & wi are **INTEGERS**

knapsack capacity W ≥ 0 & W is an **INTEGER**

Output: subset S ⊆ 1, 2, …, n items s.t.

Recap: Recipe of a DP Algorithms

15

1. Identify small # of subproblems

2. Quickly + correctly solve “larger” subproblems

given solutions to smaller ones

3. After solving all subproblems, can quickly

compute final solution

16

Knapsack DP Algorithm 1

Order the n items in arbitrary order: 1, 2, …, n.

Consider the optimal solution S*

A Claim that Doesn’t Require A Proof:

(1) n ∉ S* or (2) n ∈ S*

17

Case 1: n ∉ S*

Q: What can we assert about S* for items 1,

…, n-1?

A: S* is opt. for items 1, …, n-1 and capacity

W.

Proof: Assume ∃better S** w/ cap. W for 1, …,

n-1

S** is feasible for 1, …, n and better than

S*

Which would contradict S*’s optimality

Q.E.D.

18

Case 2: n ∈ S*

Q: What can we assert about S*-n for items

1, …, n-1?

A: S*-n is opt. for items 1, …, n-1 and cap.

W-wn.

Pf: Assume ∃better S** w/ cap ≤ W-wn for 1,

…, n-1

S** ∪n has capacity ≤ W

S** ∪n is feasible for 1, …, n and better

than S*

Which would contradict S*’s optimality

Q.E.D.

19

What Are The Subproblems?

K(i, c): opt. knapsack for the first i items and cap

c.

Q: How many subproblems are there?

A: n*W

K(i, c) =

max

K(i-1, c)

K(i-1, c - w_i) +

vi

Knapsack DP Algorithm 1 Pseudocode

procedure DP-Knapsack-1(n, W): Base Cases: A[0][i] = 0

for i = 1,2,…,n: for c = 1, …, W:

A[i][c] = maxA[i-1][c], A[i-1][c-wi]+vi

return A[n][W]20

K(i, c) =

max

K(i-1, c)

K(i-1, c - w_i)

Run-time

21

Runtime: O(nW)

Brute-Force Search: Ω(2n)

Observation: This is polynomial in n and W.

Q: Why does this not prove P=NP?

A: B/c we’re still exponential in input size.

Input

22

Input size: # bits (key strokes) to represent the

problem

n weights, values (n * (log of max weight and value))

capacity => log(W) bits.

Note: W is exponential in log(W)

w1 v1 w2 v2 … … wn vn W

Pseudo-polynomial Time Algorithm

23

An algorithm that’s polynomial in the

numeric values of the inputs but not the

# bits to represent it.

Ex: O(nW) is pseudo-polynomial

Interpretation: If we fix W to an integer

value

=> Knapsack is tractable

Called “Fixed-Parameter Tractable”

Problem

Summary of Knapsack DP Alg 1

24

1. Took Knapsack, which is NP-complete.

2. Restricted to inputs with integer wi and W

3. Got a pseudo-poly-time algorithm DP

algorithm (exponential but better than brute-

force search)

4. Further fixing W yields a full poly-time

algorithm

Recap: Approaching NP-complete Problems

25

Option 1: Focus to special-case inputs.

Option 2: Find an approximate answer.

Option 3: Be exponential time but better than

brute-force search.

Option 4: Heuristics: fast algorithms that are not

always correct (or even approximate)

Option 5: Mix some of these options

Important Note To Keep In Mind

26

When attacking NP-complete problems,

if you stick with always correct algorithms

(i.e. you don’t pick option 2 and try to

approximate)

you necessarily have to be exponential time.

However can get non-trivial speed-ups

if some aspect of the problem is

small.

27

Outline For Today





28

Now Restrict vi to be Integers

Input:

n items

values for items v1, …, vn ≥ 0 & vi are **INTEGERS**

let v* = maxi vi , and V = v1 + v2 + … + vn ≤ nv*




29

A Different DP Algorithm

DP Algorithm 1 asked:

What is the maximum value we can pack into

at most X capacity given the first k items?

We can also ask:

What is the minimum capacity needed to

pack

at least Y value into the knapsack given the

first k items?

Subproblems For Approach 2

30

M(i, v): min capacity needed to pack value v from

the first i items.

Q: How many subproblems are there?

A: n*V≤n(nv*)=n2v*

Runtime is O(n2v*)

M(i, v) =

min

M(i-1,

v)M(i-1, v – v_i) + wi

Knapsack DP Algorithm 2 Pseudocode

procedure DP-Knapsack-2(n, V = v1 + … + vn):

Base Cases: A[0][0] = 0, A[0][j] = +∞ for i = 1,2,…,n:

for v = 1, …, V:A[i][v] = minA[i-1][v], A[i-1]

[v-vi]+wireturn max v s.t. A[n][v] ≤ W

31

M(i, v) =

min

M(i-1,

v)M(i-1, v - v_i) + wi

Knapsack DP Algorithm Output

32

1 2 3 … n

1 … … … … … … 3

2 … … … … … …

3 … … … … … …

… … … … … … …

… … … … … … …

… … … … … … …

… … … … … … …

V … … … … … …

Min capacity needed to pack a value of 1


33

1 2 3 … n

1 … … … … … … 3

2 … … … … … … 3

3 … … … … … …

… … … … … … …

… … … … … … …

… … … … … … …

… … … … … … …

V … … … … … …



34

1 2 3 … n

1 … … … … … … 3

2 … … … … … … 3

3 … … … … … … 5

… … … … … … … …

45 … … … … … … W-1

… … … … … … …

… … … … … … …

V … … … … … …



35

1 2 3 … n

1 … … … … … … 3

2 … … … … … … 3

3 … … … … … … 5

… … … … … … … …

45 … … … … … … W-1

46 … … … … … … W+3

… … … … … … …

V … … … … … …



36

1 2 3 … n

1 … … … … … … 3

2 … … … … … … 3

3 … … … … … … 5

… … … … … … … …

45 … … … … … … W-1

46 … … … … … … W+3

… … … … … … … …

V … … … … … …

Min capacity needed to pack a value of V

3 Exp-time Correct Knapsack Algorithms

37

Brute-Force Search

DP1 DP2

Ω(2n) O(nW) O(n2v*)

Takeaway 1: There is good & bad exponential

times

Takeaway 2: We can get non-trivial speed-

ups if some aspect of the problem is small.

38

Outline For Today





39

Back To Original Knapsack

Input:

n items





Option 2:

Now we’ll give up

correctness.

Recall Scheduling Problem Input: Each jobi has length li AND weight wi

l1, w1Job 1

l2, w2Job 2

ln,wnJob n

…

Output: A schedule of the jobs on a processor

s.t:

is minimum over all possible n! schedules.

weighted completion time of

job i

41

A Possible Greedy Approach

Similar to Greedy Weighted Scheduling (Lecture 5)

If weights are the same, put higher value items first

If values are the same, put lighter items first

Greedy Algorithm:

1. Combine vi and wi into a single score vi / wi:

2. Sort items in increasing combined score

Assume w.l.o.g.: v1/w1 ≥ v2/w2 ≥ … ≥ vn/wn

3. Pack until can’t pack anymore

42

Example 1v1 = 5w1 =

1

v2 = 4

w2 = 2

v3 = 3

w3 = 3

W=

5

43

Example 1

v1 = 5w1 =

1

v2 = 4

w2 = 2

v3 = 3

w3 = 3

W=

5

44

Example 1

v1 = 5w1 =

1

v2 = 4

w2 = 2

v3 = 3

w3 = 3

3rd item does not fit

Output: 4 + 5 = 9

Found the optimal

Q: What can go

wrong?

W=

5

45

Bad Examplev1 = 2w1 =

1

v2=100

w2=100

W=100

46

Bad Example

v1 = 2w1 =

1

v2=100

w2=100

Output: 2

Optimal is 100

Can be

arbitrarily far

from optimal.

W=100

47

Simple Fix Fixed-Greedy Algorithm:

1. Combine vi and wi into a single score vi / wi:

2. Sort items in increasing combined score

Assume w.l.o.g.: v1/w1 ≥ v2/w2 ≥ … ≥ vn/wn

3. Pack until can’t pack anymore

4. Return Y = max step 3, max-value item

(assume all items have weight ≤ W)

Claim: Y is a ½-approximation to OPT

Runtime: O(nlogn)

48

Thought Experiment

Assume greedy packed first k items

And only x% of item k+1 fits into our

knapsack

Suppose we can slice x% of k

Fill in the rest of out knapsack

And get x% of vk.

49

Example of Slicing

v1 = 2w1 =

1

v2=100

w2=100

W=100

Slice 99%

of item 2

50

Example of Slicing

v1 = 2w1 =

1

v2=1w2=1

W=100

v2=99

w2=99

Claim: This

“Fractional”

Solution is better

than OPT.

51

Proof Sketch (Fill in the Gaps As Exercise)

Same

Take arbitrary solution S

Assume they differ in l units

Greedy picks those l units

from highest value per

weight (plus might have

extra stuff)

value(l units of greedy) ≥

value(l units of S)

Greedy ≥ S

Greedy

Fractional

Any

Solution

S

Same

Different

Different

52

Proof of ½-Approximation

Assume greedy packs first k items and gets stuck.

Fixed-Greedy returns:

maxA=first k, or max-value item

≥ maxA=first k, B=value of k+1st item

Fractional-Greedy: A + fraction of B

maxA, B >= ½ (A+B)

Fixed-Greedy ≥ ½ Fractional-Greedy ≥ ½ OPT

Q.E.D

53

Outline For Today





54

Back To Integer Values Input:

n items

values for items v1, …, vn ≥ 0 (**INTEGERS**)

let v* = maxi vi , and V = v1 + v2 + … + vn ≤

nv*




55

Very Ambitious Goal

KnapsackAlgorithm

I: (wi, vi, W)

accuracy ε (say 0.01)

≥ (1-ε)*OPT

(1-ε)-approx

**Catch: Run-time will grow as ε

decreases**Upshot

Instead of I solve incorrect input I’(wi, ṽi, W)

But solve I’ optimally with DP2

Argue opt for I’ is (1-ε) close to opt for I

56

Recall DP2

Pseudo poly-time DP Algorithm

Run-time: O(n2v*)

Performs badly only when values are very

large.Items Values

1 1005380100001

2 1421480174001

3 5801342740012

4 9925141920410

… …

Items Values

1 10053801

2 14214801

3 58013427

4 99251419

… …

Idea: Remove Low Order Bits.

57

Rounding Algorithm

Q1: How small should m be to satisfy ε

accuracy?

Q2: What is the running time of the algorithm

once we pick the the appropriate m?

procedure Knapsack-FPTAS(wi, vi, W, ε):

round each vi down to nearest mult. of

m=f(ε)

let ṽi = rounded(vi)/m,

return DP2(wi, ṽi, W)

58

Accuracy Analysis (1)

vixi=mṽi ≤ vixi=mṽi ≤ vi ≤ yi=m(ṽi + 1)

(1)(2)

Let S be our solution, and S* is optimal

solution

Hint: Weights were NOT perturbed.

Goal:Goal is to say we’re not that far from OPT.

59


Goal:

by (1)

by (2)

xi’s

yi’s

60


error term

This inequality is true for any m.

Q: What should m be so that we’re ε off?

A: Pick m s.t.

61


Picking satisfies the inequality.

62

Run-time Analysis

∀elements i, we have:

DP2’s run-time was: O(n2v*)

When inputs are ṽi each ṽi ≤ n/ε

So run-time is: O(n3/ε)

63

Key Takeaway

We took an NP-complete problem.

We are approximating it to arbitrary

precision.

And in poly-time for each precision rate!

64

Next Week

More Algorithms For Intractability

&

What’s Beyond CS 161

More on Intractability Knapsack Problem Wednesday, August 5 th 1.

Documents

items w

knapsack capacity w

np problems

np c2c2 c2c2 c

history of np

np c1c1 c1c1 c

np complete problems

bits np