Approximability & Proof Complexity Ryan O’Donnell Carnegie Mellon.

Approximability& Proof Complexity

Ryan O’DonnellCarnegie Mellon

Approximability& Proof Complexity

Ryan O’DonnellCarnegie Mellon

of optimization problems

Minimum Vertex-Cover

MIN-VC(G)= min {|S| : S ⊆ V, S touches all edges of E}

Minimum Vertex-Cover

MIN-VC(G)= min {|S| : S ⊆ V, S touches all edges of E}

“2-approximating” Min-VC

• Choose any maximal matching M ⊆ E

• MIN-VC(G) ≥ |M|

• Let S = {all endpoints in M}.It’s a vertex-cover (why?) satisfying

|S| = 2|M| ≤ 2MIN-VC(G)

“Factor 2-certifying” Min-VC

• Choose any maximal matching M ⊆ E

• Output “MIN-VC(G) ≥ |M|”

1. Output bound is always correct.2. Bound is always within factor α of truth.

A “factor α-certification” algorithm:

Linear programming (LP) relaxation

k = minimize: ∑v∈V Xv

subject to: 0 ≤ Xv ≤ 1 for all v∈V

Xu + Xv ≥ 1 for all (u,v)∈E

Output “MIN-VC(G) ≥ k”

Matching algorithm, LP algorithm are bothfactor 2-certification algorithms.

Are they also 1.99-certification algorithms?

No. MIN-VC(Kn) = n−1

maximal |M| = n/2

LP value = n/2

Kn

Matching algorithm, LP algorithm are bothfactor 2-certification algorithms.

Are they also 1.99-certification algorithms?

No. MIN-VC(Kn) = n−1

maximal |M| = n/2

LP value = n/2

Is there any poly-time 1.99-certification alg?

Kn

Is there any poly-time 1.99-certification alg?

We don’t know. Best we know is:

∃ 1.36-certification alg (unless P=NP)[Dinur-Safra’02]

Approximability &Proof Complexity

Resolution

Refutes statements encoded…

by boolean disjunctions

Cutting planes by integer inequalities

Nullstellensatz/Polynomial Calculus

[BIKPP’96,CEI’96]

by polynomial equations

Positivstellensatz/Sum-of-Squares (SOS)

[Grigoriev-Vorobjov’99]

by polynomial inequalities

ZFC (“Frege”) in ordinary math language

is infeasible if

−1 = Q0 + Q1P1 + Q2P2+ ••• +QmPm

there exist “certifying polynomials” Q0, …, Qm,

each a sum of squares, s.t. we have the identity

Positivstellensatz[Krivine’64,Stengle’73,Schmüdgen’91,Putinar’93,Wörmann’98]

and only if

assuming a mildtechnical condition

“MIN-VC(G) > k”

Xv2 ≥ Xv

Xv2 ≤ Xv

Xu+Xv ≥ 1 for all (u,v)∈E

∑v Xv ≤ k

infeasible

−1 = Q0 + Q1 (k −∑ Xv) + ∑ Quv (Xu+Xv−1) + •••

∃ certifying SOS polynomials Q such that

for all v∈V

⇔

⇔

Positivstellensatz / SOS proof system

Suggested by Grigoriev and Vorobjov in 1999.

The complexity of an SOS proof is the

maximum degree of any QiPi or Q0.

SOSd denotes the proof system

restricted to degree d.

(No longer complete.)

Example proof

Theorem: The following system is infeasible:

{X2 ≤ 1, Y2 ≤ 1, Z2 ≤ 1, XY+YZ+ZX ≤ −2}.

ZFC proof: Let f(X,Y,Z) = XY+YZ+ZX. Suppose X2 ≤ 1; i.e., X ∈ [−1,1]. Since f is linear in X, it’s maximized if X = ±1. Similarly for Y and Z. Suffices to show f(±1,±1,±1) > −2. If all three inputs same, f is 3. If not all three inputs same, f is −1.

Example proof

Theorem: The following system is infeasible:

{X2 ≤ 1, Y2 ≤ 1, Z2 ≤ 1, XY+YZ+ZX ≤ −2}.

SOSd=4 proof:

+

Show SOSd=4 certifies MIN-VC(Kn) ≥ n−1

(i.e., refutes MIN-VC(Kn) ≤ n−2).

Exercise

SOSd is ‘automatizable’!

Theorem: [Lasserre’00,Parrilo’00, cf. N.Shor’87]

If a polynomial inequality system can

be refuted in the SOSd proof system,

the certifying Qi’s can be found in

nO(d) time (using semidefinite programming).

The strongest(?) automatizableproof system that we know

SOSd is stronger than:

Width-d ResolutionDegree-d Nullstellensatz

Basic LP relaxationsBasic SDP relaxations

d/2 rounds of Lovász-Schrijver LP/SDP hierarchyd/2 rounds of Sherali-Adams LP/SDP hierarchy

(Doesn’t seem to be stronger than degree-d Polynomial Calculus.)

A very powerful poly-time algorithmfor Vertex-Cover certification:

Output the largest k ∈ [n] such that

SOSd=1000 certifies “MIN-VC(G) > k”.

Could this be a 1.99-certification algorithm?

I.e., is it true that whenever MIN-VC(G) = β,

∃ degree-1000 Qi’s certifying Min-VC(G) ≥ β/1.99?

Partial history of upper

and lower bounds for SOSd

SOSd upper bounds, 2001-2011

Nothing that we didn’t alreadyknow by other means.

E.g., SDP is a .878-certification alg for Max-Cut

∵ SDP ≤ SOSd=4 ∴ SOSd=4 also .878-certifies Max-Cut

SOSd lower bounds, 1999-2009

[Grigoriev’99]:(indep. [Schoenbeck’08])

Consider a random system of O(n)3-variable equations modulo 2.

With very high probability…

• No assignment sats > 51% of equations

• Unless d = Ω(n), SOSd cannot refute

“the system is totally satisfiable”.

Tseitin Tautologies / 3Lin(mod 2)

[Grigoriev’01]:See also [Laurent’02],

[Cheung’05]

“If n is odd and X1, …, Xn satisfy Xi2 = 1,

then X1 + ••• + Xn cannot be 0.”

Not provable in SOSd

unless d ≥ n+1.

‘Knapsack’

(Essentially equivalent: “MAX-CUT(Kn) ≥ ”)

Open Problem: Give a pleasant proof thatd needs to be at least, say, 6.

(A corollary of the 3Lin(mod 2) lower bound.)(Not rigorously proven, but seems true in all cases.)

[Tulsiani’09] Rule of Thumb

For any factor-α certification problemwhich we know is NP-hard,

there exists instances which require

degree-nΩ(1) SOS proofs.

A very powerful poly-time algorithmfor Vertex-Cover certification:

Output the largest k ∈ [n] such that

SOSd=1000 certifies “MIN-VC(G) > k”.

Could this be a 1.99-certification algorithm?

Integrality Gaps[GK’95] SDP does not 2-certify Min-VC: Frankl-Rödl graphs

[FS’00] SDP does not .879-certify Max-Cut: Noisy-sphere graphs

[KV’05] SDP+∆ does not .879-certify Max-Cut or solve Unique-Games:

KV noisy-hypercube graphs

[DKSV’06] SDP+∆ does not O(1)-certify Sparsest-Cut (Balanced-Separator):

DKSV noisy-hypercube graphs

[KS’09,RS’09] Sherali-Adams+, degree-O(1),

does not .879-certify Max-Cut or solve Unique-Games:

KV noisy-hypercube graphs

[BCGM’11] Sherali-Adams+, degree-6, (and probably degree-O(1))

does not 2-certify Min-VCFrankl-Rödl graphs

These tricky instancesaren’t so hard for SOS!

[BBHKSZ’12]: SOSd=4 solves the KV

Unique-Games instances!

[OZ’13]: SOSd=4 solves the DKSV

Balanced-Separator instances.

[OZ’13]: SOSd=O(1) .95-certifies the KV

Max-Cut instances.

[BBHKSZ’12]: SOSd=4 solves the KV

Unique-Games instances!

[OZ’13]: SOSd=4 solves the DKSV

Balanced-Separator instances.

[DMN’13]: SOSd=O(1) solves the KV

Max-Cut instances.

• [KV’05]: used ZFC to show “MAX-CUT(KV) ≈ 85%”

• [KS’09,RS’09]: SA+d=O(1) only certify “MAX-CUT(KV) ≥ 75%”

• [DMN’13] SOSd certifies “MAX-CUT(KV) ≥ 85% − od(1)”

[BBHKSZ’12]: SOSd=4 solves the KV

Unique-Games instances!

[OZ’13]: SOSd=4 solves the DKSV

Balanced-Separator instances.

[DMN’13]: SOSd=O(1) solves the KV

Max-Cut instances.

[KOTZ’13]: SOSd=O(1) solves “most of”

the Frankl-Rödl Min-VC instances.

The whole result is just that

one particular algorithm does

well on one particular instance?

I have 3 responses.

Response 1: an old joke

Q: Why did the complexity theorist work on algorithms?

A: To get lower bounds on his lower bounds.

We basically no longer know any “hard instances”.

Response 2: Evidence for algorithmic optimism?

[BBHKSZ+’13] points out that as far as we know,

SOSd=4 solves the Unique-Games

problem (i.e., refutes the UGC).

Perhaps SOSd is the killer algorithm

for combinatorial optimization.

Response 3: New proofs

Proving known theorems in restricted proofsystems can lead to new insights and proofs.

[Razborov’93]: New Switching Lemma proof

[BBHKSZ’12, Hypercontractivity insights OZ’13,KOTZ’13]:

[BHM’12,KOTZ’13]: New Frankl-Rödl Thm. proof

[MN’13,DMN’13]: New Maj. is Stablest proof

Let’s take stock

• Approximation Algs ≤ Proof Complexity:

“Is there an efficient algorithm for 100-coloring a 3-colorable graph?”

Let’s take stock

• Approximation Algs ≤ Proof Complexity:

“Given a graph that is not 100-colorable, how hard is it to prove that it’s not 3-colorable?”

• SOSd is a quirky yet strong proof system,

automatizable in time nO(d).

• SOSd=O(1) solves all the trickiest instances

we know of Unique-Games, Max-Cut.

Three closing thoughtsregarding proof complexity

1. Frankl-Rödl graphs and SOS lower bounds

2. The Dynamic SOS proof system

3. My favorite algorithm for Unique-Games

Frankl-Rödl graphs

FRm(γ): V = {0,1}m

E = {(x,y) : Δ(x,y) ≥ (1−γ) m}

[Frankl-Rödl’87]:

MIN-VC(FRm(γ)) ≥ (1−o(1)) 2m

[KOTZ’13]: SOSd=O(1/γ) can prove this.

But perhaps SOSd=O(1) cannot handle

A simpler open problem

Theorem: (a corollary of [Harper’66]’s Vertex-Isoperimetric Inequality)

Let A, B ⊆ {0,1}m satisfy dist(A,B) ≥

Then |A|, |B| aren’t both large:

Conjecture: SOSd=4 cannot prove this.

Dynamic SOS

• Lines of the proof are of form P(X1, …, Xn) ≥ 0.

• From P ≥ 0 and Q ≥ 0 can derive

P + Q ≥ 0 and P • Q ≥ 0.

• Can always derive R2 ≥ 0.

• To refute a system {P1 ≥ 0, …, Pm ≥ 0},

derive −1 ≥ 0. Complexity: max degree of any line

Dynamic SOSFacts [Grigoriev-Hirsch-Pasechnik’01]:

• Dynamic SOSd=3 refutes Knapsack

• Dynamic SOSd=5 refutes any

unsatisfiable 3XOR instanceOpen problem 1 [GHP’01]: Suggest an explicit unsatisfiable boolean

formula which SOSd=O(1) might not refute.

Open problem 2: Give negative evidence re automatizability.

Unique-Games

[Khot’02] conjectured that for the “UG” CSP, it’s NP-hard to distinguish ϵ-satisfiable instances from (1−ϵ)-satisfiable instances.

[BBHKSZ’12]: Perhaps SOSd=4 can actually do it.

⇒ UGC is false (assuming NP ≠ P)

Perhaps SOSd=log(n) can do it.

⇒ UGC is false (assuming NP ⊈ TIME[nlog n])

Why be so concerned about automatizability?

Unique-Games

My favorite UG algorithm:

Given an ϵ-satisfiable instance, nondeterministically guess a poly-lengthZFC proof that instance is ≤ (1−ϵ)-satisfiable.

[Khot’02] conjectured that for the “UG” CSP, it’s NP-hard to distinguish ϵ-satisfiable instances from (1−ϵ)-satisfiable instances.

If this algorithm works, UGC is false.(assuming NP ≠ coNP)

ありがとうございました！ Thank you!