Counting Independent Sets using BP for Sparse Graphs ...

Counting Independent Sets using BP for Sparse

Graphs

Computing the explicit bound of loop series

Michael Chertkov1 Devavrat Shah2 Jinwoo Shin2

1Theory Division, LANL2LIDS, Massachusetts Institute of Technology

October 14, 2008 DIMACS, Rutgers

IntroductionOur ResultsConclusion

Counting Independent Set

Definition (Independent set)

For a given graph G = (V ,E ) with |V | = n, S ⊂ 2V is anindependent set if ∀i , j ∈ S , (i , j) /∈ E .

Counting the number Z of independent sets

Valiant 1979 - #P-complete

Weitz 2006, Bandyopadhya, Gamarnik 2006 - (1 + ε)approximation algorithm for Z (or lnZ ) when a max-degreed ≤ 5.

Bandyopadhya, Gamarnik 2006 - when G is a randomd-regular graph and d ≤ 5, ∃ a constant αd s.t.

1

nlnZ −→

n→∞αd w.h.p.

Jinwoo Shin, MIT Counting Independent Sets using BP for Sparse Graphs


Outline

1 IntroductionSummary of ResultsBelief PropagationLoop Series

2 Our ResultsResult I: Loop Series for large girth graphsResult II: Loop Series for 3-random regular graphs

3 Conclusion



Summary of ResultsBelief PropagationLoop Series

Our Results

Result I

Let ZBP be the value BP estimated for Z . When G has a largegirth g > cd log2 n with cd = 8d ,

Z = ZBP

[

1± O

(

1

n

)]

.




Our Results

Result I

Let ZBP be the value BP estimated for Z . When G has a largegirth g > cd log2 n with cd = 8d ,

Z = ZBP

[

1± O

(

1

n

)]

.

If Cycle Double Cover Conjecture (Szekeres, Seymour 1970’) is true,cd = 4d .




Our Results

Result II

If Shortest Cycle Cover Conjecture (Alon, Tarsi 1985) is true andG is a random 3-regular graph with n vertices,

lnZ = α3n ± O(1) w.h.p,

where α3 = ln 1.545 . . . .




Our Results

Result II


lnZ = α3n ± O(1) w.h.p,

where α3 = ln 1.545 . . . .

α3 is obtained by solving the BP equation.

Recall Bandyopadhya, Gamarnik 2006 implies

lnZ = α3n ± o(n) w.h.p.




Our Results

Result II


lnZ = α3n ± O(1) w.h.p,

where α3 = ln 1.545 . . . .




O(1)-error is unexpected in Stat. Physics.




Our Results

Result II


lnZ = α3n ± O(1) w.h.p,

where α3 = ln 1.545 . . . .




O(1)-error is unexpected in Stat. Physics.

Jamshy and Tarsi 1992 - SCCC implies CDCC.




Belief Propagation

Algorithm for counting # independent sets

1. Initialize: mu→v (0)← 12 .

2. Update:

mu→v (t + 1)←1

1 +∏

w∈N (u)\v mw→u(t).




Belief Propagation



2. Update:

mu→v (t + 1)←1

1 +∏


0.50

0.50

0.50

0.50

0.50

0.50




Belief Propagation



2. Update:

mu→v (t + 1)←1

1 +∏


0.66

0.66

0.66

0.66

0.66

0.66




Belief Propagation



2. Update:

mu→v (t + 1)←1

1 +∏


0.60

0.60

0.60

0.60

0.60

0.60




Belief Propagation



2. Update:

mu→v (t + 1)←1

1 +∏


0.63

0.63

0.63

0.63

0.63

0.63




Belief Propagation



2. Update:

mu→v (t + 1)←1

1 +∏


0.62

0.62

0.62

0.62

0.62

0.62




BP Estimations

Marginal probability

τv(1)

τv(0)=

∏

w∈N (v)

mw→v τv(0) + τv (1) = 1

τu,v (0, 1) = τv (1) τu,v (1, 0) = τu(1) τu,v (1, 1) = 0

τu,v (0, 0) = 1− τv (1)− τu(1)

Partition function Z

lnZBP =∑

v∈V

H(τv )−∑

(u,v)∈E

I (τv : τu)

=∑

v∈V

(−xv ln xv + (dv − 1)(1− xv ) ln(1− xv ))

−∑

(u,v)∈E

(1− xu − xv ) ln(1− xu − xv ) (xv = τv (1))




Quality of BP: Loop Series

Convergence

BP fixed points always exist from the Brouwer fixed point theorem.

The convergence to fixed points is not guaranteed in loopy graphs.

Fixed points can be achievable via Bethe variational method.




Quality of BP: Loop Series

Convergence

BP fixed points always exist from the Brouwer fixed point theorem.

The convergence to fixed points is not guaranteed in loopy graphs.

Fixed points can be achievable via Bethe variational method.

Quality of Estimation: Loop Series (Chernyak, Chertkov 2006)

Z = ZBP

1 +∑

∅6=F⊂E

w(F )

,

where

w(F ) = (−1)|F |∏

v∈VF

τv (1)

[

1 + (−1)dF (v)

(

τv (1)

τv (0)

)dF (v)−1]

.

If ∃v with dF (v) = 1, then w(F ) = 0.

If ∄v with dF (v) = 1, F is called the generalized loop.




Recent works on Loop Series

Sudderth, Wainwright, Willsky 2007 - provide the relation ofthe loop series to the ”tree reparametrization” concept.

Chandrasekhar, Gamarnik and Shah 2008 - provide boundsbetween lnZ and lnZBP for a class of MRF using loop series.




Example: Loop Series of a triangle K3

Calculating ZBP

Solving BP equations- 6 variables (mu→v ) and 6 equations mu→v = 1

1+∏

w∈N (u)\v mw→u.

- mu→v = x , where x = 11+x

= 0.618 . . . .

τv (1)τv (0)

= x2 = 0.382 → τv (1) = 0.276.

ZBP = 4.24.





Calculating ZBP


1+∏

w∈N (u)\v mw→u.


= 0.618 . . . .

τv (1)τv (0)

= x2 = 0.382 → τv (1) = 0.276.

ZBP = 4.24.

Calculating 1 +∑

w(F )

K3 itself is only a generalized loop F .

w(F ) = −∏

v∈VFτv (1)

[

1 + τv (1)τv (0)

]

= −x6 = −0.056





Calculating ZBP


1+∏

w∈N (u)\v mw→u.


= 0.618 . . . .

τv (1)τv (0)

= x2 = 0.382 → τv (1) = 0.276.

ZBP = 4.24.

Calculating 1 +∑

w(F )


w(F ) = −∏

v∈VFτv (1)

[

1 + τv (1)τv (0)

]

= −x6 = −0.056

Therefore, Z = ZBP(1 +∑

w(F )) = 4.24(1 − 0.056) = 4.





Calculating ZBP


1+∏

w∈N (u)\v mw→u.


= 0.618 . . . .

τv (1)τv (0)

= x2 = 0.382 → τv (1) = 0.276.

ZBP = 4.24.

Calculating 1 +∑

w(F )


w(F ) = −∏

v∈VFτv (1)

[

1 + τv (1)τv (0)

]

= −x6 = −0.056

Therefore, Z = ZBP(1 +∑

w(F )) = 4.24(1 − 0.056) = 4.

In general, 1 +∑

w(F ) is hard to compute since there are exponentiallymany generalized loops!



Result I: Loop Series for large girth graphsResult II: Loop Series for 3-random regular graphs

Result I: Loop Series for large girth graphs

Theorem

If the girth of G is greater than cd log2 n with cd = 8d,

∑

∅6=F⊂E

|w(F )| = O

(

1

n

)

.

Corollary

If the girth of G is greater than cd log2 n,

Z = ZBP

[

1± O

(

1

n

)]

.




Result I: Main Issues

How can we bound∑

|w(F )|?

Issues

What is |w(F )| for a generalized loop F of size k?

How many generalized loops of size k?





How can we bound∑

|w(F )|?

Issues


Want |w(F )| < βk with small β < 1.Need to analyze BP fixed points.






How can we bound∑

|w(F )|?

Issues




Want the number < γk with small γ > 1.Need to count efficiently.





How can we bound∑

|w(F )|?

Issues




Want the number < γk with small γ > 1.Need to count efficiently.

Main observation

If βγ < 1,∑

|w(F )| becomes a geometric decaying series startingfrom the girth.




Result I: Computing Strategy using Apples

Definition (Apple)

A connected subgraph C of G is an apple if one of the followingssatisfies.

C is a cycle.

C is the union of a cycle and a line.




Result I: Computing Strategy using Apples

Definition (Apple)

A connected subgraph C of G is an apple if one of the followingssatisfies.

C is a cycle.

C is the union of a cycle and a line.

Strategy for bounding the loop series

1. Decompose the sum 1 +∑

|w(F )| into the product ofapples-terms.

1 +∑

F

|w(F )|?=

∏

C

(1 + |w(C )|) .

2. Analyze the number and weights of apples.




Result I: Decompose Sum into Product

Assumption

|w(C )| < βk for a apple C of size k

The number of apples of size k is at most γk .





Assumption



How can we bound∑

|w(F )| using this information?

If F = ∪Ci and Ci are disjoint, w(F ) =∏

i w(Ci ).





Assumption



How can we bound∑



i w(Ci ).

If F = ∪Ci with apple-vertex degree T= maxv∈F |{Ci : v ∈ Ci}|,

|w(F )| <∏

i

|w(Ci )|1T .





Assumption



How can we bound∑



i w(Ci ).


|w(F )| <∏

i

|w(Ci )|1T .

If any F has its decomposition {Ci} with apple-vertex degree ≤ T ,

1 +∑

F

|w(F )| <∏

C

(

1 + |w(C)|1T

)

< exp

[

∑

k

(γβ1T )k

]

.





Assumption



How can we bound∑



i w(Ci ).


|w(F )| <∏

i

|w(Ci )|1T .

If any F has its decomposition {Ci} with apple-vertex degree ≤ T ,

1 +∑

F

|w(F )| <∏

C

(

1 + |w(C)|1T

)

< exp

[

∑

k

(γβ1T )k

]

. Want γβ1T < 1 !!




Result I: Analysis of Apples

The weight w(C ) of apples of size k

Easy to analyze due to the simple structure of apples.

We obtained |w(C )| < βk with β = 12 .








The number of apples of size k

Each apple of size k corresponds to a k-level leaf of the selfavoiding tree.

The number of apples of size k is at most γk where γ is theexpansion rate of the tree.

As the girth grows, γ → 1.








The number of apples of size k

Each apple of size k corresponds to a k-level leaf of the selfavoiding tree.

The number of apples of size k is at most γk where γ is theexpansion rate of the tree.

As the girth grows, γ → 1.

Recall we want γβ1T < 1, hence if we find a constant T , we are

done!




Result I: Analysis of Decomposition Quality

Theorem

For any generalized loop F , there exists its decomposition into appleswith apple-vertex degree T ≤ 4d.





Theorem


Theorem (Bermond, Jackson and Jaeger 1983)

For every biconnected (bridgeless) graph, there exists a list of cycles sothat every edge is contained in exactly four cycles of the list.

The case of two is called the Cycle Double Cover Conjecture.





Theorem


Theorem (Bermond, Jackson and Jaeger 1983)

For every biconnected (bridgeless) graph, there exists a list of cycles sothat every edge is contained in exactly four cycles of the list.

The case of two is called the Cycle Double Cover Conjecture.

Decomposition Strategy

1 Decompose F into biconnected components with connecting edges.

2 From Lemma, cover each component with cycles.

3 Cover the remaining connecting edges by making cycles to be apples.

4 Apple-edge degree 4 leads to apple-vertex degree 4d .




Result II: Loop Series for 3-random regular graphs

Theorem

If the Shortest Cycle Cover Conjecture is true and G is a random3-regular graph with n vertices, there exists a finite-valued functionf : (0, 1) → R

+ such that

lnZ = lnZBP ± f (ε) with probability 1− ε,

where lnZBP = α3n and α3 = ln 1.545 . . . .

α3 is obtained from the homogeneous solution of BP equation.




Result II: Loop Series for 3-random regular graphs

Theorem

If the Shortest Cycle Cover Conjecture is true and G is a random3-regular graph with n vertices, there exists a finite-valued functionf : (0, 1) → R

+ such that

lnZ = lnZBP ± f (ε) with probability 1− ε,

where lnZBP = α3n and α3 = ln 1.545 . . . .

α3 is obtained from the homogeneous solution of BP equation.

Shortest Cycle Cover Conjecture (Alon and Tarsi 1985)

The edges of every biconnected graph with m edges can becovered by cycles of total length at most 7m/5 = 1.4m.




Result II: Upper bound

Properties of Random 3-regular graphs

The number of cycles of size k is ≤ 2k/k.

The number of apples of size k is . 2k , hence γ = 2.

→ If SCCC is true, γβ1T ≈ 0.96 < 1.

→∑

|w(F )| = O(1).(Recall

∑

|w(F )| = O( 1n) in the large girth case.)




Result II: Upper bound

Properties of Random 3-regular graphs

The number of cycles of size k is ≤ 2k/k.

The number of apples of size k is . 2k , hence γ = 2.

→ If SCCC is true, γβ1T ≈ 0.96 < 1.

→∑

|w(F )| = O(1).(Recall

∑

|w(F )| = O( 1n) in the large girth case.)

Upper bound of lnZ

From loop calculus,

lnZ = lnZBP + ln(1 +∑

w(F ))

≤ lnZBP + ln(1 +∑

|w(F )|)

= lnZBP + O(1).

How can we get thelower bound?




Result II: Lower bound

Why was it possible in the large girth case?

Since∑

|w(F )| = O( 1n) < 0.5,

Z = ZBP

(

1 +∑

w(F ))

> ZBP

(

1−∑

|w(F )|)

> ZBP/2.






Since∑

|w(F )| = O( 1n) < 0.5,

Z = ZBP

(

1 +∑

w(F ))

> ZBP

(

1−∑

|w(F )|)

> ZBP/2.

However, now∑

|w(F )| = O(1) > 1 !!






Since∑

|w(F )| = O( 1n) < 0.5,

Z = ZBP

(

1 +∑

w(F ))

> ZBP

(

1−∑

|w(F )|)

> ZBP/2.

However, now∑

|w(F )| = O(1) > 1 !!

Main idea

Find a new graph G ′ such that∑

|w ′(F )| < 0.5 but lnZ ≈ lnZ ′

and lnZBP ≈ lnZ ′BP .




Result II: Construction of G ′

Basic Idea

Reduce∑

|w(F )|

ln ZBP ≈ ln Z ′BP

ln Z ≈ ln Z ′





Basic Idea

Reduce∑

|w(F )| ← Break small cycles.

ln ZBP ≈ ln Z ′BP ← Keep regularity.

ln Z ≈ ln Z ′ ← |V | ≈ |V ′|.





Basic Idea

Reduce∑



ln Z ≈ ln Z ′ ← |V | ≈ |V ′|.

Algorithm to construct G ′

1. G ′ ← G2. While

∑

|w ′(F )| < 0.53. Find the smallest cycle C in G ′

4. Insert Hε on one of the edges of C

(Hε should have a large girth.)





Basic Idea

Reduce∑



ln Z ≈ ln Z ′ ← |V | ≈ |V ′|.


1. G ′ ← G2. While

∑








Basic Idea

Reduce∑



ln Z ≈ ln Z ′ ← |V | ≈ |V ′|.


1. G ′ ← G2. While

∑




Lemma

Given ε > 0, there exists a 3-regular large-girth graph Hε such that∑

F⊂Hε

|w(F )| < ε.



Conclusion

Beyond Correlation Decay

The loop series is a new analytic tool.

Techniques to bound the loop seriesGraphs with large girth: Decompose Sum into Product usingapples.Random 3-regular graphs: Graph-Insertion for reducing theloop series.



Conclusion

Beyond Correlation Decay

The loop series is a new analytic tool.

Techniques to bound the loop seriesGraphs with large girth: Decompose Sum into Product usingapples.Random 3-regular graphs: Graph-Insertion for reducing theloop series.

Future works

Other graphical models - k-SAT, coloring, Ising, etc.

Find bigger regimes with more errors:

lnZ = lnZBP ± o(1) −→how conditions go?

lnZ = lnZBP ± o(n).

Algorithmic implementation for reducing the loop series.



Result II: Analysis of Algorithm

In each iteration,∑

|w ′(F )| keeps decreasing if we choose a small ε.⇒ Terminate in finite K steps since

∑

|w(F )| = O(1).

G ′ becomes a bigger 3-regular graph with |Hε| more vertices.⇒ lnZ ′, lnZ ′

BP increases by at most |Hε| ln 2, |Hε|α3.






∑

|w(F )| = O(1).



Finally,

lnZ ≤ lnZ ′ ≤ lnZ + K |Hε| ln 2

lnZBP ≤ lnZ ′BP ≤ lnZBP + K |Hε|α3.






∑

|w(F )| = O(1).



Finally,

lnZ ≤ lnZ ′ ≤ lnZ + f (ε)

lnZBP ≤ lnZ ′BP ≤ lnZBP + f (ε).






∑

|w(F )| = O(1).



Finally,

lnZ ≤ lnZ ′ ≤ lnZ + f (ε)

lnZBP ≤ lnZ ′BP ≤ lnZBP + f (ε).

Since∑

|w ′(F )| < 0.5,

lnZ ′ > lnZ ′BP − ln 2

⇒ lnZ > lnZBP − f (ε) − ln 2.Jinwoo Shin, MIT Counting Independent Sets using BP for Sparse Graphs


Result II: Proof of Lemma

Theorem (McKay, Wormald, Wysocka 2004)

Xr= # cycles of length r in random 3-regular graph.

E [Xr ] ≤ 2r/r = µr

→∑

|w(F )| .∑

r Xr (0.48)r :=∑

r ar with E [ar ] . (0.96)r .

X3, X4, . . . , X 13 log n are independent poisson r.v. with mean µr .

→ Pr [X3 = X4 = · · · = Xg = 0] ≈ e−eg

.

Set g = log log log n.

Probabilistic Analysis

A1 =∑

r<g ar

A2 =∑

g≤r< 13 log n ar

A3 =∑

g≥ 13 log n ar

E1 : A1 = 0 w.p. 1log n

E2 : A2 ≤ 2E [A2] w.p. 12

E3 : A3 ≤ (3 log n)E [A3] w.p. 1− 13 log n

Pr[E1&E2&E3] > 0.

A1 + A2 + A3 ≤ 2E [A2] + (3 log n)E [A3]→ 0.Jinwoo Shin, MIT Counting Independent Sets using BP for Sparse Graphs

Counting Independent Sets using BP for Sparse Graphs ...

Documents

mitcounting independent

sparse graphs

bp equation

value bp

n vertices

nn jinwoo shin

impliesln z

ln z dw