Top Banner
Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity Monotonicity, thinning and discrete versions of the Entropy Power Inequality Joint work with Yaming Yu – see arXiv:0909.0641 Oliver Johnson [email protected] http://www.stats.bris.ac.uk/maotj Statistics Group, University of Bristol 24th June 2010 Oliver Johnson [email protected] Statistics Group, University of Bristol Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar
75

Monotonicity, thinning and discrete versions of the Entropy Power

Feb 11, 2022

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Monotonicity, thinning and discrete versions ofthe Entropy Power Inequality

Joint work with Yaming Yu – see arXiv:0909.0641

Oliver [email protected]

http://www.stats.bris.ac.uk/∼maotj

Statistics Group, University of Bristol

24th June 2010

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 2: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Abstract

I Differential entropy h = −∫

f (x) log f (x)dx has many niceproperties.

I Often Gaussian provides case of equality.I Focus on 3 such properties:

1. Maximum entropy2. Entropy power inequality3. Monotonicity

I Will discuss discrete analogues for discrete entropyH =

∑x p(x) log p(x).

I Infinite divisibility suggests Poisson should be case of equality.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 3: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Abstract

I Differential entropy h = −∫

f (x) log f (x)dx has many niceproperties.

I Often Gaussian provides case of equality.I Focus on 3 such properties:

1. Maximum entropy2. Entropy power inequality3. Monotonicity

I Will discuss discrete analogues for discrete entropyH =

∑x p(x) log p(x).

I Infinite divisibility suggests Poisson should be case of equality.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 4: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Abstract

I Differential entropy h = −∫

f (x) log f (x)dx has many niceproperties.

I Often Gaussian provides case of equality.

I Focus on 3 such properties:

1. Maximum entropy2. Entropy power inequality3. Monotonicity

I Will discuss discrete analogues for discrete entropyH =

∑x p(x) log p(x).

I Infinite divisibility suggests Poisson should be case of equality.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 5: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Abstract

I Differential entropy h = −∫

f (x) log f (x)dx has many niceproperties.

I Often Gaussian provides case of equality.I Focus on 3 such properties:

1. Maximum entropy2. Entropy power inequality3. Monotonicity

I Will discuss discrete analogues for discrete entropyH =

∑x p(x) log p(x).

I Infinite divisibility suggests Poisson should be case of equality.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 6: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Abstract

I Differential entropy h = −∫

f (x) log f (x)dx has many niceproperties.

I Often Gaussian provides case of equality.I Focus on 3 such properties:

1. Maximum entropy

2. Entropy power inequality3. Monotonicity

I Will discuss discrete analogues for discrete entropyH =

∑x p(x) log p(x).

I Infinite divisibility suggests Poisson should be case of equality.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 7: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Abstract

I Differential entropy h = −∫

f (x) log f (x)dx has many niceproperties.

I Often Gaussian provides case of equality.I Focus on 3 such properties:

1. Maximum entropy2. Entropy power inequality

3. Monotonicity

I Will discuss discrete analogues for discrete entropyH =

∑x p(x) log p(x).

I Infinite divisibility suggests Poisson should be case of equality.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 8: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Abstract

I Differential entropy h = −∫

f (x) log f (x)dx has many niceproperties.

I Often Gaussian provides case of equality.I Focus on 3 such properties:

1. Maximum entropy2. Entropy power inequality3. Monotonicity

I Will discuss discrete analogues for discrete entropyH =

∑x p(x) log p(x).

I Infinite divisibility suggests Poisson should be case of equality.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 9: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Abstract

I Differential entropy h = −∫

f (x) log f (x)dx has many niceproperties.

I Often Gaussian provides case of equality.I Focus on 3 such properties:

1. Maximum entropy2. Entropy power inequality3. Monotonicity

I Will discuss discrete analogues for discrete entropyH =

∑x p(x) log p(x).

I Infinite divisibility suggests Poisson should be case of equality.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 10: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Abstract

I Differential entropy h = −∫

f (x) log f (x)dx has many niceproperties.

I Often Gaussian provides case of equality.I Focus on 3 such properties:

1. Maximum entropy2. Entropy power inequality3. Monotonicity

I Will discuss discrete analogues for discrete entropyH =

∑x p(x) log p(x).

I Infinite divisibility suggests Poisson should be case of equality.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 11: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 1: Maximum entropy

Theorem (Shannon 1948)

If X has mean µ and variance σ and Y ∼ N(µ, σ2) then

h(X ) ≤ h(Y ),

with equality if and only if X ∼ N(µ, σ2).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 12: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 2: Entropy Power Inequality

I Define E(t) = h(N(0, t)) = 12 log2(2πet).

I Define entropy power v(X ) = E−1(h(X )) = 22h(X )/(2πe).

Theorem (EPI)

Consider independent continuous X and Y . Then

v(X + Y ) ≥ v(X ) + v(Y ),

with equality if and only if X and Y are Gaussian.

I First stated by Shannon.

I Lots of proofs (Stam/Blachman, Lieb,Dembo/Cover/Thomas, Tulino/Verdu/Guo).

I Restricted versions easier to prove? (cf Costa).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 13: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 2: Entropy Power Inequality

I Define E(t) = h(N(0, t)) = 12 log2(2πet).

I Define entropy power v(X ) = E−1(h(X )) = 22h(X )/(2πe).

Theorem (EPI)

Consider independent continuous X and Y . Then

v(X + Y ) ≥ v(X ) + v(Y ),

with equality if and only if X and Y are Gaussian.

I First stated by Shannon.

I Lots of proofs (Stam/Blachman, Lieb,Dembo/Cover/Thomas, Tulino/Verdu/Guo).

I Restricted versions easier to prove? (cf Costa).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 14: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 2: Entropy Power Inequality

I Define E(t) = h(N(0, t)) = 12 log2(2πet).

I Define entropy power v(X ) = E−1(h(X )) = 22h(X )/(2πe).

Theorem (EPI)

Consider independent continuous X and Y . Then

v(X + Y ) ≥ v(X ) + v(Y ),

with equality if and only if X and Y are Gaussian.

I First stated by Shannon.

I Lots of proofs (Stam/Blachman, Lieb,Dembo/Cover/Thomas, Tulino/Verdu/Guo).

I Restricted versions easier to prove? (cf Costa).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 15: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 2: Entropy Power Inequality

I Define E(t) = h(N(0, t)) = 12 log2(2πet).

I Define entropy power v(X ) = E−1(h(X )) = 22h(X )/(2πe).

Theorem (EPI)

Consider independent continuous X and Y . Then

v(X + Y ) ≥ v(X ) + v(Y ),

with equality if and only if X and Y are Gaussian.

I First stated by Shannon.

I Lots of proofs (Stam/Blachman, Lieb,Dembo/Cover/Thomas, Tulino/Verdu/Guo).

I Restricted versions easier to prove? (cf Costa).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 16: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 2: Entropy Power Inequality

I Define E(t) = h(N(0, t)) = 12 log2(2πet).

I Define entropy power v(X ) = E−1(h(X )) = 22h(X )/(2πe).

Theorem (EPI)

Consider independent continuous X and Y . Then

v(X + Y ) ≥ v(X ) + v(Y ),

with equality if and only if X and Y are Gaussian.

I First stated by Shannon.

I Lots of proofs (Stam/Blachman, Lieb,Dembo/Cover/Thomas, Tulino/Verdu/Guo).

I Restricted versions easier to prove? (cf Costa).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 17: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 2: Entropy Power Inequality

I Define E(t) = h(N(0, t)) = 12 log2(2πet).

I Define entropy power v(X ) = E−1(h(X )) = 22h(X )/(2πe).

Theorem (EPI)

Consider independent continuous X and Y . Then

v(X + Y ) ≥ v(X ) + v(Y ),

with equality if and only if X and Y are Gaussian.

I First stated by Shannon.

I Lots of proofs (Stam/Blachman, Lieb,Dembo/Cover/Thomas, Tulino/Verdu/Guo).

I Restricted versions easier to prove? (cf Costa).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 18: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 2: Entropy Power Inequality

I Define E(t) = h(N(0, t)) = 12 log2(2πet).

I Define entropy power v(X ) = E−1(h(X )) = 22h(X )/(2πe).

Theorem (EPI)

Consider independent continuous X and Y . Then

v(X + Y ) ≥ v(X ) + v(Y ),

with equality if and only if X and Y are Gaussian.

I First stated by Shannon.

I Lots of proofs (Stam/Blachman, Lieb,Dembo/Cover/Thomas, Tulino/Verdu/Guo).

I Restricted versions easier to prove? (cf Costa).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 19: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Equivalent formulation

Theorem (ECI – not proved here!)

For independent X ∗,Y ∗ with finite variance, for all α ∈ [0, 1],

h(√αX ∗ +

√1− αY ∗) ≥ αh(X ∗) + (1− α)h(Y ∗).

LemmaEPI is equivalent to ECI.

I Key role played in Lemma by fact about scaling:

v(√αX ) = αv(X ). (1)

I This holds since h(√αX ) = h(X ) + 1

2 logα, and

v(√αX ) = 22h(

√αX )/(2πe).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 20: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Equivalent formulation

Theorem (ECI – not proved here!)

For independent X ∗,Y ∗ with finite variance, for all α ∈ [0, 1],

h(√αX ∗ +

√1− αY ∗) ≥ αh(X ∗) + (1− α)h(Y ∗).

LemmaEPI is equivalent to ECI.

I Key role played in Lemma by fact about scaling:

v(√αX ) = αv(X ). (1)

I This holds since h(√αX ) = h(X ) + 1

2 logα, and

v(√αX ) = 22h(

√αX )/(2πe).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 21: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Equivalent formulation

Theorem (ECI – not proved here!)

For independent X ∗,Y ∗ with finite variance, for all α ∈ [0, 1],

h(√αX ∗ +

√1− αY ∗) ≥ αh(X ∗) + (1− α)h(Y ∗).

LemmaEPI is equivalent to ECI.

I Key role played in Lemma by fact about scaling:

v(√αX ) = αv(X ). (1)

I This holds since h(√αX ) = h(X ) + 1

2 logα, and

v(√αX ) = 22h(

√αX )/(2πe).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 22: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Proof of Lemma: EPI implies ECI

I By the EPI (where X =√αX ∗ and Y =

√1− αY ∗) and

scaling relation (1),

v(√αX ∗ +

√1− αY ∗) ≥ v(

√αX ∗) + v(

√1− αY ∗)

= αv(X ∗) + (1− α)v(Y ∗).

I Applying E to both sides and using Jensen (since E ∼ log, sois concave):

h(√αX ∗ +

√1− αY ∗) ≥ E

(αv(X ∗) + (1− α)v(Y ∗)

)≥ αE(v(X ∗)) + (1− α)E(v(Y ∗))

= αh(X ∗) + (1− α)h(Y ∗)

which is the ECI.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 23: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Proof of Lemma: EPI implies ECII By the EPI (where X =

√αX ∗ and Y =

√1− αY ∗) and

scaling relation (1),

v(√αX ∗ +

√1− αY ∗) ≥ v(

√αX ∗) + v(

√1− αY ∗)

= αv(X ∗) + (1− α)v(Y ∗).

I Applying E to both sides and using Jensen (since E ∼ log, sois concave):

h(√αX ∗ +

√1− αY ∗) ≥ E

(αv(X ∗) + (1− α)v(Y ∗)

)≥ αE(v(X ∗)) + (1− α)E(v(Y ∗))

= αh(X ∗) + (1− α)h(Y ∗)

which is the ECI.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 24: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Proof of Lemma: EPI implies ECII By the EPI (where X =

√αX ∗ and Y =

√1− αY ∗) and

scaling relation (1),

v(√αX ∗ +

√1− αY ∗) ≥ v(

√αX ∗) + v(

√1− αY ∗)

= αv(X ∗) + (1− α)v(Y ∗).

I Applying E to both sides and using Jensen (since E ∼ log, sois concave):

h(√αX ∗ +

√1− αY ∗) ≥ E

(αv(X ∗) + (1− α)v(Y ∗)

)≥ αE(v(X ∗)) + (1− α)E(v(Y ∗))

= αh(X ∗) + (1− α)h(Y ∗)

which is the ECI.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 25: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Proof of Lemma: ECI implies EPI

I For some α, define X ∗ = X/√α and Y ∗ = Y /

√1− α.

I Then the ECI and scaling (1) imply that

h(X + Y ) = h(√αX ∗ +

√1− αY ∗)

≥ αh(X ∗) + (1− α)h(Y ∗)

= αE(v(X ∗)) + (1− α)E(v(Y ∗))

= αE(

v(X )

α

)+ (1− α)E

(v(Y )

1− α

)

I Pick α = v(X )v(X )+v(Y ) and the above inequality becomes

h(X + Y ) ≥ E(v(X ) + v(Y )),

and applying E−1 to both sides gives the EPI.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 26: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Proof of Lemma: ECI implies EPI

I For some α, define X ∗ = X/√α and Y ∗ = Y /

√1− α.

I Then the ECI and scaling (1) imply that

h(X + Y ) = h(√αX ∗ +

√1− αY ∗)

≥ αh(X ∗) + (1− α)h(Y ∗)

= αE(v(X ∗)) + (1− α)E(v(Y ∗))

= αE(

v(X )

α

)+ (1− α)E

(v(Y )

1− α

)

I Pick α = v(X )v(X )+v(Y ) and the above inequality becomes

h(X + Y ) ≥ E(v(X ) + v(Y )),

and applying E−1 to both sides gives the EPI.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 27: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Proof of Lemma: ECI implies EPI

I For some α, define X ∗ = X/√α and Y ∗ = Y /

√1− α.

I Then the ECI and scaling (1) imply that

h(X + Y ) = h(√αX ∗ +

√1− αY ∗)

≥ αh(X ∗) + (1− α)h(Y ∗)

= αE(v(X ∗)) + (1− α)E(v(Y ∗))

= αE(

v(X )

α

)+ (1− α)E

(v(Y )

1− α

)

I Pick α = v(X )v(X )+v(Y ) and the above inequality becomes

h(X + Y ) ≥ E(v(X ) + v(Y )),

and applying E−1 to both sides gives the EPI.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 28: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Proof of Lemma: ECI implies EPI

I For some α, define X ∗ = X/√α and Y ∗ = Y /

√1− α.

I Then the ECI and scaling (1) imply that

h(X + Y ) = h(√αX ∗ +

√1− αY ∗)

≥ αh(X ∗) + (1− α)h(Y ∗)

= αE(v(X ∗)) + (1− α)E(v(Y ∗))

= αE(

v(X )

α

)+ (1− α)E

(v(Y )

1− α

)

I Pick α = v(X )v(X )+v(Y ) and the above inequality becomes

h(X + Y ) ≥ E(v(X ) + v(Y )),

and applying E−1 to both sides gives the EPI.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 29: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Rephrased EPI

I Note that this choice of α makesv(X ∗) = v(Y ∗) = v(X ) + v(Y ).

I This choice of scaling suggests the following rephrased EPI:

Corollary (Rephrased EPI)

Given independent X and Y with finite variance, there exist X ∗

and Y ∗ such that X =√αX ∗ and Y =

√1− αY ∗ for some α,

and such that h(X ∗) = h(Y ∗).The EPI is equivalent to the fact that

h(X + Y ) ≥ h(X ∗), (2)

with equality if and only if X and Y are Gaussian.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 30: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Rephrased EPI

I Note that this choice of α makesv(X ∗) = v(Y ∗) = v(X ) + v(Y ).

I This choice of scaling suggests the following rephrased EPI:

Corollary (Rephrased EPI)

Given independent X and Y with finite variance, there exist X ∗

and Y ∗ such that X =√αX ∗ and Y =

√1− αY ∗ for some α,

and such that h(X ∗) = h(Y ∗).The EPI is equivalent to the fact that

h(X + Y ) ≥ h(X ∗), (2)

with equality if and only if X and Y are Gaussian.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 31: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Rephrased EPI

I Note that this choice of α makesv(X ∗) = v(Y ∗) = v(X ) + v(Y ).

I This choice of scaling suggests the following rephrased EPI:

Corollary (Rephrased EPI)

Given independent X and Y with finite variance, there exist X ∗

and Y ∗ such that X =√αX ∗ and Y =

√1− αY ∗ for some α,

and such that h(X ∗) = h(Y ∗).The EPI is equivalent to the fact that

h(X + Y ) ≥ h(X ∗), (2)

with equality if and only if X and Y are Gaussian.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 32: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 3: Monotonicity

I Exciting set of strong recent results, collectively referred to as‘monotonicity’.

I First proved by Artstein/Ball/Barthe/Naor, alternative proofsby Tulino/Verdu and Madiman/Barron.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 33: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 3: Monotonicity

I Exciting set of strong recent results, collectively referred to as‘monotonicity’.

I First proved by Artstein/Ball/Barthe/Naor, alternative proofsby Tulino/Verdu and Madiman/Barron.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 34: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Property 3: Monotonicity

I Exciting set of strong recent results, collectively referred to as‘monotonicity’.

I First proved by Artstein/Ball/Barthe/Naor, alternative proofsby Tulino/Verdu and Madiman/Barron.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 35: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Monotonicity theorem

TheoremGiven independent continuous Xi with finite variance, for anypositive αi such that

∑n+1i=1 αi = 1, writing α(j) = 1− αj , then

nh

(n+1∑i=1

√αiXi

)≥

n+1∑j=1

α(j)h

∑i 6=j

√αi/α(j)Xi

.

I Choosing αi = 1/(n + 1) for IID Xi shows h(∑n

i=1 Xi/√

n)

ismonotone increasing in n.

I Equivalently relative entropy D(∑n

i=1 Xi/√

n∥∥Z ) is

monotone decreasing in n.

I Means CLT is equivalent of 2nd Law of Thermodynamics?

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 36: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Monotonicity theorem

TheoremGiven independent continuous Xi with finite variance, for anypositive αi such that

∑n+1i=1 αi = 1, writing α(j) = 1− αj , then

nh

(n+1∑i=1

√αiXi

)≥

n+1∑j=1

α(j)h

∑i 6=j

√αi/α(j)Xi

.

I Choosing αi = 1/(n + 1) for IID Xi shows h(∑n

i=1 Xi/√

n)

ismonotone increasing in n.

I Equivalently relative entropy D(∑n

i=1 Xi/√

n∥∥Z ) is

monotone decreasing in n.

I Means CLT is equivalent of 2nd Law of Thermodynamics?

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 37: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Monotonicity theorem

TheoremGiven independent continuous Xi with finite variance, for anypositive αi such that

∑n+1i=1 αi = 1, writing α(j) = 1− αj , then

nh

(n+1∑i=1

√αiXi

)≥

n+1∑j=1

α(j)h

∑i 6=j

√αi/α(j)Xi

.

I Choosing αi = 1/(n + 1) for IID Xi shows h(∑n

i=1 Xi/√

n)

ismonotone increasing in n.

I Equivalently relative entropy D(∑n

i=1 Xi/√

n∥∥Z ) is

monotone decreasing in n.

I Means CLT is equivalent of 2nd Law of Thermodynamics?

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 38: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Monotonicity theorem

TheoremGiven independent continuous Xi with finite variance, for anypositive αi such that

∑n+1i=1 αi = 1, writing α(j) = 1− αj , then

nh

(n+1∑i=1

√αiXi

)≥

n+1∑j=1

α(j)h

∑i 6=j

√αi/α(j)Xi

.

I Choosing αi = 1/(n + 1) for IID Xi shows h(∑n

i=1 Xi/√

n)

ismonotone increasing in n.

I Equivalently relative entropy D(∑n

i=1 Xi/√

n∥∥Z ) is

monotone decreasing in n.

I Means CLT is equivalent of 2nd Law of Thermodynamics?

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 39: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Monotonicity strengthens EPI

I By the right choice of α, monotonicity implies the followingstrengthened EPI.

Theorem (Strengthened EPI)

Given independent continuous Yi with finite variance, the entropypowers satisfy

nv

(n+1∑i=1

Yi

)≥

n+1∑j=1

v

∑i 6=j

Yi

,

with equality if and only if all the Yi are Gaussian.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 40: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Monotonicity strengthens EPI

I By the right choice of α, monotonicity implies the followingstrengthened EPI.

Theorem (Strengthened EPI)

Given independent continuous Yi with finite variance, the entropypowers satisfy

nv

(n+1∑i=1

Yi

)≥

n+1∑j=1

v

∑i 6=j

Yi

,

with equality if and only if all the Yi are Gaussian.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 41: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 1: Poisson maximum entropy

DefinitionFor any λ, define class of ultra-log-concave V with mass functionpV satisfying

ULC(λ) = {V : EV = λ and pV (i)/Πλ(i) is log-concave}.

That is

ipV (i)2 ≥ (i + 1)pV (i + 1)pV (i − 1), for all i .

I Class includes Bernoulli sums and Poisson.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 42: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 1: Poisson maximum entropy

DefinitionFor any λ, define class of ultra-log-concave V with mass functionpV satisfying

ULC(λ) = {V : EV = λ and pV (i)/Πλ(i) is log-concave}.

That is

ipV (i)2 ≥ (i + 1)pV (i + 1)pV (i − 1), for all i .

I Class includes Bernoulli sums and Poisson.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 43: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 1: Poisson maximum entropy

DefinitionFor any λ, define class of ultra-log-concave V with mass functionpV satisfying

ULC(λ) = {V : EV = λ and pV (i)/Πλ(i) is log-concave}.

That is

ipV (i)2 ≥ (i + 1)pV (i + 1)pV (i − 1), for all i .

I Class includes Bernoulli sums and Poisson.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 44: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Maximum entropy and ULC(λ)

Theorem (Johnson, Stoch. Proc. Appl. 2007)

If X ∈ ULC(λ) and Y ∼ Πλ then

H(X ) ≤ H(Y ),

with equality if and only if X ∼ Πλ.

(see also Harremoes, 2001)

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 45: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Key operation: thinning

DefinitionGiven Y , define the α-thinned version of Y by

TαY =Y∑

i=1

Bi ,

where B1,B2 . . . i.i.d. Bernoulli(α), independent of Y .

I Thinning has many interesting properties.

I We believe Tα is the discrete equivalent of scaling by√α.

I Preserves several parametric families.

I ‘Mean-preserving transform’ TαX + T1−αY equivalent to‘variance-preserving transform’

√αX +

√1− αY in

continuous case? (Matches max. ent. condition).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 46: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Key operation: thinning

DefinitionGiven Y , define the α-thinned version of Y by

TαY =Y∑

i=1

Bi ,

where B1,B2 . . . i.i.d. Bernoulli(α), independent of Y .

I Thinning has many interesting properties.

I We believe Tα is the discrete equivalent of scaling by√α.

I Preserves several parametric families.

I ‘Mean-preserving transform’ TαX + T1−αY equivalent to‘variance-preserving transform’

√αX +

√1− αY in

continuous case? (Matches max. ent. condition).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 47: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Key operation: thinning

DefinitionGiven Y , define the α-thinned version of Y by

TαY =Y∑

i=1

Bi ,

where B1,B2 . . . i.i.d. Bernoulli(α), independent of Y .

I Thinning has many interesting properties.

I We believe Tα is the discrete equivalent of scaling by√α.

I Preserves several parametric families.

I ‘Mean-preserving transform’ TαX + T1−αY equivalent to‘variance-preserving transform’

√αX +

√1− αY in

continuous case? (Matches max. ent. condition).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 48: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Key operation: thinning

DefinitionGiven Y , define the α-thinned version of Y by

TαY =Y∑

i=1

Bi ,

where B1,B2 . . . i.i.d. Bernoulli(α), independent of Y .

I Thinning has many interesting properties.

I We believe Tα is the discrete equivalent of scaling by√α.

I Preserves several parametric families.

I ‘Mean-preserving transform’ TαX + T1−αY equivalent to‘variance-preserving transform’

√αX +

√1− αY in

continuous case? (Matches max. ent. condition).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 49: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Key operation: thinning

DefinitionGiven Y , define the α-thinned version of Y by

TαY =Y∑

i=1

Bi ,

where B1,B2 . . . i.i.d. Bernoulli(α), independent of Y .

I Thinning has many interesting properties.

I We believe Tα is the discrete equivalent of scaling by√α.

I Preserves several parametric families.

I ‘Mean-preserving transform’ TαX + T1−αY equivalent to‘variance-preserving transform’

√αX +

√1− αY in

continuous case? (Matches max. ent. condition).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 50: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 2: EPI

I Define E(t) = H(Πt), an increasing, concave function.

I Define V (X ) = E−1(H(X )).

Conjecture

Consider independent discrete X and Y . Then

V (X + Y ) ≥ V (X ) + V (Y ),

with equality if and only if X and Y are Poisson.

I Turns out not to be true!

I Even natural restrictions e.g. ULC, Bernoulli sums don’t help

I Counterexample (not mine!): X ∼ Y ,PX (0) = 1/6, PX (1) = 2/3, PX (2) = 1/6.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 51: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 2: EPI

I Define E(t) = H(Πt), an increasing, concave function.

I Define V (X ) = E−1(H(X )).

Conjecture

Consider independent discrete X and Y . Then

V (X + Y ) ≥ V (X ) + V (Y ),

with equality if and only if X and Y are Poisson.

I Turns out not to be true!

I Even natural restrictions e.g. ULC, Bernoulli sums don’t help

I Counterexample (not mine!): X ∼ Y ,PX (0) = 1/6, PX (1) = 2/3, PX (2) = 1/6.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 52: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 2: EPI

I Define E(t) = H(Πt), an increasing, concave function.

I Define V (X ) = E−1(H(X )).

Conjecture

Consider independent discrete X and Y . Then

V (X + Y ) ≥ V (X ) + V (Y ),

with equality if and only if X and Y are Poisson.

I Turns out not to be true!

I Even natural restrictions e.g. ULC, Bernoulli sums don’t help

I Counterexample (not mine!): X ∼ Y ,PX (0) = 1/6, PX (1) = 2/3, PX (2) = 1/6.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 53: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 2: EPI

I Define E(t) = H(Πt), an increasing, concave function.

I Define V (X ) = E−1(H(X )).

Conjecture

Consider independent discrete X and Y . Then

V (X + Y ) ≥ V (X ) + V (Y ),

with equality if and only if X and Y are Poisson.

I Turns out not to be true!

I Even natural restrictions e.g. ULC, Bernoulli sums don’t help

I Counterexample (not mine!): X ∼ Y ,PX (0) = 1/6, PX (1) = 2/3, PX (2) = 1/6.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 54: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 2: EPI

I Define E(t) = H(Πt), an increasing, concave function.

I Define V (X ) = E−1(H(X )).

Conjecture

Consider independent discrete X and Y . Then

V (X + Y ) ≥ V (X ) + V (Y ),

with equality if and only if X and Y are Poisson.

I Turns out not to be true!

I Even natural restrictions e.g. ULC, Bernoulli sums don’t help

I Counterexample (not mine!): X ∼ Y ,PX (0) = 1/6, PX (1) = 2/3, PX (2) = 1/6.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 55: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 2: EPI

I Define E(t) = H(Πt), an increasing, concave function.

I Define V (X ) = E−1(H(X )).

Conjecture

Consider independent discrete X and Y . Then

V (X + Y ) ≥ V (X ) + V (Y ),

with equality if and only if X and Y are Poisson.

I Turns out not to be true!

I Even natural restrictions e.g. ULC, Bernoulli sums don’t help

I Counterexample (not mine!): X ∼ Y ,PX (0) = 1/6, PX (1) = 2/3, PX (2) = 1/6.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 56: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 2: EPI

I Define E(t) = H(Πt), an increasing, concave function.

I Define V (X ) = E−1(H(X )).

Conjecture

Consider independent discrete X and Y . Then

V (X + Y ) ≥ V (X ) + V (Y ),

with equality if and only if X and Y are Poisson.

I Turns out not to be true!

I Even natural restrictions e.g. ULC, Bernoulli sums don’t help

I Counterexample (not mine!): X ∼ Y ,PX (0) = 1/6, PX (1) = 2/3, PX (2) = 1/6.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 57: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Thinned Entropy Power Inequality

Conjecture (TEPI)

Consider independent discrete ULC X and Y . For any α,conjecture that

V (TαX + T1−αY ) ≥ αV (X ) + (1− α)V (Y ),

with equality if and only if X and Y are Poisson.

I Again, not true in general!

I Perhaps not all α?

I Have partial results, but not full description of which α.

I For example, true for Poisson Y with H(Y ) ≤ H(X ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 58: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Thinned Entropy Power Inequality

Conjecture (TEPI)

Consider independent discrete ULC X and Y . For any α,conjecture that

V (TαX + T1−αY ) ≥ αV (X ) + (1− α)V (Y ),

with equality if and only if X and Y are Poisson.

I Again, not true in general!

I Perhaps not all α?

I Have partial results, but not full description of which α.

I For example, true for Poisson Y with H(Y ) ≤ H(X ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 59: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Thinned Entropy Power Inequality

Conjecture (TEPI)

Consider independent discrete ULC X and Y . For any α,conjecture that

V (TαX + T1−αY ) ≥ αV (X ) + (1− α)V (Y ),

with equality if and only if X and Y are Poisson.

I Again, not true in general!

I Perhaps not all α?

I Have partial results, but not full description of which α.

I For example, true for Poisson Y with H(Y ) ≤ H(X ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 60: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Thinned Entropy Power Inequality

Conjecture (TEPI)

Consider independent discrete ULC X and Y . For any α,conjecture that

V (TαX + T1−αY ) ≥ αV (X ) + (1− α)V (Y ),

with equality if and only if X and Y are Poisson.

I Again, not true in general!

I Perhaps not all α?

I Have partial results, but not full description of which α.

I For example, true for Poisson Y with H(Y ) ≤ H(X ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 61: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Thinned Entropy Power Inequality

Conjecture (TEPI)

Consider independent discrete ULC X and Y . For any α,conjecture that

V (TαX + T1−αY ) ≥ αV (X ) + (1− α)V (Y ),

with equality if and only if X and Y are Poisson.

I Again, not true in general!

I Perhaps not all α?

I Have partial results, but not full description of which α.

I For example, true for Poisson Y with H(Y ) ≤ H(X ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 62: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Two weaker results

I Analogues of the continuous concavity and scaling results dohold. (Again, proofs not given here!)

Theorem (TECI, Johnson/Yu, ISIT ’09)

Consider independent ULC X and Y . For any α,

H(TαX + T1−αY ) ≥ αH(X ) + (1− α)H(Y ).

Theorem (RTEPI, Johnson/Yu, arXiv:0909.0641)

Consider ULC X . For any α,

V (TαX ) ≥ αV (X ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 63: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Two weaker results

I Analogues of the continuous concavity and scaling results dohold. (Again, proofs not given here!)

Theorem (TECI, Johnson/Yu, ISIT ’09)

Consider independent ULC X and Y . For any α,

H(TαX + T1−αY ) ≥ αH(X ) + (1− α)H(Y ).

Theorem (RTEPI, Johnson/Yu, arXiv:0909.0641)

Consider ULC X . For any α,

V (TαX ) ≥ αV (X ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 64: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete EPI?

I Duplicating steps from the continuous case above, we deducean analogue of rephrased EPI

Theorem (Johnson/Yu, arXiv:0909.0641)

Given independent ULC X and Y , suppose there exist X ∗ and Y ∗

such that X = TαX ∗ and Y = T1−αY ∗ for some α, and such thatH(X ∗) = H(Y ∗). Then

H(X + Y ) ≥ H(X ∗), (3)

with equality if and only if X and Y are Poisson.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 65: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete EPI?

I Duplicating steps from the continuous case above, we deducean analogue of rephrased EPI

Theorem (Johnson/Yu, arXiv:0909.0641)

Given independent ULC X and Y , suppose there exist X ∗ and Y ∗

such that X = TαX ∗ and Y = T1−αY ∗ for some α, and such thatH(X ∗) = H(Y ∗). Then

H(X + Y ) ≥ H(X ∗), (3)

with equality if and only if X and Y are Poisson.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 66: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 3: Monotonicity

I Write D(X ) for D(X‖ΠEX ).I By convex ordering arguments, Yu showed that for IID Xi :

1. relative entropy D(∑n

i=1 T1/nXi

)is monotone decreasing in n,

2. for ULC Xi the entropy H(∑n

i=1 T1/nXi

)is monotone

increasing in n.

I In fact, implicit in work of Yu is following stronger theorem:

TheoremGiven positive αi such that

∑n+1i=1 αi = 1, and writing

α(j) = 1− αj , then for any independent ULC Xi ,

nD

(n+1∑i=1

Tαi Xi

)≤

n+1∑j=1

α(j)D

∑i 6=j

Tαi/α(j)Xi

.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 67: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 3: Monotonicity

I Write D(X ) for D(X‖ΠEX ).

I By convex ordering arguments, Yu showed that for IID Xi :1. relative entropy D

(∑ni=1 T1/nXi

)is monotone decreasing in n,

2. for ULC Xi the entropy H(∑n

i=1 T1/nXi

)is monotone

increasing in n.

I In fact, implicit in work of Yu is following stronger theorem:

TheoremGiven positive αi such that

∑n+1i=1 αi = 1, and writing

α(j) = 1− αj , then for any independent ULC Xi ,

nD

(n+1∑i=1

Tαi Xi

)≤

n+1∑j=1

α(j)D

∑i 6=j

Tαi/α(j)Xi

.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 68: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 3: Monotonicity

I Write D(X ) for D(X‖ΠEX ).I By convex ordering arguments, Yu showed that for IID Xi :

1. relative entropy D(∑n

i=1 T1/nXi

)is monotone decreasing in n,

2. for ULC Xi the entropy H(∑n

i=1 T1/nXi

)is monotone

increasing in n.

I In fact, implicit in work of Yu is following stronger theorem:

TheoremGiven positive αi such that

∑n+1i=1 αi = 1, and writing

α(j) = 1− αj , then for any independent ULC Xi ,

nD

(n+1∑i=1

Tαi Xi

)≤

n+1∑j=1

α(j)D

∑i 6=j

Tαi/α(j)Xi

.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 69: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Discrete Property 3: Monotonicity

I Write D(X ) for D(X‖ΠEX ).I By convex ordering arguments, Yu showed that for IID Xi :

1. relative entropy D(∑n

i=1 T1/nXi

)is monotone decreasing in n,

2. for ULC Xi the entropy H(∑n

i=1 T1/nXi

)is monotone

increasing in n.

I In fact, implicit in work of Yu is following stronger theorem:

TheoremGiven positive αi such that

∑n+1i=1 αi = 1, and writing

α(j) = 1− αj , then for any independent ULC Xi ,

nD

(n+1∑i=1

Tαi Xi

)≤

n+1∑j=1

α(j)D

∑i 6=j

Tαi/α(j)Xi

.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 70: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Generalization of monotonicity

Theorem (Johnson/Yu, arXiv:0909.0641)

Given positive αi such that∑n+1

i=1 αi = 1, and writingα(j) = 1− αj , then for any independent ULC Xi ,

nH

(n+1∑i=1

Tαi Xi

)≥

n+1∑j=1

α(j)H

∑i 6=j

Tαi/α(j)Xi

.

I Exact analogue of Artstein/Ball/Barthe/Naor result,

nh

(n+1∑i=1

√αiXi

)≥

n+1∑j=1

α(j)h

∑i 6=j

√αi/α(j)Xi

,

replacing scalings by thinnings.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 71: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Generalization of monotonicity

Theorem (Johnson/Yu, arXiv:0909.0641)

Given positive αi such that∑n+1

i=1 αi = 1, and writingα(j) = 1− αj , then for any independent ULC Xi ,

nH

(n+1∑i=1

Tαi Xi

)≥

n+1∑j=1

α(j)H

∑i 6=j

Tαi/α(j)Xi

.

I Exact analogue of Artstein/Ball/Barthe/Naor result,

nh

(n+1∑i=1

√αiXi

)≥

n+1∑j=1

α(j)h

∑i 6=j

√αi/α(j)Xi

,

replacing scalings by thinnings.

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 72: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Future work

I Resolve for which α, the

V (TαX + T1−αY ) ≥ αV (X ) + (1− α)V (Y ).

I Relation to Shepp-Olkin conjecture

I Conjecture: if there exist X ∗ and Y ∗ such that X = TαX ∗

and Y = T1−αY ∗, where α = V (X )/(V (X ) + V (Y )), then

V (X + Y ) ≥ V (X ) + V (Y ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 73: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Future work

I Resolve for which α, the

V (TαX + T1−αY ) ≥ αV (X ) + (1− α)V (Y ).

I Relation to Shepp-Olkin conjecture

I Conjecture: if there exist X ∗ and Y ∗ such that X = TαX ∗

and Y = T1−αY ∗, where α = V (X )/(V (X ) + V (Y )), then

V (X + Y ) ≥ V (X ) + V (Y ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 74: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Future work

I Resolve for which α, the

V (TαX + T1−αY ) ≥ αV (X ) + (1− α)V (Y ).

I Relation to Shepp-Olkin conjecture

I Conjecture: if there exist X ∗ and Y ∗ such that X = TαX ∗

and Y = T1−αY ∗, where α = V (X )/(V (X ) + V (Y )), then

V (X + Y ) ≥ V (X ) + V (Y ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar

Page 75: Monotonicity, thinning and discrete versions of the Entropy Power

Gaussian maximum entropy EPI Monotonicity Poisson maximum entropy Discrete EPI Discrete Monotonicity

Future work

I Resolve for which α, the

V (TαX + T1−αY ) ≥ αV (X ) + (1− α)V (Y ).

I Relation to Shepp-Olkin conjecture

I Conjecture: if there exist X ∗ and Y ∗ such that X = TαX ∗

and Y = T1−αY ∗, where α = V (X )/(V (X ) + V (Y )), then

V (X + Y ) ≥ V (X ) + V (Y ).

Oliver Johnson [email protected] Statistics Group, University of Bristol

Monotonicity, thinning and discrete versions of the Entropy Power Inequality: Warwick Statistical Mechanics Seminar