Quantization Design Jie Ren [email protected]Adaptive Signal Processing and Information Theory Group Department of Electrical and Computer Engineering Drexel University, Philadelphia, PA 19104 July 28 th and 30 th , 2014 Jie Ren (Drexel ASPITRG) QD July 28 th and 30 th , 2014 1 / 35
35
Embed
Quantization Design - Drexel Engineering · • N-level Lloyd-Max quantizer : minimize the average distortion for a xed number of levels N.[1][2] • N-level Optimum Quantizer : minimize
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Adaptive Signal Processing and Information Theory GroupDepartment of Electrical and Computer Engineering
Drexel University, Philadelphia, PA 19104
July 28th and 30th, 2014
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 1 / 35
References
S. P. Lloyd, “Least squares quantization in pcm,” IEEE Transactions on Acoustics, Speechand Signal Processing, vol. 28, no. 2, pp. 129–137, March 1982.
J. Max, “Quantizing for minimum distortion,” IEEE Transactions on Acoustics, Speechand Signal Processing, vol. 6, no. 1, pp. 7–12, March 1960.
N. Farvardin and J. W. Modestino, “Optimum quantizer performance for a class ofnon-gaussian memoryless sources,” IEEE Trans. Inform. Theory, vol. 30, no. 3, pp.485–497, May 1984.
D. K. Sharma, “Design of absolutely optimal quantizers for a wide class of distortionmeasures,” IEEE Trans. Inform. Theory, vol. 24, no. 6, pp. 693–702, November 1978.
P. A. Chou, T. Lookabaugh, and R. M. Gray, “Entropy-constrained vector quantization,”IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 37, no. 1, pp. 31–42,Juanuary 1989.
L. R. Varshney, “Unreliable and resource-constrained decoding,” Ph.D. dissertation,Massachusetts Institute of Technology, 2010.
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 2 / 35
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 9 / 35
Lloyd-max Quantizer Design Local Optimality Conditions
Local Optimality Conditions
• Nearest Neighbor Condition : For fixed reconstruction levels {ak},Given any θ ∈ [ak , ak+1],
Q(θ) = d(θ, ak) ≤ d(θ, ak+1) ? ak : ak+1 (8)
• Centroid Condition : For fixed regions {Rk} with thresholds {bk},
ak = arg mina
∫ bk
bk−1
d(θ, a)p(θ)dθ (9)
• Zero Probability Boundary Condition, for all bk , k = 1, . . . ,K − 1
P(θ = bk) = 0 (10)
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 10 / 35
Lloyd-max Quantizer Design Local Optimality Conditions
Necessary and Sufficient Conditions
TheoremThe nearest neighbor condition, the centroid condition, and the zeroprobability of boundary condition are necessary for a Lloyd-Max quantizerto be optimal.
TheoremIf the following conditions hold for a source Θ and distortion functiond(θ, a) :
1 p(θ) is positive and continuous in (0, 1)
2∫ 1
0 d(θ, a)p(θ)dθ is finite for all a
3 d(θ, a) is zero only for θ = a, is continuous in θ for all a, and iscontinuous and convex in a
then the nearest neighbor condition, centroid condition, and zeroprobability of bound- ary conditions are sufficient to guarantee localoptimality of a quantizer.
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 11 / 35
Lloyd-max Quantizer Design By Alternating Optimization
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 12 / 35
Lloyd-max Quantizer Design By Alternating Optimization
Lloyd-Max algorithm
Alternating Minimization
• For fixed {ak}, minimize D w.r.t. {bk}• For fixed {bk}, minimize D w.r.t. {ak}• D monotone decreasing
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 13 / 35
Lloyd-max Quantizer Design By Alternating Optimization
Lloyd-Max algorithm
Algorithm 1: Lloyd-Max
Result: Minimize the average distortion for a N-level Lloyd-Max quantizerstep 1) Choose an arbitrary set of initial reconstruction levels {an}step 2) For each n = 1, . . . ,N set Rn = {θ|d(θ, an) ≤ d(θ, aj), j 6= n}step 3) For each n = 1, . . . ,N set an = arg mina E [d(Θ, a)|Θ ∈ Rn]step 4) Repeat step 2 and 3 until change in average distortion is negligiblestep 5) Revise {an} and {Rn} to satisfy the zero probability of boundarycondition
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 14 / 35
Lloyd-max Quantizer Design By Alternating Optimization
Analysis
• Local optimum guaranteed by Theorem. 2
• Monotonic Convergence in N
D∗N(b∗, a∗) =N∑
n=1
∫ b∗n
b∗n−1
d(θ, a∗n)p(θ)dθ (11)
D∗ = limN→∞
D∗N (12)
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 15 / 35
Lloyd-max Quantizer Design By Alternating Optimization
Monotonic Convergence in N
The Lloyd-Max N-level quantizer is the solution of the followingproblem:
D∗N = minN∑
n=1
∫ bn
bn−1
d(θ, an)p(θ)dθ
s.t. b0 = 0
bN = 1
bn−1 ≤ bn, n = 1, . . . ,N
an ≤ bn, n = 1, . . . ,N
(13)
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 16 / 35
Lloyd-max Quantizer Design By Alternating Optimization
Monotonic Convergence in N
• Degenerate the N-level Lloyd-Max quantizer to N − 1
• By adding the additional constraint bN−1 = 1 to (13) and forcingaN = 1, hence
D∗N−1 ≥ D∗N (14)
• D∗N bounded below by 0
• D∗N converges
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 17 / 35
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 18 / 35
Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using DynamicProgramming
• For discrete Θ
• One construction level in the interval (β1, β2) ⊆ [0, 1]
T1(β1, β2) = mina
∑θ∈Θ∩(β1,β2)
d(θ, a)p(θ) (15)
• K construction levels in the interval (β1, β2)
TK (β1, β2) = mina,b:β1<b1<···<bK−1<β2
K∑k=1
∑θ∈Θ∩(bk−1,bk )
d(θ, a)p(θ) (16)
• Notice thatD∗N = TN(0, 1) (17)
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 19 / 35
Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using DynamicProgramming
TheoremLet b∗1, . . . , b
∗K−1 be the optimizing boundary points for
TK (b∗0 = 0, b∗K = 1), then b∗1, . . . , b∗K−2 must be the optimizing boundary
points for TK−1(b∗0, b∗K−1), and
TK (b∗0, b∗K ) = min
bK−1:b∗0<bK−1<b∗K[TK−1(b∗0, bK−1) + T1(bK−1, b
∗K )] (18)
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 20 / 35
Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using DynamicProgramming
• For any 1 < k ≤ K and any discrete β ∈ (b∗0, b∗K ]
Tk(b∗0, β) = minb:b∗0<b<β
[Tk−1(b∗0, b) + T1(b, β)] (19)
• Optimizing threshold
b∗k−1(b∗0, β) = arg minb:b∗0<b<β
[Tk−1(b∗0, b) + T1(b, β)] (20)
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 21 / 35
Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using DynamicProgramming
Algorithm 2: DP algorithm for Lloyd-Max Quantizer Design
Result: Minimize the average distortion for a N-level Lloyd-Max quantizerstep 1) Compute the values of T1(β1, β2) for all discrete β1 and β2 in [0, 1]step 2) For each n = 2, . . . ,N compute Tn(0, β) and b∗n−1(0, β) for all βin (0, 1] using (19) and (20)step 3) Let bN = 1, for each n = N, . . . , 2 set bn−1 = b∗n−1(0, bn)step 4) For each n = 1, . . . ,N, set
ak = arg mina E [d(Θ, a)|Θ ∈ (bk−1, bk)]
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 22 / 35
Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using DynamicProgramming
TheoremThe boundaries {bk}Kk=0 and reconstruction levels {ak}Kk=1 returned byAlgorithm 2 represent the optimal quantizer.[4]
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 23 / 35
Result: Minimize the average distortion for a N-level variable rateoptimum quantizer
step 1) Given N and fixed λ, set b0 = 0, choose an initial value for b1 andset n = 1step 2) For the present values of bn−1 and bn, use (28) and (32) to findbn+1. If n ≤ N − 1, replace n by n + 1 and go to step 2). Otherwisecontinuestep 3) If bN obtained in step 2) is equal to 1, the initial guess for b1 isgood and the resulting b and a satisfy the necessary conditions foroptimality. Otherwise go to step 1) and change b1
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 33 / 35
Result: Minimize the average distortion for a N-level variable rateoptimum quantizer
step 1) Given N and fixed λ, set b0 = 0, bN = 1, and choose an initialvalue for b1, . . . , bN−1.step 2) Compute {a1, . . . , aN} by (28)step 3) Compute {b1, . . . , bN−1} by (34)step 4) Run step 2) and step 3) ` times. If D∗N converges, output b and a.Otherwise go to step 1) and change initial guess for b
Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 35 / 35