Advanced Intra Coding
Intra-Picture Prediction
Intra-Picture Prediction
Transform coding: Typical design2D Discrete Cosine Transform of type II (or integer approximation)Scalar quantization: URQs with same quantization step size ∆
Entropy coding (employing remaining statistical dependencies)Can only utilize dependencies within transform blocks
Intra-picture predictionCan additionally utilize dependencies between transform blocksVery simple variant (JPEG and H.262 | MPEG-2 Video):Predict DC coefficient using DC coefficient of previous blockMore advanced approaches can significantly increase coding efficiency
Two approaches of intra-picture predictionPrediction in transform domainPrediction in spatial domain (before transform coding)
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 2 / 22
Intra-Picture Prediction / Prediction of Transform Coefficients
Prediction of DC Coefficient
Simplest Variant of Intra-Picture Prediction (JPEG & MPEG-2 Video)Predict DC level with DC level of previous block (left to current block)Transmit only difference to predictor
∆DCk = DCk −DCk−1
Reduces variance of pmf of transmitted DC levelImproves coding efficiency
Example: Histogram with and without DC prediction (8×8 blocks)
0 10 20 30 40 50 60
without DC prediction
-30 -20 -10 0 10 20 30
with DC prediction
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 3 / 22
Intra-Picture Prediction / Prediction of Transform Coefficients
Example: Effect of DC Prediction (8×8 Blocks)
22
24
26
28
30
32
34
36
38
40
42
44
46
48
50
52
0 0.5 1 1.5 2 2.5 3 3.5 4
Lum
a P
SN
R [d
B]
bits per luma sample
testImage02
no dcPredmid range dcPred
adaptive dcPred
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 4 / 22
Intra-Picture Prediction / Prediction of Transform Coefficients
Advanced Intra Prediction in Transform Domain
Advanced Intra Coding mode of H.263: Three coding modesDC prediction and zig-zag scanHorizontal prediction and alternate-vertical scan
Suitable for blocks with mainly horizontal structuresVertical prediction and alternate-horizontal scan
Suitable for blocks with mainly vertical structures
Similar concept is also used in MPEG-4 VisualT. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 5 / 22
Intra-Picture Prediction / Prediction of Transform Coefficients
How to Select Suitable Prediction Mode?
Mode DecisionSyntax supports multiple intra prediction modes (mode is transmitted)Encoder has to select one of the supported modesGoal: Maximize coding efficiency
Lagrangian Mode Decision
Evaluate all supported coding modesPerform prediction, quantization, entropy coding, reconstructionCalculate distortion D =
∑k(s ′k − sk)2
Determine number of bits R for transmitting mode and transformcoefficient levels
Select mode that minimizes Lagrangian cost function
J = D + λ · R
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 6 / 22
Intra-Picture Prediction / Spatial Intra Prediction
Intra Prediction: Transform Domain — Spatial Domain
verticalprediction
intransformdomain
equivalentvertical
predictionin spatialdomain
simplifiedand
improvedvertical
predictionin spatialdomain
Example: Vertical predictionTransform domain: Predict first row of transform coefficientsEquivalent prediction in spatial domain
sver[x , y ] =1N
N−1∑k=0
s ′[x ,−1− k]
Simplified prediction in spatial domain
sver[x , y ] = s ′[−1, y ]
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 7 / 22
Intra-Picture Prediction / Spatial Intra Prediction
Spatial Intra Prediction
Spatial intra predictionSimilar complexity than similar operation in transform domainUsage of directly adjacent samples =⇒ Improved coding efficencyMain advantages:
Can also be applied if neighboring blocks are coded in an inter modeStraightforward extension to multiple prediction directions(can include interpolation of border samples)
Intra prediction in video coding standardsH.262 | MPEG-2: Predict DC coefficient from previous blockH.263 & MPEG-4: DC, horizontal, vertical (in transform domain)H.264 | AVC: 9 spatial intra prediction modes (for 4×4/8×8 blocks)H.265 | HEVC: 35 spatial intra prediction modes (for all block sizes)
Number of supported intra prediction modes is increased from one generationof video coding standards to the next
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 8 / 22
Intra-Picture Prediction / Spatial Intra Prediction
Spatial Intra Prediction in H.264 | MPEG-4 AVC
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 9 / 22
Intra-Picture Prediction / Spatial Intra Prediction
Spatial Intra Prediction in H.265 | MPEG-H HEVC
33 directional prediction modes (linear interpolation / ref. sample smoothing)
DC prediction mode (similar to H.264 | MPEG-4 AVC)
Planar prediction mode
In total: 35 spatial intra prediction modes
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 10 / 22
Intra-Picture Prediction / Spatial Intra Prediction
HEVC Planar Mode: General Idea
Not all image blocks fit an edge model (well representable by directionalimage model)DC prediction only very coarse approximation (order-0 model)Alternative: Planar prediction mode
s[x , y ] =N − 1− x
2N· s ′[−1, y ] +
1 + x
2N· s[N − 1, y ]+
N − 1− y
2N· s ′[x ,−1] +
1 + y
2N· s[x ,N − 1]
s[N − 1, y ] =N − 1− y
N· s ′[N − 1,−1] +
1 + y
2N· s[N − 1,N − 1]
s[x ,N − 1] =N − 1− x
N· s ′[−1,N − 1] +
1 + x
2N· s[N − 1,N − 1]
s[N − 1,N − 1] =12· s ′[N − 1,−1] +
12· s ′[−1,N − 1]
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 11 / 22
Intra-Picture Prediction / Spatial Intra Prediction
HEVC Planar Mode: Actual Implementation
sH[x , y ] = (N − 1− x) · s ′[−1, y ] + (1 + x) · s ′[N − 1,−1]
sV[x , y ] = (N − 1− y) · s ′[x ,−1] + (1 + y) · s ′[−1,N − 1]
s[x , y ] =12N
(sH[x , y ] + sV[x , y ]
)T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 12 / 22
Intra-Picture Prediction / Spatial Intra Prediction
Spatial Intra Prediction — Coding Efficiency
0
10
20
30
40
50
30 32 34 36 38 40
bit-
rate
sav
ing
vs D
C p
red.
[%]
PSNR (Y) [dB]
Cactus (1920×1080, 50 Hz), 8×8 blocks
DC, horizontal, and vertical prediction
9 prediction modes
35 prediction modes
0
10
20
30
40
50
34 36 38 40 42 44
bit-
rate
sav
ing
vs D
C p
red.
[%]
PSNR (Y) [dB]
Kimono (1920×1080, 24 Hz), 8×8 blocks
DC, horizontal, and vertical prediction
9 prediction modes
35 prediction modes
Experimental investigation with H.265 | MPEG-H HEVCRestricted to 8×8 blocks (effect of block size is discussed later)Limited number of used prediction modes (reference: DC prediction only)
Coding efficiency increases with number of supported intra prediction modes
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 13 / 22
Variable Block Sizes for Prediction and Transform Coding
Block Sizes for Prediction and Transform Coding
Impact of block size selection for transform codingCoding efficiency of transform coding typically increases with block sizeCoding efficiency improvement becomes small beyond a certain block sizeComplexity increases with block size
Impact of block size selection for spatial predictionCorrelation decreases with increasing sample distancesIntra prediction is more effective for smaller block sizesSide information rate (for intra modes) increases with decreasing block size
Combination of intra prediction and transform codingOptimal block size depends on actual signal propertiesNatural images: Highly non-stationary statistical propertiesNo single optimal block sizeAdaptive block size selection can improve coding efficiency
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 14 / 22
Variable Block Sizes for Prediction and Transform Coding
Block Sizes in Video Coding Standards
H.262 | MPEG-2 Video, H.263, MPEG-4 VisualFixed block sizes for prediction and transform coding16×16 macroblocks (for signaling intra prediction mode)8×8 transform blocks
H.264 | MPEG-4 AVC (High profile)16×16 macroblocks3 intra coding modes: Intra4x4, Intra8x8, Intra16x16Block sizes for prediction and transform coding: 4×4, 8×8, 16×16Intra prediction mode selected on basis of transform blocksIntra16x16: Only 4 prediction modes & low-complexity 16×16 transform
Intra4x4 Intra8x8 Intra16x16
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 15 / 22
Variable Block Sizes for Prediction and Transform Coding
Picture Partitioning in H.265 | MPEG-H HEVC
Picture partitioning into coding tree blocksCoding tree blocks (CTBs): Fixed size of16×16, 32×32 or 64×64 luma samplesSize of CTBs chosen by encoderLuma and chroma CTBs together withsyntax are called coding tree unit (CTU)
Partitioning of coding tree blocksQuad-tree partitioning into coding blocks (CBs)Luma and chroma CBs together with syntax arecalled coding unit (CU)Minimum CU size: Selected by encoder, butequal to or larger than 8×8 luma samplesCoding mode (intra or inter) is chosen for CUCoding order: Z-scan
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 16 / 22
Variable Block Sizes for Prediction and Transform Coding
Example: Picture Partitioning into Coding Units
Example for picture partitioning into coding unitsPicture with 2560×1600 luma samples of HEVC test sequence “Traffic”Quadtree-based partitioning into coding unit represents a simple scheme forlocally adapting the block sizes to the image structure
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 17 / 22
Variable Block Sizes for Prediction and Transform Coding
Intra Coding of Coding Units in H.265 | MPEG-H HEVC
Partitioning of a CB into transform blocks (TBs)Nested quad-tree partitioningTB corresponds to a single block transformSupported sizes: 4×4, 8×8, 16×16, 32×32Luma and chroma TBs together with syntaxform a transform unit (TU)
Intra prediction and mode signalingOne or four luma intra predictionmodes per coding unitOne chroma prediction mode per CUActual intra prediction is performedtransform block by transform blockImproved prediction accuracy
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 18 / 22
Variable Block Sizes for Prediction and Transform Coding
Block Sizes for Intra-Picture Coding — Coding Efficiency
30
32
34
36
38
40
0 20 40 60 80 100
PS
NR
(Y
) [d
B]
bit rate [Mbit/s]
Cactus (1920×1080, 50 Hz), DC prediction
4×4 blocks
8×8 blocks
16×16 blocks
32×32blocks
all block sizes
34
36
38
40
42
44
0 10 20 30 40 50
PS
NR
(Y
) [d
B]
bit rate [Mbit/s]
Kimono (1920×1080, 24 Hz), DC prediction
4×4 blocks
8×8 blocks
16×16 blocks
32×32 blocks
all block sizes
First coding experiment with H.265 | MPEG-H HEVCReduce impact of intra prediction: Only DC prediction is enabledCheck different fixed block sizes & variable block sizesFixed block sizes: Coding efficiency increases with block sizeVariable block sizes provide coding gains
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 19 / 22
Variable Block Sizes for Prediction and Transform Coding
Block Sizes for Intra-Picture Coding — Coding Efficiency
30
32
34
36
38
40
0 20 40 60 80 100
PS
NR
(Y
) [d
B]
bit rate [Mbit/s]
Cactus (1920×1080, 50 Hz), all pred. modes
4×4 blocks
8×8 blocks
16×16 blocks
32×32 blocks
all block sizes
34
36
38
40
42
44
0 10 20 30 40 50
PS
NR
(Y
) [d
B]
bit rate [Mbit/s]
Kimono (1920×1080, 24 Hz), all pred. modes
4×4 blocks
8×8 blocks
16×16 blocks
32×32 blocks
all block sizes
Second coding experiment with H.265 | MPEG-H HEVCAll intra prediction modes are enabledPrediction increases effectiveness of smaller block sizesFixed block sizes: Medium block sizes provide best coding efficiencyVariable block sizes provide coding gains
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 20 / 22
Variable Block Sizes for Prediction and Transform Coding
Block Sizes for Intra-Picture Coding — Coding Efficiency
0
10
20
30
40
50
30 32 34 36 38 40
bit-
rate
sav
ing
vs 8×
8 bl
ocks
[%]
PSNR (Y) [dB]
Cactus (1920×1080, 50 Hz), all pred. modes
4×4 and 8×8 blocks
4×4, 8×8 blocks,and 16×16 blocks
all block sizes (4×4 to 32×32)
0
10
20
30
40
50
34 36 38 40 42 44
bit-
rate
sav
ing
vs 8×
8 bl
ocks
[%]
PSNR (Y) [dB]
Kimono (1920×1080, 24 Hz), all pred. modes
4×4 and 8×8 blocks
4×4, 8×8 blocks,and 16×16 blocks
all block sizes (4×4 to 32×32)
Third coding experiment with H.265 | MPEG-H HEVCAll intra prediction modes are enabledStart with 8×8 blocks and successively enable additional block sizesAdditional block sizes provide coding efficiency improvementsBeside intra-picture prediction, the support of additional block sizes is a mainfactor for the improvement in intra-picture coding
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 21 / 22
Summary
Part Summary
Transform coding of sample blocksSeparable orthogonal transform: DCT or integer approximationScalar quantization: URQs with same quantization step size ∆
Entropy coding: Utilize remaining dependencies between quantization indexes
Intra-picture predictionUtilize dependencies between transform blocksTwo methods: Prediction in transform domain or spatial domainSpatial prediction: Straightforward realization of multiple prediction modesCoding efficiency typically increases with number of supported intra modes
Block sizes for intra prediction and transform codingDetermine efficiency of prediction and transform codingNon-stationary character of natural images =⇒ Variable block sizesSimple and flexible partitioning: Quadtree-based approachesVariable block sizes significantly increase coding efficiency
T. Wiegand (TU Berlin) — Image and Video Coding: Advanced Intra Coding 22 / 22