Norm Hurst - IABM...•Scenes are mostly ^diffuse reflectors. SDR/709 puts these ~100%) •:

IABM Copyright 2020@THEIABM www.theiabm.org

Norm HurstSenior Principal Research Engineer

SRI International


Norm Hurst• 1976 – 1980: RCA Broadcast Studio Camera design

• 1980 – 1990: RCA Labs support of consumer video products

• 1990 – 2008: Advanced TV system design & test (Grand Alliance, etc)

• Since 2008: design video test patterns for professional use


Norm Hurst• SMPTE Fellow

• SMPTE Digital Processing Medal 2013

• SMPTE Camera Origination Medal 2016

• Three Dozen Patents


• Designed to reveal Digital TV problems:• Compression• Mismatched YUV Matrix• Lip Sync• Bit Depth• Resolution Scaling• Chroma Subsampling (4:2:2, 4:2:0)• Dropped frames• and more...

• No need for waveform monitor• Evaluation by direct observation: just look!

2008: I designed this pattern for what we now call “SDR”.

2015: “Will you make a version for HDR/WCG?”“Sure,” I said, not really knowing what that meant!


• What is dynamic range?• HDR Photography vs HDR Video• What kinds of images benefit from HDR?

• What is color gamut?• What kinds of images benefit from WCG?

• What is video? (How does a camera make video?)• Why is there Gamma?

• Optimizing the Gamma curve• Optimal: PQ• Compatible: HLG

• Making an HDR Test Pattern• Convert color space, or redesign?• Revealing the pitfalls of HDR/WCG

• The OOTF and why you should care

Outline








Outline


• HDR Basics• What is an f-stop?• What is Dynamic Range?• What is a ‘nit’?


• What is an f-stop?• It’s a lens thing – the aperture (iris) ring has calibrated click “stops” • Changes the light by a factor of 2:1• Indicates a ratio of light. “3 stops” is an 8:1 change (23)

f-stop 1.4 2 2.8 4 5.6 8 11 16 22

squared 2 4 8 16 32 64 128 256 512

f-stops

The funny numbers (1.4, 2, 2.8, …) make more sense when you square them to make them proportional to the area instead of the diameter.


Dynamic Range• Dynamic Range: the ratio of the brightest to the darkest• For a camera sensor: saturation to noise floor• For a display: peak luminance to darkest lowlight• For a human eye: brightest tolerable to darkest perceivable• Often described in f-stops


• What is the Dynamic Range of the human eye?• Within a single scene, about 100:1 (about 6.6 stops)• Absolute range is about 20 stops (1,000,000:1)

• What is the Dynamic Range of a camera?• “Full-well” (can’t take anymore photons!) down to noise (can’t tell if there’s anything)• State of the art: ~14 f-stops

• What is a ‘nit’?• Unit of luminance: candela/meter2: cd/m2

• Shorthand (like “click” for kilometer)

Dynamic Range


• “My phone’s camera has an ‘HDR’ mode!” • “HDR” is an overloaded term• A phone takes multiple SDR exposures and merges them • Compresses an HDR range into an SDR range

merge or

HDR Photography != HDR Video

An aside…




HDR Photography

SDR Display

An aside…




HDR Video

(Can’t show HDR on your SDR computer)

(Can’t show HDR on your SDR computer)

An aside…


• Scenes that could benefit from HDR

Highlights blown out!Could look better in HDR!



Crowd is blown out!Could look better in HDR!



Dark areas crushed!Could look better in HDR!

Game of Thrones“The Long Night”S8E3










• Scenes that could benefit from HDR Tone mapping reveals bad quantization









Outline


• WCG Basics• What is a Color Gamut?• What types of content look better in WCG?


• A color is represented by three values: R, G, B• But these values alone cannot nail down a particular color• These values are relative to a set of “Color Primaries”• Part of a defined “Color Space”• If you don’t know the Color Space, you don’t know what color an RGB triplet represents!


CIE Color Diagram


CIE Color Diagram

The edge of the horseshoe represents spectrally-pure colors like from a laser.


CIE Color Diagram


The inside is all the colors most humans can see


CIE Color Diagram



Near the middle is the “white point” like “D65”


CIE Color Diagram




R

G

B

The corners of this triangle define a “color space”. The corners are called the “primaries”.


CIE Color Diagram




R

G

B

The corners of this triangle define a color space.

You can describe any color inside the triangle, but not outside. This is your Gamut

of colors.


CIE Color Diagram




R

G

B

The corners of this triangle define a color space.

You can describe any color inside the triangle, but not outside. This is your Gamut

of colors.

A bigger triangle contains more intense colors. It is a Wider Color Gamut.



Extra colors that WCG gives you.


• Scenes that could benefit from WCG• Most everyday colors fit in the 709 color gamut (range)• Very few real-life colors exceed it



Flowers


Flowers Some Fabrics (umbrellas)



Flowers Some Fabrics (umbrellas) Neon SignsOther “emissive” sources

like fireworks



• HDR: High Dynamic Range • As opposed to SDR, Standard Dynamic Range, what we used to call “video”• Shows more detail in bright areas and dark areas• Whites not blown out, blacks not crushed

• WCG: Wide Color Gamut• As opposed to, uh, … what we used to call “video”• Allows capture and display of more saturated colors

• HDR and WCG are independent features• You could define a system with just one or the other• But today they always travel together• HDR works without WCG, but WCG needs HDR

• (HDR on your cell phone’s camera is not the same!)


• This sounds like a radical change to the TV systems

• But is it?• In theory (how it works), no.

• In practice (how we use it), yes.

• Let’s review how video works








Outline


• What is “Video”?• A stream, a file

• A signal on a wire

• But where does it come from?• It comes from a camera

• Let’s remind ourselves of what happens

between the camera lens and the wire…


“Camera”

LENS


Sensor

Raw

“Camera”


Sensor

RGBRaw

Raw to “RGB”

“Camera”


Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’

“Camera”


Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’

“Camera”

• Wait – why is there “gamma”?

• “To pre-compensate for CRT non-linearity?” No!

• Even if CRTs were linear, you would still want gamma!

• Gamma/OETF gives more quantization steps near black

• Perceptual: Humans are more sensitive to small changes in dark than in bright


RGBQuantization

5 bits (32 steps)

Lossy/NoisyTransmission


RGBQuantization

5 bits (32 steps)

Light into Camera

Ligh

t on

Dis

pla

y

5-bit TV System with no Gamma

Gamma = 1.0



RGBQuantization

5 bits (32 steps)

Light into Camera

Ligh

t on

Dis

pla

y


Gamma = 1.0

Human Perception: Need finer steps near black

Human Perception: Fewer steps near white.



RGBQuantization

5 bits (32 steps)

Light into Camera

Ligh

t on

Dis

pla

y


Gamma = 1.0

Human Perception: Need finer steps near black

Human Perception: Fewer steps near white.

Need about 12 bits (4096 steps) to make this look OK!



RGB

Gamma/OETF

R’G’B’Quantization

5 bits (32 steps)

Gamma/EOTF



RGB

Gamma/OETF


5 bits (32 steps)

Gamma/EOTF

Inverse Curves



RGB

Gamma/OETF


5 bits (32 steps)

Gamma/EOTF

5-bit TV System with 709 Gamma (SDR)

Gamma = ~2.0

Inverse Curves



RGB

Gamma/OETF


5 bits (32 steps)

Gamma/EOTF


Gamma = ~2.0

Inverse Curves

Smaller steps here

Bigger steps here



RGB

Gamma/OETF


5 bits (32 steps)

Gamma/EOTF


Gamma = ~2.0

Inverse Curves

Smaller steps here

Bigger steps here

The use of Gamma makes the system a better perceptual match to the human eye’s sensitivity.



RGB

Gamma/OETF


5 bits (32 steps)

Gamma/EOTF


Gamma = ~2.0

Inverse Curves

Smaller steps here

Bigger steps here

The use of Gamma makes the system a better perceptual match to the human eye’s sensitivity.

“Television doesn’t have to work;It only has to look like it works!”



Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’

“Camera”


Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’

R’G’B’ to Y’CbCr

Y’CbCr

“Camera”


Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”

• “Television doesn’t have to work;• It only has to look like it works!”• Human Perception: good for B&W, not so much for color (rods & cones)• Convert from RGB to YUV (Y: B&W; U&V: color differences)• So we can throw away resolution in U & V! (4:2:2, 4:2:0)• Heads up: changing the RGB Primaries (WCG) requires changing this matrix a little bit.


Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”

Y’CbCr 8-bits


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”

Y’CbCr

Y’CbCr 8-bits


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”

R’G’B’ Y’CbCr

Y’CbCr to R’G’B’

Y’CbCr 8-bits


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”

RGB R’G’B’ Y’CbCr

Y’CbCr to R’G’B’Gamma/EOTF

Y’CbCr 8-bits


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”


Y’CbCr to R’G’B’Gamma/EOTF“RGB” to Native

Native

Y’CbCr 8-bits


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr 8-bits

Display

A Complete TV System


Q: How does HDR change this System Diagram?

Q: How does WCG change this System Diagram?

A: It doesn’t, it only changes the parameters in the boxes!


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr 8-bits

Display

WCG changes the Color Primaries to Rec.2020


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr 8-bits

Display

HDR changes the Gamma/OETF curve to PQ or HLG


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr 8-bits

Display

WCG changes the YCbCr Matrix to Rec.2020


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr 10-bits

Display

HDR changes distribution bit depth (HEVC Main10 Profile)


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

HDR/WCG modifies existing TV system parameters

10-bits








Outline


• Traditional Gamma “optimizes” the TV system for human Perception (kind of)• Not really optimized• But in 1939, needed to use CRT curve to make cheap TVs

• Dolby said, “Let’s make a new curve that is truly Perceptually optimized!”• Started with Barten’s model of human contrast sensitivity• Barten: Visibility of a brightness change requires a bigger change at higher brightnesses

How to optimize the Gamma/OETF curve for Human Perception?


Brightness cd/m2 (‘nits’)

Smal

lest

ste

p y

ou

ca

n s

ee

Barten’s Contrast Sensitivity Function


• Traditional Gamma “optimizes” the TV system for human Perception (kind of)• Not really optimized• But in 1939, needed to use CRT curve to make cheap TVs

• Dolby said, “Let’s make a new curve that is truly perceptually optimized!”• Started with Barten’s model of human contrast sensitivity• Barten: Visibility of a brightness change requires a bigger change at higher brightnesses

• Barten: it depends on the absolute brightness in nits• Dolby: each quantization step should be just a little less than what you can

perceive: Perceptual Quantization: “PQ”• Note: PQ maps code values to absolute brightness values!

• 75% = 983 nits (“How many nits is color bars?”)• 100% = 10,000 nits

How to optimize the Gamma/OETF curve for Human Perception?


Rec.709

Gamma/OETF

Linear Light in

Co

de

Val

ue

“IR

E” o

ut


Rec.709

Gamma/OETF

Linear Light in

Co

de

Val

ue

“IR

E” o

ut

PQ


Rec.709

Gamma/OETF

Linear Light in

Co

de

Val

ue

“IR

E” o

ut

PQ709 Gamma, 32-steps


Rec.709

Gamma/OETF

Linear Light in

Co

de

Val

ue

“IR

E” o

ut

PQPQ, 32-steps


Rec.709

Gamma/OETF

Linear Light in

Co

de

Val

ue

“IR

E” o

ut

PQ

PQ/2020 as Rec. 709

Rec. 709 as Rec. 709

PQ is not intended to be watchable (compatible) on old TVs


• BBC & ARIB said, “Compatibility (watchability) on existing TVs is important”• Proposed a different curve• Lower half: Gamma (compatible part)• Upper half: Log (HDR part)• Smooth transition• “Hybrid: Log/Gamma”: HLG




First half: Gamma



First half: Gamma

Second half: Log



Rec.709

Gamma/OETF

Linear Light in

Co

de

Val

ue

“IR

E” o

ut

PQ/2020 as Rec. 709


PQ

HLG

HLG/2020 as Rec. 709

HLG is more compatible on old TVs(and existing OB vans!)


Rec.709

Gamma/OETF

Linear Light in

Co

de

Val

ue

“IR

E” o

ut

PQ/2020 as Rec. 709


PQ

HLG

HLG/2020 as Rec. 709

HLG is more compatible on old TVs(and existing OB vans!)








Outline


How can we make an HDR pattern from a 709 SDR pattern?

SDR (Rec.709) Test PatternConvert to HDR?


• Isn’t HDR just a different color space?

• A “color space” has a few parameters:• Gamut: RGB Color primary x,y positions (x,y) (x,y) (x,y)• Gamma/OETF curve definition• White point (e.g. D65) position: (x,y) (Same in HDR!)

• SDR uses • Gamut=Rec.709 Primaries• OETF=709 Gamma• White point=D65

• HDR uses • Gamut= Rec.2020 Primaries • OETF=HLG or OETF=PQ • White point=D65


• Can’t we make an HDR test pattern by just converting the color space?

• Converting between color spaces is well-defined• SMPTE RP-177M




Rec.709 R’G’B’ Code values

Example: Converting Rec.709 to PQ/2020



Remove Gamma/OETF encoding

Linear Light RGB 709








Convert 709 Primaries to “Universal” Primaries

(XYZ)

Linear Light XYZ









(XYZ)

Convert “Universal” Primaries (XYZ) toRec.2020

Linear Light XYZ Linear Light 2020









(XYZ)



Apply desired OETF e.g. PQ

PQ/2020 Code Values









(XYZ)



Apply desired OETF e.g. PQ

PQ/2020 Code Values


(Naïve)BT.709 to BT.2020/PQColor Space Converter




BT.709 to BT.2020/PQColor Space Converter

(RP-177M)

HDR by

Color space Conversion?

709 2020/PQ

709 2020/PQ



(RP-177M)

HDR by


100 IRE maps to 100 IRE

709 2020/PQ

709 2020/PQ



(RP-177M)

HDR by


100 IRE maps to 10,000 Nits

709 2020/PQ

709 2020/PQ


Rec. 2020/PQ

100 IRE == 10,000 nitsMost displays max at 1000


Rec. 2020/PQ

Clipped at 1000 nits

100 IRE == 10,000 nitsMost displays max at 1000

That’s not useful!


Rec.709

OETF

Linear Light in

Co

de

Val

ue

“IR

E” o

ut

HLG

PQ

70

9: 1

00

nit

s m

ax

HLG

: 100

0 ni

ts m

ax(?

)

PQ: 1

0,00

0 ni

ts m

ax

Curves compared with equal range

light inputs

Curves compared with intended

range light inputs

OETF

Linear Light in

Co

de

Val

ue

“IR

E” o

ut


• What is wrong with saying “HDR is just another color space”?• Scenes are mostly “diffuse” reflectors. SDR/709 puts these ~100%

• Most CRTs max out around 100-120 nits (cd/m2)• SDR has very little headroom for specular reflectors

• HDR is not about making everything bright• Just making it “sparkly” here and there – not the whole picture!• Most of our test pattern should be “exposed” around 100-200 nits• Just a few features should be bright – like the real world

• Two kinds of peak white:• Diffuse (~200 nits)• Specular (1,000 - 10,000 nits)• 5X to 50X headroom (2 to 6 f-stops headroom)

10,000

6400

3200

1600

800

400

200

100

0

nit

s

Diffuse white peak100% for SDR

Specular white peak100% for HDR


Instead of this…


Only a few features are scaled to

“specular” 10,000 nits.

Most features are scaled to “diffuse.”


These Features are “bright”

Scaled to “Max Specular Reflectance” = 10,000 nits


These Features: scaled to “diffuse”

Scaled to “100% Diffuse Reflectance” = 203 nits


• What can go wrong in HDR?

• How does this test pattern reveal it?

• Back to the Camera/Display diagram…


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

10-bitsMismatch?

• YUV Matrix Mismatch?

• Three standards: Rec.601, Rec.709, Rec.2020


YUV Matrix Mismatch:Ramp in Red and Green Channels


0%

100%

0%

100%

0%

R

G

B

Clipping Matched


0%

100%

0%

100%

0%

R

G

B

2020

RGBTo

YCC

Clipping Matched

R

G

B Cb

Y

Cr


0%

100%

0%

100%

0%

R

G

B

2020

RGBTo

YCC

709

YCCTo

RGB

Clipping Matched

R

G

B

R

G

B

Y

Cr

Cb

Mis-Matched Matrices


0%

100%

0%

100%

0%

R

G

B

2020

RGBTo

YCC

709

YCCTo

RGB

Clipping Matched Clipping Not Matched

R

G

B

R

G

B

Y

Cr

Cb

Mis-Matched Matrices


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

10-bitsMismatch?

• OETF/EOTF Mismatch? a.k.a. tone map mismatch


Sinewave sweep: linear in light

Sinewave sweep: linear in code values


Sine Waves

Linear in Code Values

LightCode Values


EOTF

Sine Waves NOT Sine Waves


LightCode Values


EOTF



Appears to showMoiré Patterns

LightCode Values


EOTF




LightCode Values

NOT Sine Waves

EOTF-1


EOTF




LightCode Values

EOTF

NOT Sine Waves Sine Waves Linear in Light Values

EOTF-1


EOTF




LightCode Values

EOTF


Moiré Nulls Outwhen gamma matches

EOTF-1


EOTF




Code Values

EOTF


Moiré Nulls Outwhen gamma matches

EOTF-1


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

10-bits

Loss of bit depth?

• HDR requires 10 bits to the home

• 8-bits is common, but can you see it?


Bit depth detector:Shallow, rotating gradient

8-bit

Banding in 8-bit


Bit depth detector:Shallow, rotating gradient

10-bit

No banding in 10-bit


“Display”

Sensor

RGB

Gamma/OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

10-bitsMismatch?

• OETF/EOTF Mismatch? a.k.a. tone map mismatch


Mismatched gamuts

Matched Gamuts


Mismatched gamuts

2020-signal, 709 display



10,000 nits

8,000 nits

4,000 nits

2,000 nits

1,000 nits


1/3 stop down

1/3 stop up

10,000 nits

8,000 nits

4,000 nits

2,000 nits

1,000 nits


18% chip

1/3 stop down

1/3 stop up

10,000 nits

8,000 nits

4,000 nits

2,000 nits

1,000 nits


18% chip

1/3 stop down

1/3 stop up

Moving “delta step”

10,000 nits

8,000 nits

4,000 nits

2,000 nits

1,000 nits


No

rmal

Sta

irst

ep

Clip

ped

at

10

00

nit

s

different

same

different

different

same

same


• One more detail: OOTF• OETF: Optical-to-Electronic transfer function (camera)• EOTF: Electronic-to-Optical transfer function (display)• OOTF: Optical-to-optical transfer function (the whole system, glass-to-glass)• Linear, right?• No.• A net “gamma” of about 1.2 is desirable


“Display”

Sensor

RGB

OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”


Y’CbCr to R’G’B’EOTF“RGB” to Native

Native

Y’CbCr

Display

10-bitsInverses


“Display”

Sensor

RGB

OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

10-bitsInverses

~1.2

PQ defines the OOTF at the Camera


“Display”

Sensor

RGB

OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

10-bitsInverses

~1.2*

*OOTF function depends on display brightness: 1.2 for 1000 nit displayMax nits

HLG defines the OOTF at the Display


“Display”

Sensor

RGB

OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

10-bitsInverses

~1.2*

Max nits

Different light functions!


“Display”

Sensor

RGB

OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

10-bitsInverses

~1.2*

Max nits


SceneReferred


“Display”

Sensor

RGB

OETF

Raw

Raw to “RGB”

R’G’B’


Y’CbCr

“Camera”



Native

Y’CbCr

Display

10-bits

~1.2*

Max nits


SceneReferred

DisplayReferred

Inverses


• Should a test pattern be scene referred or display referred?• It depends• Will the test pattern be used to evaluate a display? Display Referred.• Will the pattern be used to evaluate signal processing after the camera? Scene Referred.


Display referred


Scene referred


Scene referred


Scene referred


Scene referred


• SMPTE ST-2084: defines PQ curve• PQ also known as HDR10

• BT.2020 defines WCG color primaries (and UHD and 8K scanning params)• BT.2100 defines color space parameters for PQ and HLG• BT.2111 defines “HDR” color bars. Actually two kinds: HLG bars and PQ bars

• 203 nits (how many nits is color bars?)• BT.2408: Guidance for live production with mixed SDR and HDR sources and outputs

Selected standards


Too many features to describe today.

For more info about test patterns:

Contact:Arkady [email protected]


Too many features to describe today.

For more info about test patterns:

Contact:Arkady [email protected]

THANK YOU!

Norm [email protected]


Norm Hurst - IABM...•Scenes are mostly ^diffuse reflectors. SDR/709 puts these ~100%) •:

Documents