IST-083 Symposium Progressive Still Image Transmission over a TDL Network A Study Case: JPEG2000 Compressed Images over a Link 16 Network Author: Cdr,

IST-083 Symposium

Progressive Still Image Transmission over a TDL Network

A Study Case: JPEG2000 Compressed Images over a Link 16 Network

Author: Cdr, Manuel Martinez, PHD (SPN)Co author: Prof, Antonio Artes, PHD (UC3M)Co author: Maj, Roberto Sabatini, PHD (IT AF)

Scope

• Operational Requirements

• Image Technical Requirements

• JPEG2000 Standard

• Link 16 Case Study

• Some Examples

Imagery Operational Requirements

• Target images required before authorizing weapon release.

• Strike Aircraft aircrew using an image depicting a time critical target (TCT) for identification.

• Damage Assessment.

• UAV imagery supporting urban/roads battlefield.

• To develop a “composite tactical picture” available to all Joint Units by using “multimedia” information content provided by TDLs in support of network centric warfare.

Why in TDLs?

Why in TDLs?

• Extend TDLs concept from “textual data” to “multimedia elements” including still image & stream-video for a variety of image types– Visual– IR/FLIR– SAR

• Each image shows different characteristics based on the sensor performance and image type.

• Don’t ask for new “spectrum allocation” in dense urban areas– Use the current TDL’s (Link 16) allocated spectrum.– Use current AJP capabilities provided by TDLs (Link

16) to avoid jammers /civilian interferences.

“Still Image” Compression Requirements

• Compression Efficiency.– Excellent performance at low or very low data rate (disadvantage

links).• Random access to the “code-stream”.

– Embedded code stream that allows to extract the image in different ways.

– This shall allow to reconstruct an image based on a progressive scheme.

• Scalable.– Resolution.– Quality: definition of QoS and quality layers.– Components: color, B&W,…– Position.

• Lossless and Lossy Compression.• Open architecture and standardized. • Definition of Region of Interest (ROI), Zoom, Tiling,..• Robustness in channel error conditions.

Current Efforts. DLWG

• DLCP NUMBER: ML277-054-P33-US-M32-R6• J16.0 Image Transfer Message definition.• NPG 11 dedicated for Imagery• Minimum 32 Time Slots required per frame.• Recommended P4 Packing Limit (Link 16).• Supplement messages from surveillance and

control.– Secondary role.

• Imagery and image amplification information defined in the standard.

• Voice coordination required.

Current Efforts. DLWG (2)

• Fulfill NATO Secondary Image Format (NSIF).– STANAG 4545, Edition 1 Amendment 1 dated

14 April 2002 – NATO Secondary Imagery Format (NSIF) Version 1.0.

• Compression Algorithms: JPEG & EagleEye.• Image packet data of 45 bytes per J 16.0

message.• Geo-position Information is included as a non

displayable image segment.• File Transfer protocols: one way & two ways.

Proposed Still Image Compression Standard:JPEG2000 (1).

• Defines a “canvas coordinate system” based on tiles – Since different components can have different sizes.

• Each tile is DWTd Provides “spatial” access to the image (can be operationally required).– Discrete Wavelet Transform (DWT).

• Wavelets provides a robust tool for signal and image processing.

• More accurate time-frequency representation Can handle signals with discontinuities.

• Transform Domain: each sub-band of every resolution level is partitioned into blocks/precints/packets.

Proposed Still Image Compression Standard:JPEG2000 (2)

• Embedded block coding algorithm (EBCOT): one code-stream can be decompressed in many ways.– Each sub-band image divided into blocks that

are coded independently. Introduces quality for “layers” concept. – Progressive compression.

• By quality / resolution / components and position

• State of the art compression efficiency.

JPEG2000 Canvas Coordinated System

T19T18T17T16T15

T14T13T12T11T10

T9T8T7T6T5

T4T3T2T1T0

Xsiz

XOsiz

XTOsiz

Ysi

z YO

siz

YT

Osi

zXTsiz

YT

siz

(0,0)

JPEG2000 Block Diagram

ICT[RGB YCbCr]

DWT_Irr[Num Niv, D]

DZQ

RCT[RGB Y’DbDr]

DWT_Rev[Num Niv, D]

Ranging[eb]

Offset

ROI[Max-Shift, U]

Transf_Color Transf_Wavelet Quantif

Image Samples Quantification Indexes

Muestras Sub_bd

EBCOTCoder

ICT: Inter-component TransformationDWT: Discrete Wavelet Transform: Daubechies 9/7 (NR) & Daubechies 5/7 (Reversible) DZQ: Dead Zone Quantification

JPEG2000: Wavelet Transform

•Sub-band Images from low & high pass filtering and sub-sampling•Same samples as in original image•Transform is reversible•Sub-bands are quantized

(Image from David Taubman)

JPEG2000: Wavelet Transform (2)

•Multi-resolution representation of the image•Image is LL/HL/LH/HH filtered in various levels

•LL: Approximation of image (low frequency)•LH: Horizontal details.•HL: Vertical details.•HH: Diagonal Details (high frequency)

Note: Image by Konstantinos Kamaras Thesis, March 2002.

JPEG2000: EBCOT(Embedded Block Coded with

Optimal Truncation)• EBCOT defines quality layers: each block

stream is optimally truncated.– From low quality imagery (coarse

quantization) to the best image (finest quantization)

JPEG 2000 Standard. Different Parts

• Part #1: Core coding & basic technology (JP2)• Part #2: Extensions (JPX)• Part#3: Motion JPEG2000 (MJ2)• Part #5: Reference SW (Java & C)• Part#8: Encryption and Watermarking (JPSEC)• Part #9: Interactive Protocols and API (JPIP)• Part#11: robust communication for wireless

(JPWL)• Part #12: ISO Base Media File Format (Common

with MPEG-4)

JPEG2000 Code stream (JP2)

• JPEG2000 coder produces an embedded code-stream that can be decoded at any length to produce an image.– From 0 bit rate to the maximum (lossless).

• JPEG produces an independent code-stream per “target bit-rate”.– Implies multiple coding passes less efficient.

• Embedded code stream support random access to image properties:– Region of Interest (ROI).– Zoom-in / out.

JPEG2000 Code stream (JP2)

MainHeader

Tilestream

Tilestream

Tilestream

EOC

Tileheader

packetstream

packet

packet

SOC

SIZ

COD

QCD

+

COCQCCRGNPOCPPMPLMTLMCRGCOM

SOT

CODQCDCOCQCCRGNPOCPPTPLTCOM

SOD

+

OptionalMarkers Packet 0

TileHeader

Packet i

Packet j

Packet k

Packet l

Packet m

Tile Stream: t

TileHeader

Packet 0

Packet i

Tile part 0

Packet l

Packet m

Tile-partHeader

Tile part N-1

POC

PPT

SOT

PLT

COM

SOD

TileStream

Geo Referenced Images(Geography Markup Language)

• JPEG2000 doesn’t specify mechanism for geo-referencing the image, describing the sensor characteristics,…

• JPEG2000 provides room for “boxes” containing arbitrary XML data.

• GML (by Open Geo-Spatial Consortium) provides geo referencing information as XML encoded metadata.

• GML & JPEG2000 are interoperable and compatible.

JPEG2000/JPEGComparative Analysis

• Progressive lossy to lossless compression– JPEG: not integrated capability.

• Random Access to the code-stream:– Implementation of Region of Interest capability.– Better data rate control.

• JPEG2000: better compression efficiency.(Image1 & 2 are referenced in paper)

Bpp 0.125 0.5 2.00

Img1 JPEG 24.42 31.17 35.15

Img1 JPEG2000 28.12 32.95 37.35

Img2 JPEG 22.60 28.92 35.99

Img2JPEG2000 24.85 31.13 38.80

JPEG2000/JPEGComparative Analysis in Noisy Channels

• Better performance in noisy channels. NOT EDC Technique Implemented WORS CASE SCENARIO better if R-S is modeled.

– See example: BER: 10^ (-4).– Error Symbols in the code-stream:

• JPEG2000: 28• JPEG: 18

• Method:– JPEG2000 code stream is built (*.J2C)– A Binary vector representation of the code stream is developed. – A binary vector with the same BER channel is generated: x=(rand(tamano,1)<=ber)– An exclusive OR is made between both vectors for generating a LINK-16 channel corrupted

JPEG2000 code stream.– Image reconstruction with “Kakadu”

JPEG JPEG2000

Requirements Flow Down: From Design to Operation

Link 16NetworkRqmts

ImageryRqmts

-One / Two ways-Image Type:SAR/FLIR/Visual-Quality Min_PSNR

-% TS Imagery-Relé-VoIice- AJP

NETWORK DESIGN PHASE

OPERATION PHASE

Link 16TX (BER)

JPEG2000CodingPARAMETERS

-Link 16 (TSR)-Interactive Mode (TWO WAYS)Voice / data coordination (service channel)

ImageryCompression

Note: Link 16 Service channel: 8 TS assigned if required (Free Text)

Req

uire

men

ts f

low

dow

n

JPEG2000 Compressed Image Over Link 16

• Method proposed– Transmit the most sensitive part of JPEG2000 code-stream with

maximum Link 16 Anti-Jamming Protection: STD Packing Limit.• Headers/markers/first layers/tiles/

– Rest of the image with a less robust Packing Limit. (less AJP) • JPEG2000 Coding option: progressive by quality.

– Quality Layers. • Link 16.

– “free text” unformatted messages/ JXX.YY standard format message.

– NPGs: from 1..4 (NPG 11 –assigned- & spare NPGs: 15 to 18)– Max TSBs: 64 per Link 16 Terminal (48 in TSR)– Time slots available for imagery

• Two study cases: NMT 10% and NMT 5%.

JPEG2000 Quality Layer toTime Slot Allocation Algorithm

2.-Time Slots per NPG: I1: X_TS I2: Y_TSI3: Z_TSI4: T_TS

4. Image SizeM x N bpp

1. NPGsavailable for Imagery1..4

3.-Bit Rate Computing (bits)

-Layer #1: TB#1=X (TS)*PL_Std / TS................

-Layer #N: TB#N =i (TS) * PL_j / TS

5.-BPP computing

BPP#1=TB#1 / MxN....................

BPP#N=BPP#(N-1)+TB#N / MxN

Definition OriginalImage

ORIGINAL IMAGELINK 16 RESOURCES

6.-JPEG2000:LRCP

Match Link 16 available resources with JPEG2000 LRCP “quality layers”

Matching Numbers: An Example

• Step 1 &2. Define the bit rate iaw network design: 32+x+y+z (time slots available for imagery)

• Step 3. Bit rate layer#1: 32 TS *XXX bits/TS_PL=YYY bits.

• Step 4. Size of the image=640 x 437 pixel.• Step 5. Bit rate layer #1: r_1=YYY/640*437= 0.02574

bpp (P2DP).• Step 6. Layer #n: r (n)= r(1) + (r), with (r ) iaw time

slots allocation by network design• Step 7. Image JPEG2000 coded LRCP iaw layer(n) bpp.

– Nota: Each additional quality layer can be allocated to a different

NPG or TSAB of the corresponding NPG.

Communication Architecture& Protocol

• Sensor Node (SN)– Captures the image.– Image Preprocessing & Compression:

filtering/PSNR/Quality Criteria.– Build Geo referencing code stream (If required).

• Imagery Management Node (IMN)– Receive the image and request retransmission or

changes in quality based on R-S Symbol threshold. (Errors in channel)

– Request for ROI.– Request for more time-slots if in TSR.– Reconstruct the original image based on code stream

received and operational/tactical requirements.

Protocol

Mgmt Unit Sensor Unit

CS: Msg: “Req_Img”

Capture of Img

Preproc Img: filtering, zoom, etcCompres_Img

CP: -Preparation Tx link 16 channel.-Msg: “Tx_Img”

CP: RX_ImgProces_Img

CS: Msg: “Mod_Img” “change_prm”Interactive Mode

CS: Msg + Q:“Mod_Img”

Modify Compression (+ F_C)CP: “(Tx_Img)´”

(Proces_Img)´

CP: (Tx_Img)´´

CS: Msg: “Balance_TS”TSR processing

(TSR)

CS: Msg confirmation

Post Proces: PSNR If > TSendIf <=T Q.M.)

CS: “Param_Img”

Quality Mngmt)

Note: All joint units receive the same code-stream but only IMN can interact with Sensor Node unless otherwise specified.

Quality Management (QM).(In terms of PSNR)

Reconstructed image JPEG2000 Decoded iaw Time Slots available+Original Image.

PSNR>T YN

Transfer toInteractiveto modifyCompression param.

EXAMPLE: -Size-Zoom-TILES-ROI

PSNR / MSE

*.J2C

Threshold: T(OperationalRequirement)

(1)

(2) (3)

Ready toTX L16

InteractiveMode

Y

Request moreLink 16 T-S :

AP 257-259-261J0.6 OR J0.7

(TSR)

N

Reduce T

N

Service MSG

Ready toTX L16

Y

Concept. Example 1

JPEG2000Coder

J2C bitstream

#1 #2 #3 #4

Mode LRCP:Quality Layers

#1

#2

#3

#4

NPG_V1 (Dedicated)

NPG_V2 (Dedicated)

NPG_V3 (TSR)

NPG_V4 (TSR)

Alg #1

Alg #2TSR

L16Transmission

Layers jpeg2000:

•Picture JPEG2000 coded•Progressive LRCP Mode: 4 layers•NGP allocation (if available): Packing Limit and Access Mode

•Layers 1 & 2: Dedicated Algorithm #1•Layers 3 & 4: TSR Algorithm #2

•Link 16 Transmission

Example 2: P/L increased by layer

JPEG2000Coder

Code streamJPEG2000: *.J2C

#1 #2 #3 #4

Progressive #1&#2:XYZT:

#1

#2

#3

#4

NPG_V1 (Ded)

NPG_V2 (Ded)

NPG_V3 (TSR)

NPG_V4 (TSR)

Alg #1

Alg #2

TXL-16

jpeg2000 layers:

#1’ #2’ #3’

#1’

#2’

#3’

1 2 Note:1.- QL Image 12.- QL Image 2

Img #1(FLIR_Land)

Img #2(FLIR_Naval)

Img #1.j2c

Img #2.j2c

JPEG2000TransCoder

(MGMI)

TS available: 128•Img#1: 3 layers (96 TS/27.26 dB)•Img#2: 1 layer (32 TS/23.21 dB)

JPEG2000 & Link 16 Combined Error Detection and Correction Tecniques

Bit to Chip for MSK Modulation

Spread Spectrum: FH-DS

Symbol interleaving

Jitter at the start of the MSG

EDC Techniques: -Reed-solomon-Polinomial

LINK 16

JPEG2000Error D&C

Packet Organization:Markers: SOP-EPH

SEGMARK-ERTERM-RESTART

MSG Type-Std-P2SP-P2DP-P4

Block Coding

PacketSequentialOrganization:-Resincronization.

Error Resilient Mode (Symplified).• JPEG2000 Coded Image: Always include SOP-EPH

markers in the code-stream.Much better syncronization mechanism.

• Error Management Protocol Implemented.• Monitor Link 16 Parameter MER (Message Error Rate)

– Provides Reed-Solomon Coder Errors and Erasures.

“If MER >Threshold (iaw AJ enviroment): Retransmit same packet with a more robust P/L”.– Increase AJP.– Decrease BPP.

Commitment: Binary Rate AJ Protection

Error Recovery. An Example (1)

Pq#1 Pq#2 Pq#3 Pq#4 Pq#N....

STD

X- TS

P2DP

Y- TS

P4

Z- TSCode stream partFor Quality of Service

P4_NEDC

Q- TS

JPEG2000: j2c code-stream

LINK 16

X-TS

Packetwitherrors

Time Slot with errors

TS retransmitted with PL STD

..............................

JPEG200 Code-stream

#Bytes P/LResilient Mode # Time Slots P/L No resilient Mode #Time Slots

MAIN HEADER 96 STD 4 STD 4

TILEPART #1 12 STD 1 STD 1

PACKET #1 HEADER 11 STD 1 P2 1

PACKET#1 13 P2 1 P2 1

……………….. …. …………… …………. ……………… …

TOTAL Time Slots 25 24

1 packet header corrupted +1 TIME SLOT

Error Recovery. An Example (2)True JPEG2000 Coded Image

Trama Codificada JPEG2000

#Bytes Packing Limit Resilient # Time Slots Re-TX Delta #Time Slots

MAIN HEADER 96 STD 4 STD =

TILEPART #1 12 STD 1 STD =

PACKET #1 HEADER 11 STD 1 STD =

PACKET#1 13 P2 1 STD (0.462)= 1 (+0)

PACKET #2 HEADER 17 STD 1 STD =

PACKET#2 38 P2 1 STD (1.3511)= 2 (+1)

PACKET #3 HEADER 17 STD 1 STD =

PACKET#3 95 P2 2 STD (3.3777)= 4 (+2)

PACKET #4 HEADER 17 STD 1 STD =

PACKET#4 207 P4 2 P2DP (3.68)= 4 (+2)

PACKET #5 HEADER 17 STD 1 STD =

PACKET#5 392 P4 4 P2DP (6.96)=7 (+3)

PACKET #6 HEADER 17 STD 1 STD =

PACKET#6 440 P4 4 P2DP (7.82)=8 (+4)

TOTAL Time Slots 25 37 (+12 time slots)

ALL packet corrupted +12 TIME SLOTS

Some Simulations and Results

• Great amount of simulations has been conducted:– One NPG STD P/L (by default)– Two NPGs STD/P2 – Three NPGs STD/P2/P4– Four NPGs STD/P2/P4/P4NEDC

• DLWG: Only one NPG allocated for imagery (NPG11)– But several NPGs available for future use.

• Packing limit can also be changed before each Link 16 transaction via AP: Only one NPG will be required More “host overhead”.

119 TS/3 Layers/ 31.53 dB.

140 TS/ 9 Layers/ 37.97 dBm

FLIR Original Image

32 TS (STD) / 1 Layer/ Cr=706.71

128 TS (P4) / 3 Layer/ Cr=78.5

Change in ResolutionAlgorithm

stepPixels bytes BPP Time Slots. NPGs/PL

Original Image 640x480 921600 1

#1 Resolution: 4 40x30 3600 0.7138 4 NPG V1 (STD-TSR)

#2 Resolution 3 80x60 14400 0.7277 +16 =20 NPG V1 (STD-TSR)

#3 Resolution 2 160x120 57600 0.7851 +34 (TSR) =54 NPG V2 (P2DP-TSR)

#4 Resolution 1 320x240 230400 0.8830 +37 (TSR) =91 NPG V3 (P4-TSR)

Original Resolution 1 Resolution 2 Resolution 3

JPEG In Noisy Channels BER: 10^-4Original JPEG

JPEG 2000 In Noisy Ch. BER: 10^-4

ORIGINAL

JPEG JPEG2000

Original SAR Image: 8 bpp

JPEG2000 SAR Images

1 Layer / 30 TS (P4)/0.05088 bpp/18.84 dB 5 Layers/ 80 TS (P4)/ 0.1384 bpp/ 19.84

JPEG2000 Filtered SAR Images

Image BPP PSNR(Filt)

PSNR(Orig)

T/S P4 (Filtd)

T/S P4(Orig)

Original

8 bpp - -

CompLay #1

0.0505 22.97 18.84 30 30

CompLay#5

0.1395 25.48 19.84 81 80

CompLay#10

0.5143 -- 22.30 --- --

SAR Image median filtered. 5 layersPSNR regarding to filtered image

SAR Image. Region Of InterestLAYERS BPP PSNR db Link-16 Time Slots

Layer 1 0.02783 64

Layer 2 0.083497 64+64=128

Layer 3 0.1948341 10.93 128+32=160 > 154

Layer #1 Layer #3

SAR Image. Tiling

Lay bitrate TS PSNR Tiles

1 0.0139 32 10.39 T0

2 0.0695 96 11.44 T0 a T3

3 0.1252 128 12.60 T0 a T5

4 0.18 144 20.07 T0 a T8

Layer #1 layer #2

Layer #3 layer #4

Summary (1)

• Recent conflicts have shown that real time imagery availability is one of the most demanding capabilities required by operational/tactical commanders.

• Integrating “imagery content” into the “common tactical picture” provides a better target identification, weapon release criteria, damage assessment and surveillance data in support of network centric warfare. Also avoids “request for frequency allocation.”

• Current efforts are focused mainly to support JPEG/EagleEye compressed images packed into J16.0 messages and released in the NPG 11.

Summary (2)• JPEG2000 is a more robust open source standard that produces a

totally embedded code-stream. • Method proposed to send the JPEG2000 image in a antijamming

adaptative way– Most important part is more protected.– Rest of the image in a less protected packing limit.

• Algorithm Proposed to adapt JPEG2000 quality layers to Link 16 available time slots.

• Results shown a very good compression ratio with less than 10% and good performance with 5% of the available Link 16 time slots.

• JPEG2000 also offers a great variety of new capabilities well suited for tactical exploitation such as– ROI– Watermarks– Geo-referencing– More robust error and detection schemes.

Way Ahead

• Develop a technological demonstrator– Plan on going.

• Propose an ad-hoc J message J XX.YY or modify J16.0 specification for including JPEG2000 codification syntax.

• Implement a GML compatible geo-referencing capabilities as a part of XML boxes available at JPX file format.

• Implement security and encryption capabilities iaw JPSEC file format.

• MPEG2000 for stream-video under assessment.• And more…

Some of these new proposals require an “enhance throughput” Link 16 capability.

QUESTIONS ?QUESTIONS ?