1 Ubiquitous Multimedia Computing and Communication: Challenges and Future Trends C.-C. Jay Kuo University of Southern California
1
Ubiquitous Multimedia Computing and Communication:
Challenges and Future Trends
C.-C. Jay Kuo
University of Southern California
2
Content
Introduction Review of embedded system design
– Choice of RISC, VLIW and ASIC– Software or hardware?– Low power design
Several new design issues– Design of multi-format codec– Joint compression/encryption algorithm– Joint R-D-C optimization
Conclusion
3
Mobile Multimedia
Features– Portability
• Pervasive computing• Ubiquitous computing
– Communication capability
• 2G/2.5G/3G• WLAN, Bluetooth, UWB• Overlay networks
– Multimedia capability• A/V capturing• A/V display
– Marketing• Consumer electronics
oriented• Rather than PC-oriented
Embedded processors
4
Main Challenges
00111010001111010110100
wireless channels
power management
security and rights issue
Poor Channel Conditions:Low SNRMulti-path FadingDoppler Fading
5
Embedded Computing Systems
Two-module solution– Communication module
• RF-Modem• Baseband processing-channel decoding, despreading,
etc.
– Multimedia module• Codec, DRM, etc.
Embedded Media Processor ArchitectureCPU
DSP
Image Co-Processor
On-Chip Memory
CPUImage Co-Processor
On-Chip Memory
6
Standards of Mobile Communications
Resource: http://plus.ric.co.jp/wireless/wl003_01_0410.html
Eff
ecti
ve D
ista
nce
Data Rate
7
Trends of Mobile Communications (2)
Orthogonal Frequency Division Modulation Wireless Systems
– Wi-Fi (IEEE 802.11a/g/n)
– WiMax (IEEE 802.16)
– DVB-H Broadcasting Systems
8
Trends of Mobile Multimedia Applications
Two-way communication– Video telephony– 3G provides the solution
Web access– Short video clips for on-demand service
• Download or streaming
– Video gaming– Something better than 3G is desirable
Live video broadcasting– DVB-H or DMB
• Vision: Bring live TV programs to your cellular phone
9
Content
Introduction Review of embedded system design
– Choice of RISC, VLIW and ASIC– Software or hardware?– Low power design
Several new design issues– Design of multi-format codec– Joint compression/encryption algorithm– Joint R-D-C optimization
Conclusion
10
Tradeoff between Different Architectures
Dedicated coprocessorsASIC
Specific processorsSIMD, VLIW
General processorsCISC or RISC
Power consumption, chip area
Fle
xibi
lity
11
CISC versus RISC
Main difference– CISC: multi-clock complex instructions
• Minimizing the no. of instructions per program, while sacrificing the no. of cycles per instruction
• Example: Intel Pentium
– RISC: one-clock reduced instructions• Go to the opposite direction of CISC• Example: ARM
Advantages of RISC design– Lower power (suitable for mobile applications)– Lower chip area (for lower cost)
12
Single Instruction Multiple Data (SIMD)
Why is SIMD?- Multimedia data’s low-precision
- 8-bit pixels for image/video application- 16-bit samples for audio application Challenges: representation, storage and processing
- Multimedia algorithm’s inherit data parallelism- Add, subtract, and simple forms of multiplication and
division are common operations
First developed by UIUC – Used as imaging processing engine (CM series)
in early days
Popular engine: Intel MMX, TI iMX
13
Examples of Media Processors
Texas Instruments (TI)– DSC-25, DM-270, DM-320– OMAP for cellular phone– C64xx series
Intel– XScale Processor
Trimedia: TM1300– Speech/Image/Video– Somehow, not well received
Equator media processor
14
Integrated Solution to Video Coding
RISC (ARM processor)– Irregularity operations – Sequential computation (no inherent parallelism)– e.g. streaming parsing & entropy decoding
SIMD (Image co-processor)– Semi-regular operations– e.g. Intra prediction, quantization, motion
prediction/compensation
ASIC (dedicated coprocessors)– Regular yet computational intensive operations– To save SOC area and power – e.g. DCT, loop filtering, half-pel interpolation etc.
15
Content
Introduction Review of embedded system design
– Choice of RISC, VLIW and ASIC– Software or hardware?– Low power design
Several new design issues– Design of multi-format codec– Joint compression/encryption algorithm– Joint R-D-C optimization
Conclusion
16
Paradigm shift from hardware to software– Faster algorithm adoption– Easier for add-on functionalities
– Reuse of platform independent codes
Increased use of programmable processor core in system-on-chip (SoC)– Easier use of high level languages such
as C, C++
Advantages of Software Solution
17
Relative importance of embedded software development tools
Compiler for Embedded Processors
Need more sophisticated compiler to generate codes to meet stringent embedded application requirements
Use of C code versus assembly language by design team at
Northern Telecom area
18
Content
Introduction Review of embedded system design
– Choice of RISC, VLIW and ASIC– Software or hardware?– Low power design
Several new design issues– Design of multi-format codec– Joint compression/encryption algorithm– Joint R-D-C optimization
Conclusion
19
Why Multi-Codec Design?
Joint S/W & H/W Decoder
S/W ComponentsEntropy decoding
Selected mode decoding
H/W ComponentsInverse transform
Inverse quantizationMotion search
De-blocking filter
MPEG-2Stream
H.264/MPEG-4 AVC Stream
VC-1 Stream
OutputVideo/Audio
20
Flexible Design (1): Processor Consideration
Bitstream parsing– Sequential processing– Mutimedia standards define different bitstream
formats
Solution– Programmable architecture
• Pros: flexible, easy to be upgraded to compliance multiple standards
• Cons: not cost-effective on power and area
– Dedicated architecture• Pros: Save power, chip area• Cons: fixed, not easy to be modified
21
Flexible Design (2): Bus Consideration
Bus architecture– Integrating components on multimedia SoC
Multi-format codec is usually a hybrid SoC• RISC, SIMD, ASIC, on-chip memory
– Interfacing with off-chip memory
How to integrate efficiently?– Common bus architecture
• Flexible, accommodate to modules from all parties• Example: AMBA (used in ARM)
– Dedicated bus architecture• High utilization of bus bandwidth, cost-effective
22
Flexible Design (3) ---- A Case Study
“Real-time Audio/Video Decoders for Digital Multimedia Broadcasting”Victor H. S. Ha, Samsung, IWSOC’04
Common busAMBA
High bandwidthDedicated bus
23
Content
Introduction Review of embedded system design
– Choice of RISC, VLIW and ASIC– Software or hardware?– Low power design
Several new design issues– Design of multi-format codec– Joint compression/encryption algorithm– Joint R-D-C optimization
Conclusion
24
Confidentiality for Multimedia Data
What separates multimedia data from traditional alpha numeric data?– Large in file size– May require real-time processing (especially for
continuous media)– Portable and mobile applications
25
Approaches to Multimedia Encryption
Signal scrambling– Historical approach– Not compatible with modern multimedia
compression– Fast speed but low security
Total encryption with cryptographic ciphers– Trivial solution– High security but slow speed
Selective encryption– Most popular approach today– Limited in its range of application
Integrating encryption into entropy coding– Complementary to selective encryption– Very fast computation speed
Others
26
Selective Encryption
Select the most important coefficients and then encrypt them with traditional ciphers such as DES
Advantages– Lower complexity– High security level provided by traditional
cryptology– Less error correction coding redundancy– Compatible with existing software and hardware
modules
MediaCompression
System
Coefficient
Selection
Cryptographic
Cipher
ErrorCorrection
Coding
DigitizedAudiovisual
data
Coefficients SelectedCoefficients
Non-selectedCoefficients
Transmission channel or storage media
27
Example: Selective Encryption for G.723.1 Speech Coder
ITU-T Recommendation G.723.1– A popular low bit rate speech codec
Based on the human voice generation model– Vocoder– Decoder synthesizes speech using the model
LSPDecoder
PitchDecoder
ExcitationDecoder
+Synthesis
FilterPitch
Postfilter
LSPInterpolator
FormantPostfilter
Gain Scaling
Unit
LSP codebook indices
Lag of pitch predictorsGain vectors
Fixed codebook gainsand others
Vocal CordExcitation signal
generation
Vocal TractLinear filter
28
Effect of Encrypting Different Coefficients on Speech Intelligibility
Decoder Coefficient Type Size
(bits per frame)
LSP Decoder LSP codebook indices 24
lag of pitch predictors 14
Differential adaptive codebook lag 4Pitch Decoder
Pitch gain vectors 30
fixed codebook gains 18
Pulse positions index 76 / 48
Pulse sign index 22 / 16Excitation Decoder
Grid index 4
Original Speech
29
Effective Selective Encryption Scheme
Encrypt the most significant bits of all important coefficients given below
The best restoration approach– replace encrypted bits with the average value of its
rangeCoefficient Type Coefficient Name
LSP codebook indices
LPC_B23, LPC_B22, LPC_B21, LPC_B20,
LPC_B19, LPC_B18, LPC_B17, LPC_B16,
LPC_B15, LPC_B14, LPC_B13, LPC_B12,
LPC_B7, LPC_B6
lag of pitch predictorsACL0_B6, ACL0_B5, ACL0_B4,
ACL1_B6, ACL1_B5, ACL1_B4
Pitch gain vectorsGAIN0_B11, GAIN0_B10, GAIN1_B11, GAIN1_B10,
GAIN2_B11, GAIN2_B10, GAIN3_B11, GAIN3_B10
fixed codebook gainsGAIN0_B4, GAIN0_B3, GAIN1_B4, GAIN1_B3,
GAIN2_B4, GAIN2_B3, GAIN3_B4, GAIN3_B3
VAD mode flag VADFLAG_B0
Encrypt 37 bits/frame
roughly 20% of total encryption
30
Randomized Huffman Table Encryption
0
0
0 0
11
1
1 1
A
0 1
B CD E F G
0
1
0 1
10
1
1 0
A
0 1
B CD E F G
0 0
BADCAEFG
Huffman code #0 Huffman code #1
00000000
10011010
100011001010110111101111
110011001110110110111111
isomorphic tree!
31
Multimedia Encryption with Randomized Entropy Coder
Select a good PRBG Select an r-bit random seed s (encryption key) Pseudo-random sequence output from PRBG(s)
becomes the key hoping sequence (KHS)
Entropy
Coder
PRBGs KHS = 011000110 …
Input symbol
1110110001…
1011110
32
Simulation Results
Pentium III CPU, 600MHz, 256MB RAM Video compression standard: H.264 Cipher: the Randomized Huffman Table (RHT)
encryption PRBG: emulated by 128-bit MD5 hash function Test video clip: “foreman”
CIF size: 352 x 288 YUV 4:2:0 format The first 10 frames encrypted using key
0x246CCA6B103C95
33
Encoding Speed
Frame No.
Bit-stream size (no encryption)
Bit-stream size(encryption)
Encoding time (no encryption)
Encoding time (encryption)
Time increase
1
2
3
4
5
6
7
8
9
10
47848 bits
12712
15232
14776
16744
15384
11864
15640
15928
18184
47848 bits
12712
15232
14776
16744
15384
11864
15640
15928
18184
414 ms
338
350
351
358
350
330
356
360
365
415 ms
340
351
354
361
355
334
362
364
369
0.24% 0.59%
0.28%
0.85%
0.85%
1.41%
1.21%
1.68%
1.11%
1.09%
34
Video Clip “Foreman”
35
Standard H.264 Decoding
36
Attacking with Random Seed0x17460FD05B9EDF
37
Content
Introduction Review of embedded system design
– Choice of RISC, VLIW and ASIC– Software or hardware?– Low power design
Several new design issues– Design of multi-format codec– Joint compression/encryption algorithm– Joint R-D-C optimization
Conclusion
38
Research Motivation
There is no trivial relationship between quality (R-D performance) and complexity– Does the variable block size motion estimation always
help?• From 16x16 to 2 modes 16x16 and 8x8 (MPEG-4)• From 16x16 to 7 modes 16x16, 8x16, 16x8, 8x8, 4x8, 8x4 and 4x4
– Does the subpel motion search always help?• The subpel interpolation is one of the most time-consuming jobs in
the decoder implementation
– Does the deblocking filter always help?• The deblocking filer is another time-consuming job in the decoder
implementation
– Does the long-term memory always help?• This is probably not to be used in mobile video
The answer:– It is content dependent
39
Decoding-Friendly Encoder Design
The encoder selects the decoding-friendly modes– Stick to the integer motion pel– Stick to the option of no deblocking filter– Stick to the 1-reference frame– Stick to blocks of larger sizesif the other “fancier” choices do not help much
How to cast these in a formal framework– Rate-Distortion-Complexity optimization
40
Content
Introduction Review of embedded system design
– Choice of RISC, VLIW and ASIC– Software or hardware?– Low power design
Several new design issues– Design of multi-format codec– Joint compression/encryption algorithm– Joint R-D-C optimization
Conclusion
41
Conclusion (1)
Mobile multimedia communication is the major trend– Convergence of IT, CE, telecom, gaming, etc.
Requirements from consumers– Low power– Low cost– Broadband access– Reliability (quality)– Mobility
Requirements from company executives– Short design cycle, fast turn-around time– Fast adaptation to new markets
42
Conclusion (2)
Requirements from content owners– Digital Rights Management
Technology barriers and R&D opportunities– More flexible architectures
• Example: multi-format video decoder design (MPEG-2,H.264 and VC-1)
– Lightweight encryption algorithms• Examples: selective speech encryption, randomized
Huffman entropy coder
– Decoder-friendly coding methods• Example: Joint R-D-C optimization• Complexity is introduced effectively and in a controlled
fashion
43
Conclusion (3)
Technology barriers and R&D opportunities (Cont’d)– Broadband wireless communication
• More spatial diversity (MIMO) to be exploited– MIMO-OFDM
• New standard activities– 4G & 802.11n
– Cross-layer design• No clear layer boundary as observed in wired
communication systems• Integrated QoS across physical, MAC, transport,
application layers