Page 1
Using Graphics Rendering Contexts to Enhance the Real-Time Video Coding
for Mobile Cloud Gaming
Shu Shi †, Cheng-Hsin Hsu ‡, Klara Nahrstedt †, Roy Campbell †
† Dept. of Computer Science University of Illinois at Urbana-Champaign, USA
ACM Multimedia, Scottsdale, Nov 29, 2011
‡ Deutsche Telekom Inc. R&D Lab, USA (Now at NTHU, Taiwan)
Tuesday, November 29, 11
Page 2
What is Cloud Gaming?
Tuesday, November 29, 11
Page 3
What is Cloud Gaming?
http://www.wallcoo.net/game/car_race_series/html/wallpaper11.html
Tuesday, November 29, 11
Page 4
What is Cloud Gaming?
http://www.wallcoo.net/game/car_race_series/html/wallpaper11.html
Tuesday, November 29, 11
Page 5
What is Cloud Gaming?
BroadbandInternet
Tuesday, November 29, 11
Page 6
What is Cloud Gaming?
BroadbandInternet
Tuesday, November 29, 11
Page 7
What is Cloud Gaming?
BroadbandInternet
Mobile Cloud Gaming
Tuesday, November 29, 11
Page 8
Mobile Cloud Gaming
Rendering Results
RenderingServer
Mobile Client
Wireless Networks
User Interaction
RenderingViewpoint
Control Updates
3D Game Scene Data
Display
Tuesday, November 29, 11
Page 9
Mobile Cloud Gaming
Tuesday, November 29, 11
Page 10
Mobile Cloud Gaming
• Research Prototype
• Remote Rendering Games for iPhone
• BZFlag: an open source 3D tank shooting game
Tuesday, November 29, 11
Page 11
Mobile Cloud Gaming
• Research Prototype
• Remote Rendering Games for iPhone
• BZFlag: an open source 3D tank shooting game
• Demo Video
Tuesday, November 29, 11
Page 12
Tuesday, November 29, 11
Page 13
Tuesday, November 29, 11
Page 14
Benefits
Tuesday, November 29, 11
Page 15
Benefits
No Need of Powerful Graphics Hardware
Same Game Console 3D Experience
Cross-Platform Solution
Easy Distribution
Prevent Pirating
... ...
Tuesday, November 29, 11
Page 16
Challenges
Interaction Latency
Tuesday, November 29, 11
Page 17
Rendering Results
RenderingServer
Mobile Client
Wireless Networks
User Interaction
RenderingViewpoint
Control Updates
3D Game Scene Data
Display
Challenges
Interaction Latency
Tuesday, November 29, 11
Page 18
Rendering Results
RenderingServer
Mobile Client
Wireless Networks
User Interaction
RenderingViewpoint
Control Updates
3D Game Scene Data
Display
Challenges
Interaction Latency
Tuesday, November 29, 11
Page 19
Rendering Results
RenderingServer
Mobile Client
Wireless Networks
User Interaction
RenderingViewpoint
Control Updates
3D Game Scene Data
Display
Challenges
Interaction Latency
Tuesday, November 29, 11
Page 20
Challenges
Interaction Latency
• [Shi10] Shu Shi, Mahsa Kamali, Klara Nahrstedt, John Hart, Roy Campbell. A High-Quality Low-Delay Remote Rendering System for 3D Video. In ACM Multimedia (MM’10)
• [Shi09] Shu Shi, Won Jeon, Klara Nahrstedt, Roy Campbell. Real-Time Remote Rendering of 3D Video for Mobile Devices. In ACM Multimedia (MM’09)
Tuesday, November 29, 11
Page 21
Challenges
• Interaction Latency
• [Shi10] Shu Shi, Mahsa Kamali, Klara Nahrstedt, John Hart, Roy Campbell. A High-Quality Low-Delay Remote Rendering System for 3D Video. In ACM Multimedia (MM’10)
• [Shi09] Shu Shi, Won Jeon, Klara Nahrstedt, Roy Campbell. Real-Time Remote Rendering of 3D Video for Mobile Devices. In ACM Multimedia (MM’09)
Network Bandwidth
Tuesday, November 29, 11
Page 22
Network Bandwidth Usage
Mobile Devices
Video Requirements
Raw Bandwidth
320x240 @ 30fps 26.4 Mbps
640x480 @ 30fps 105 Mbps
1280x720 @ 30fps 316 Mbps
Tuesday, November 29, 11
Page 23
Real-Time Video Coding
Tuesday, November 29, 11
Page 24
Real-Time Video Coding
• Low Coding Efficiency
• No bi-directional prediction
• Strict encoding deadline
• One-pass rate control
• Low buffer occupancy
• Intra refresh coding
• ... ...
Tuesday, November 29, 11
Page 25
Real-Time Video Coding
• Low Coding Efficiency
• No bi-directional prediction
• Strict encoding deadline
• One-pass rate control
• Low buffer occupancy
• Intra refresh coding
• ... ...
• How bad is it?
Tuesday, November 29, 11
Page 26
Real-Time Video Coding
Tuesday, November 29, 11
Page 27
0 10 20 30 40 50 60012345678910
Time (s)
Bit
Rat
e(M
bps)
DownlinkMean
OnLive Video Bandwidth Usage
Real-Time Video Coding
Tuesday, November 29, 11
Page 28
0 10 20 30 40 50 60012345678910
Time (s)
Bit
Rat
e(M
bps)
DownlinkMean
OnLive Video Bandwidth Usage
Real-Time Video Coding
Tuesday, November 29, 11
Page 29
0 1 2 3 4 5 6 6.505
10152025303540455055
Bit Rate (Mbps)Q
ualit
yin
PSN
R(d
B)
40 dB: Excellant Quality
x264 Encoding Performance
0 10 20 30 40 50 60012345678910
Time (s)
Bit
Rat
e(M
bps)
DownlinkMean
OnLive Video Bandwidth Usage
Real-Time Video Coding
40dB: Excellent Quality
Tuesday, November 29, 11
Page 30
0 1 2 3 4 5 6 6.505
10152025303540455055
Bit Rate (Mbps)Q
ualit
yin
PSN
R(d
B)
40 dB: Excellant Quality
x264 Encoding Performance
0 10 20 30 40 50 60012345678910
Time (s)
Bit
Rat
e(M
bps)
DownlinkMean
OnLive Video Bandwidth Usage
Real-Time Video Coding
40dB: Excellent Quality
Tuesday, November 29, 11
Page 31
0 1 2 3 4 5 6 6.505
10152025303540455055
Bit Rate (Mbps)Q
ualit
yin
PSN
R(d
B)
40 dB: Excellant Quality
x264 Encoding Performance
0 10 20 30 40 50 60012345678910
Time (s)
Bit
Rat
e(M
bps)
DownlinkMean
OnLive Video Bandwidth Usage
Real-Time Video Coding
6 - 7 times worse!
40dB: Excellent Quality
Tuesday, November 29, 11
Page 32
Real-Time Video Coding
• Low Coding Efficiency
• No bi-directional prediction
• Strict encoding deadline
• One-pass rate control
• Low buffer occupancy
• Intra refresh coding
• ... ...
Tuesday, November 29, 11
Page 33
Video Coding Basics
I B B B P B B B P
Tuesday, November 29, 11
Page 34
Real-Time Video Coding
1
Tuesday, November 29, 11
Page 35
Real-Time Video Coding
1 2
Tuesday, November 29, 11
Page 36
Real-Time Video Coding
1 2 3
Tuesday, November 29, 11
Page 37
Real-Time Video Coding
41 2 3
Tuesday, November 29, 11
Page 38
Real-Time Video Coding
541 2 3
Tuesday, November 29, 11
Page 39
Real-Time Video Coding
6541 2 3
Tuesday, November 29, 11
Page 40
Real-Time Video Coding
76541 2 3
Tuesday, November 29, 11
Page 41
Real-Time Video Coding
876541 2 3
Tuesday, November 29, 11
Page 42
Real-Time Video Coding
9876541 2 3
Tuesday, November 29, 11
Page 43
Real-Time Video Coding
... ...9876541 2 3
Tuesday, November 29, 11
Page 44
Real-Time Video Coding
... ...9876541 2 3
Tuesday, November 29, 11
Page 45
Real-Time Video Coding
... ...
• No B Frame!!!
PPPPPPI P P
Tuesday, November 29, 11
Page 46
Real-Time Video Coding
P ... ...BBBPB1 B B
Tuesday, November 29, 11
Page 47
Real-Time Video Coding
... ...1 BB B P BBB P
Tuesday, November 29, 11
Page 48
Real-Time Video Coding
... ...1 BB BP BBBP
Tuesday, November 29, 11
Page 49
Auxiliary Frame Generation
... ...1 BB B BBBP P
Tuesday, November 29, 11
Page 50
Auxiliary Frame Generation
... ...1 BB B BBB
P P
Tuesday, November 29, 11
Page 51
Auxiliary Frame Generation
1
Tuesday, November 29, 11
Page 52
Auxiliary Frame Generation
1 AUX1
1
Tuesday, November 29, 11
Page 53
Auxiliary Frame Generation
1 AUX1 2
1 P
1
AUX1
Tuesday, November 29, 11
Page 54
Auxiliary Frame Generation
31 AUX1 2
1
AUX1
1 P B
Tuesday, November 29, 11
Page 55
Auxiliary Frame Generation
431 AUX1 2
1 P B
1
AUX1
B
Tuesday, November 29, 11
Page 56
Auxiliary Frame Generation
5431 AUX1 2
1 P B B
1
AUX1
B
Tuesday, November 29, 11
Page 57
Auxiliary Frame Generation
65431 AUX1 2
1 P B B B B
1
AUX1
Tuesday, November 29, 11
Page 58
Auxiliary Frame Generation
AUX25431 AUX1 2
1 P B B B B
Tuesday, November 29, 11
Page 59
Auxiliary Frame Generation
6AUX25431 AUX1 2
1 P B B B B P
AUX1
AUX2
Tuesday, November 29, 11
Page 60
Auxiliary Frame Generation
76AUX25431 AUX1 2
1 P B B B B P B
AUX1
AUX2
Tuesday, November 29, 11
Page 61
Auxiliary Frame Generation
76AUX25431 AUX1 2
1 P B B B B P B
AUX1
AUX2
8
B
... ...
Tuesday, November 29, 11
Page 62
Auxiliary Frame GenerationAUX AUX
1 P B B B B P B B B B
Tuesday, November 29, 11
Page 63
Auxiliary Frame GenerationAUX AUX
1 P B B B B P B B B B
Tuesday, November 29, 11
Page 64
Basics of Video Coding
• Motion Estimation
http://www.cmlab.csie.ntu.edu.tw/cml/dsp/training/coding/motion/me1.html
Tuesday, November 29, 11
Page 65
Basics of Video Coding
• Search Method and Range
http://www.cs.cmu.edu/~yvchen/projects.html
Tuesday, November 29, 11
Page 66
3D Image Warping
Tuesday, November 29, 11
Page 67
3D Image Warping
• Input:
• Image: I1
• Depth: D1
• Viewpoints:
• v1
• v2
v1
v2
Tuesday, November 29, 11
Page 68
3D Image Warping
• Input:
• Image: I1
• Depth: D1
• Viewpoints:
• v1
• v2
• Output
• Image: I2
v1
v2
Tuesday, November 29, 11
Page 69
Double Warping
Tuesday, November 29, 11
Page 70
Double Warping
v1
v2
Tuesday, November 29, 11
Page 71
Double Warping
v1
v3
v2
Tuesday, November 29, 11
Page 72
Double Warping
v1
v3
v2
Tuesday, November 29, 11
Page 73
Double Warping
v1
v3
v2
Tuesday, November 29, 11
Page 74
Tuesday, November 29, 11
Page 75
Warping Based Real-Time Video Coding
I P B B B B P B B ... ...B B
AUX1AUX2
Tuesday, November 29, 11
Page 76
Warping Based Real-Time Video Coding
I P B B B B P B B ... ...B B
AUX1AUX2
Tuesday, November 29, 11
Page 77
Warping Based Real-Time Video Coding
R R W W W W R W W ... ...W W
AUX1AUX2
Tuesday, November 29, 11
Page 78
Warping Based Real-Time Video Coding
R R W W W W R W W ... ...W W
AUX1AUX2
Tuesday, November 29, 11
Page 79
Warping Based Real-Time Video Coding
R R W W W W R W W ... ...W W
AUX1AUX2
Tuesday, November 29, 11
Page 80
Results
30
32
34
36
38
40
42
44
46
1 1.5 2 2.5 3
Qua
lity
in P
SNR
(dB
)
Bit Rate (Mbps)
Original x264 Encoding with only I/P framesProposed warping based encoding method using auxiliary frames
30
32
34
36
38
40
42
44
46
1 1.5 2 2.5 3Q
ualit
y in
PSN
R (
dB)
Bit Rate (Mbps)
A short sequence of 315 frames
A long sequence of 4072 frames
Tuesday, November 29, 11
Page 81
Discussion
Tuesday, November 29, 11
Page 82
Discussion
• Interaction Latency Reduction
Tuesday, November 29, 11
Page 83
Discussion
• Interaction Latency Reduction
• Limitations
• Only applicable to cloud gaming or other remote rendering applications
• Mostly useful for first person perspective games
• Can’t handle graphical effects well
Tuesday, November 29, 11
Page 84
Tuesday, November 29, 11
Page 85
Tuesday, November 29, 11
Page 86
Discussion
• Interaction Latency Reduction
• Limitations
• Only applicable to cloud gaming or other remote rendering applications
• Mostly useful for first person perspective games
• Can’t handle graphical effects well
• Optimizations
• Integration with x264
Tuesday, November 29, 11
Page 87
Conclusion
• Reduce video bandwidth for real-time coding in cloud gaming
• Generate auxiliary frames based on camera motion to enable B frame encoding
• Replace search based motion estimation with efficient 3D image warping using depth maps and camera vectors
• Think differently for different applications
Tuesday, November 29, 11
Page 88
Thank you!
• Acknowledgement
• The work is supported by Deutsche Telekom R&D Lab USA
• Contact Author
• Shu Shi: [email protected]
• Questions?
Tuesday, November 29, 11