N ETWORKed M EDIA L AB. D EPT. OF I NFO. & C OMM., K-JIST Video over IP as a VoIP extension JongWon Kim, Ph.D. Advanced Network Conference (APAN Busan.

NETWORKed MEDIA LAB.

DEPT. OF INFO. & COMM., K-JIST

Video over IP Video over IP as a VoIP extensionas a VoIP extension

JongWon Kim, Ph.D.

Advanced Network Conference (APAN Busan 2003)

Aug 27th, 2003

Networked Media LaboratoryDept. of Information & Communications

Kwang-Ju Institute of Science & Technology (K-JIST)[email protected]

http://netmedia.kjist.ac.kr



ContentsContents

Media Delivery over IP

Network-adaptive Media Deliveryo Networking Supporto Network Adaptation

Video over IP as a VoIP extension

Future of Video over IP - Toward Access Grid & Immersive Media



Multimedia Delivery over IP



Multimedia over IP NetworksMultimedia over IP Networks

Media BroadcastingHosted StreamingInteractive Conferencing

TNR

Internet

StreamingSender

Manager

Proxy

Proxy

Receivers

Conferencing



Multimedia Delivery - Multimedia Delivery - TechnologiesTechnologies

Efficient Media Representation Techniques: Scalable source representation to deal with dynamically varying

bandwidth

Robust source representation to deal with high error/loss rate

Dynamic and Reliable Media Delivery Techniques: Network adaptation to map scalable and robust source

representations onto the MM characteristic-aware transport protocols

Adaptive resource provisioning/management to guarantee the desired QoS / multicast service

Universal Media Access with Security Techniques: Unified integrated media support with a scalable information

structure and a choice of efficient file format for MM

End-system and application customization with security concerns



0%

Pac

ket

Lo

ss

Command/ control (eg Telnet, Interactive

games)

Conversationalvoice and video

Voice/videomessaging

Streamingaudio/video

Transactions (eg E-commerce,Web-browsing, E-

mail access)

Paging, Downloads

(eg FTP,still image)

Fax

Background(eg Usenet)

5%

100 msec 1 sec 10 sec 100 sec

Zeroloss

Interactive Responsive Timely Non-critical

Delay

Multimedia Application’s QoS RequirementsMultimedia Application’s QoS Requirements



Media Delivery – Application vs Media Delivery – Application vs NetworkNetwork

Application/Source

Network

Network Characteristics and Policies: Rate (CBR, VBR, ABR, …), Error

(random/bursty in BER/PLR/CRL), Delay/Jitter (tolerance)

Content-aware Transport

Application/Source Characteristic :

steady/burst, real-time?, loss-resilient?

Network-adaptive Delivery



Network Adaptive Media Delivery

- Networking Support Part -



Media Delivery – Adaptation & Media Delivery – Adaptation & SupportSupport

Media Representation(Compression)

Error Control

MediaSynchronizati

on

Inter-operableProtocol

SystemSupport

(Transport,OS,

Storage)

Congestion(Rate, Flow)

Control

DynamicNetwork

AdaptationFramework

NetworkMulticastSupport

Network QoSSupport

OtherMiddleware

Support



Inter-operable Delivery with Inter-operable Delivery with Protocol SupportProtocol Support

Standard-based vs. Proprietary Protocols? Related issue: Standardized media content

representation (e.g. MPEG-X) vs. Others?

Security Protocols

RSVP

IPv4, IPv6

TCPUDP

RTCP RTSPSIPSAP

SDP

RTPHTTP SMTP

RTP Payload

Media Contents Conference ControlShared Tools



End-to-end QoS guarantee with

QoS provisioning (admission control and QoS controls) & Network adaptation

The Internet

Economical and reliable network with

Scalable multicasting & Optimized server location and load sharing

including contents routing

Paid services withCopyright management,

authentication,billing, etc

Required Network Infrastructure: Multicast, QoS,

and others

Content broadcast network



Multicast Support for Multipoint Multicast Support for Multipoint Media DistributionMedia Distribution

Multicast streaming?Model (ASM, SSM, ALM/Overlay/P2P)

Reliable Multicast TransportMobile/Wireless MulticastResilient Multicasting of continuous media

I nternetI nternet

232/ 8232/ 8

I GMPv3I GMPv3I GMPv3I GMPv3

Source filtering Source filtering capabilitycapability

Source filtering Source filtering capabilitycapability



QoS-network Support for Quality Media DeQoS-network Support for Quality Media Deliverylivery

QoS Model: IntServ, DiffServ, Traffic Engineering (MPLS), …

QoS signaling

Reservation and Allocation

DiffServNetworkMIBMIB MIBMIB

WFQEF

BEWFQ

EF

BE

Console orClient User

JAVA InterfaceProgram

MIBMIB

ResourceManager-aware

Application

SLA

Resource Manager

RouteDB

Interface

Mon.

DB

K-JIST KISTI

KOREN21

Configuration

Reservationrequest

QoS monitor

BrowsingT

CP

/IP

TC

P/IP

BBTPAuth.

module

Route Info

moduleCAC

moduleMonitor-

ingmodule

Execution module SNMP module

SLA module Re source



Toward Consolidated Support Toward Consolidated Support via Middleware?via Middleware?

Issues to be covered +++o Identificationo Directories, …o Security: Authentication, Authorization, …o …

Middleware:o Specialized networked services that are shared by

applications and userso A set of core software components that permit

scaling of applications and networks

GRID Middleware Home Networking Middleware



Network Adaptive Media Delivery

- Network Adaptation Part:Bandwidth (rate), loss, and delay -



Media Delivery – Adaptation & Media Delivery – Adaptation & SupportSupport

Media Representation(Compression)

Error Control

MediaSynchronizati

on

Inter-operableProtocol

SystemSupport

(Transport,OS,

Storage)

Congestion(Rate, Flow)

Control

DynamicNetwork

AdaptationFramework

NetworkMulticastSupport

Network QoSSupport

OtherMiddleware

Support



Required Network Adaptation Required Network Adaptation FunctionalitiesFunctionalities

In case of Reliable Unicast with feedbacko Network Congestion Control => Quality Adaptation: Source Rate Control / Network Rate Adapt

ationo Error control: Error Resiliency and Error Recoveryo Quality Recovery: Post-processing & Concealment o Synchronization Control & Adaptive Playout

Media Pump

Adaptable Source

Congestion Control

(TCP-friendly)

Client

Quality Adaptation

Network

Quality Recovery

Error Control

Synchronization Control



TCP-Friendly Congestion Control TCP-Friendly Congestion Control / Rate Control/ Rate Control

TCP-Friendly Congestion Control: equation-based …

Toward unified congestion control handling: Congestion Manager (CM)

Congestion Control / Rate Control / Flow Control

Multicast congestion control?



TCP-Friendly Congestion Control TCP-Friendly Congestion Control / Rate Control (Cont.)/ Rate Control (Cont.)

Rate Control byo Source Rate Control (on-line

encoding only)o Source Rate Adaptation (or Shaping)

o Smoothing – via Bufferingo Packet Schedulingo Pre-fetching

I

B B B B B B

PP

Td 2Td 3Td 4Td 5Td 6Td 7Td 8Td 9Td

B B B B B BP P

Td0 2Td 3Td 4Td 5Td 6Td 7Td 8Td 9Td

I

Ravg

Time

Time

Transmission Rate

Transmission Rate Non-Smoothed

VBR stream

Smoothed near-CBR stream

I

B B B B B B

PP

Td 2Td 3Td 4Td 5Td 6Td 7Td 8Td 9Td

B B B B B BP P

Td0 2Td 3Td 4Td 5Td 6Td 7Td 8Td 9Td

I

Ravg

Time

Time

Transmission Rate

Transmission Rate Non-Smoothed

VBR stream

Smoothed near-CBR stream

I B B P B B P B B P B B I B B P

New connectionrequest

I-pictureStarting point

Superposed stream(existing sources)

period

(1) (2) (3) (4)

time

Forward Proxy (Delegate)

Reverse Proxy

(Surrogate)

Web Server

+ Content Distribution+ + Proxy Plug-ins

Forward Proxy (Delegate)

Reverse Proxy

(Surrogate)

Web Server

+ Content Distribution+ + Proxy Plug-ins



Proactive and Reactive Error Proactive and Reactive Error ControlsControls

Network error controlso Proactive (FEC), Reactive (ARQ), and Hybrido Delay is key controlling factor

Network (Channel) error control vs Source error resiliency?

Quality Recovery: Error Concealment and Post-processing



Source EncoderNetwork

Adaptation SinkDecoderNetwork

Adaptation

Network FramingApplication

FramingCoding

Error Detection

Loss Detection

Error Correction

Erasure Correction

CRC

Sequence No.

Retransmission

FEC

Quality Control(Layered Coding)

Data Structuring(Synchronization)

Resynchronization

Loss Concealment

Post Processing

Courtesy of Dr. Noh NVW 2001

Error Control and Quality Error Control and Quality RecoveryRecovery



Synchronization Control and Adaptive PlaySynchronization Control and Adaptive Playoutout

Adaptive playouto Utilizing silenceo Utilizing time-scale modifica

tion

Synchronizationo Intra-mediao Inter-mediao Inter-client

o Tight vs Loosely-coupled



Media Delivery – Network Media Delivery – Network AdaptationAdaptation

< Source prioritization for content-aware network adaptation >

MM Content

Scalable streamin rate/error

Integrated &

prioritizedprioritized stream

Channelstream Network

Source encoding

Source transcoding &Framing/Multiplexing

Trans-MUX &Channel codingwith FEC/ARQ

Transmission

Feedback from Host/Application

Feedback from Network



Wired/WirelessNetworks

Layered Video Encoding Receiver/User

NetworkAdaptation

Network Adaptation &Prioritized Packetization

VideoPre-

process-ing

R-DAnalysis

Layereden-

coding

TargetMinimumBuffer Size(Receiver

Constraint)

TargetMinimumBandwidth(Network

Constraint)Frame

Complexity(Quality

Constraint)

ConstantQualityRate

Control

(SourceRate/ErrorResilience)

(only ifapplicable)

PrioritizedPacketi-zation

LayeredRPI

NetworkMonitoring

&FeedbackHandling Application Feedback

EstimatedAvailable

Bandwidth,Loss/Delay

Network Feedback

PrioritizedLayeredPackets

R-D /Corruption

Model

Network Feedback (end-to-end)

NetworkAdaptation

(NetworkFiltering)

ReceiverSelection

Media Delivery – Network Adaptation

NetworkAdaptation

(NetworkRate/Loss/

Delay)

NetworkAdaptation

(Receiver /User

Adaptation)

Corrup-tion

Analysis



Video over IP as VoIP extension (V2oIP)



VoIP Extension: Voice + Video + DataVoIP Extension: Voice + Video + Data

We want to achieve IP TELEPHONY (a.k.a. voice/video/data convergence): Real time voice, video and data communication over Internet Protocol (IP) networks

However, latest high-end VC system supportso H.263+ or H.263++ (w/ Annex U for background restoration)o 384 kbit/s or 768 kbit/so TDM/ISDN (H.320) or IP (H.323)o CIF (352x288) or CIF-Interlace 2x(352x288)o Capture and rendering typically interlaceo 25/30 fps, 50/60 fields per secondo Round-trip delay around 400ms +o Round-trip delay for multipoint (w/ MCU) more than one second



VV22oIP Building BlocksoIP Building Blocks

SIP MGCP H.323

Proxy Announcement Toolkit Multi-point Control Gatekeeper Toolkit

Service Creation Gate-

keepers GatewaysCall AgentsAnnouncement

Player Conference Bridges

Service Creation Gate-

keepers Gateways

Protocol stacks (H.323, MGCP, SIP) Switching cores (e.g. Gatekeepers, Call Agents) Media engines (Announcement, Mixing, Gateway) Service creation environment



MCU

Multicast Audio and Video

Unicast Audio and VideoDecentralized Centralized

POTS

xDSL

Cable

PSTN

BackendServers (services)

GKC/BEC

AdministrativeDomain A

AdministrativeDomain A

AdministrativeDomain B

GKA/BEA BEB/GKB

H.323Entity

TA

H.323Entity

TB

H.323Entity

TC

GKC1

GKA1 GKB1

Non-H.323Entity

TP

GW

BECHNon-H.323System

ClearingHouse

Signaling

PhysicalConnection

VV22oIP DeploymentoIP Deployment



Video Quality and Function Issues (1)Video Quality and Function Issues (1)Improve Video Quality and Size

Upcoming H.264 video encodingo Product design stage now, 640x480 at full frame rate on one high-end DSPo Bit rate around 384 kbit/s yields acceptable picture qualityo Subjective quality significantly better than H.263++ w/ CIF at same bit rate

Variable, content-dependent capture rateo Linking with region-oriented (maybe combined with segmentation) encoding on top of variable f

rame skipping

Wide-screen video support o Can support video resolution like 2560x480 with single CCD camera with custom, wide-angle

lens; Multiple DSPs for distributed codec is required; 1.5 – 3 Mbps Progressive scan hardware

o No A/D and D/A losses; Better quality and good delay characteristics; 250ms round-trip delay over IP demonstrated?

Enhance error resilience with ERPS and so on



Video Quality and Function Issues (2)Video Quality and Function Issues (2)Provide more natural, cooperative environment for multi-

site conferencing

Let’s avoid transcoding and enhance user experienceo Create all representations needed in the sending endpoint and simulcast them (in line wit

h layered multicast video) o Ideally suited for IP multicast environment; But can be with overlay (application-layer) multi

cast; o May need some innovative changes in the control and mux protocolso No transcoding artifacts whatsoevero No additional end-to-end delay for multipoint when using IP multicast

Let’s get rid of fixed environment that discourages uncooperative use of the technologyo Toward flexible meeting environment without fixed camera position, fixed background, stud

io lighting, seating

Let’s add more functionalities while keeping user interface simple and straightforward



Automatic Automatic Broadcast Broadcast and and Recording Recording

Note takingNote taking

Remote ParticipationRemote Participation

VoIP Extension: Real-time CollaborationVoIP Extension: Real-time Collaboration Bring Voice, Screen, Computation together to improve: Real-time C

ollaboration Vision from Microsofto Rich presence and IMo PC and phone integrationo Data/screen conferencingo Information agento Enhanced Meetings

incoming incoming communicommunicationcation

to task to task listlist

to web to web phonephone

contextcontext

2003 Gurdeep Singh Pall from Microsoft Corporation



Future of Video over IP

- Access Grid & Immersive Media -



The Access GridThe Access GridAccess Grid does for people what the computation

al Grid does for machinesMuch more than teleconferencing, if

possible at commodity pricesTo enable groups of people to interact with Grid res

ources and to use the Grid technology to support group to group collaboration at a distanceo Distributed Lectures and seminarso Remote participation in panel discussionso Virtual site visits meetingso Complex distributed grid based demonstrations



Usage AreasUsage Areas The Academic and Research, Government, Private Sectors Making Remote Collaborations Work across Boundaries Check “Multi-Sector Collaboration over the Access Grid” by J. T. von H

offman (Boston Univ)”



Access GridAccess GridComponents of an AG NodeComponents of an AG Node

Digital Video

Digital Video

Digital Audio

NETWORK

MixerControl

Computer

NTSC Video

RGB Video

Analog Audio

Video Capture

Computer

DisplayComputer

AudioCapture

Computer

EchoCanceller

Ambient mic(tabletop)

Presentermic

Presentercamera

Audience camera

Stream

TypeMax.

latencyMax. Jitter

Min. BWMax

. Loss

Multicast

Text 100 ms N/A 64 kbps 0 % No

Audio 400 ms 60 ms 64 kbps 5 % Yes

Video 400 ms 30 ms 256 kbps x 4 = 1 Mbps25 %

Yes

Requires Multicast



AG extended toward ACEAG extended toward ACE

Immersive Media SupportedAdvanced Collaboration Environment

Immersive Media SupportedAdvanced Collaboration Environment

Advanced Collaboration EnvironmentAdvanced Collaboration Environment

Access Grid 2.0Access Grid 2.0

Access Grid

Enable group-to-group interactions at a distance

Improve the user experience Enable complex multi-site visual and

collaborative experiences Build on integrated grid services

architecture Use quality but affordable digital IP based

audio/video



Broadband Content Requires Broadband Content Requires More… More…

Music CDs160 Kbps

Music

Browsing144 Kbps

½ Screen Video300Kbps

TV Shows

Full Screen600Kbps – 1.5Mbps

Live Events

High Resolution 6-8 Mbps

Pay Per View

ImmersiveMedia

> 100 Mbps

What kinds of broadband contentso High speed Internet access: WWW + Mail + Messagingo Video focused: Basic TV + Pay TV + Personalized TVo Networked games + Music downloads + Online gamblingo …



Future Broadband Contents Future Broadband Contents Service & Immersive MediaService & Immersive Media

Interactive collaboration - Collaborative design and engineering; Remote customer support; Distance learning

Remote visualization & Large-scale, multi-site computation and data mining

Shared virtual reality - Military and industrial team training / simulation; Multiplayer games; Virtual shopping malls (e-commerce); Mobile entertainment; Online tradeshows and conferences; Virtual Heritage

Any combination of the above



Core Techniques for Immersive Media Service Integration

Multi-modal Immersive Multi-modal Immersive Media Service TechnologyMedia Service TechnologyMulti-modal Immersive Multi-modal Immersive

Media Service TechnologyMedia Service Technology

3D Audio Generation

& Reconstruction

3D Geometry Generation & Modeling

Digital Contents Management &

Protection

Multi-D Haptic Generation & Reconstruction

Immersive Media

Compression & Integration

3D Video Synthesis &

Reconstruction

Immersive MediaDelivery

Immersive Media Storage &

Query/Browsing

Contents Service (Protocol &

System)

Immersive Media Content

Adaptation

Interactive User Interface



Thank You!Thank You!

Send Inquiries to Send Inquiries to jongwonjongwon@@kjistkjist.ac..ac.krkr

N ETWORKed M EDIA L AB. D EPT. OF I NFO. & C OMM., K-JIST Video over IP as a VoIP extension JongWon Kim, Ph.D. Advanced Network Conference (APAN Busan.

Documents