7/31/2019 Ms Synchronisation
1/38
E0262 MIS - Multimedia Pla back S stems
Prof Pallapa Venkataram,
Electrical Communication Engineering,
Indian Institute of Science,
Bangalore 560012, India
Multimedia StreamsSynchronization
7/31/2019 Ms Synchronisation
2/38
E0262 MIS - Multimedia Pla back S stems
Synchronization in multimedia systems referes to thetemporal relations between media stream data units in
multimedia system.
Time dependent media data units: Temporal relations
between consecutive units of media stream exist.Time independent media data unit is any kind of
traditional media like text and image.
7/31/2019 Ms Synchronisation
3/38
E0262 MIS - Multimedia Pla back S stems
Classification of Media Use
7/31/2019 Ms Synchronisation
4/38
E0262 MIS - Multimedia Pla back S stems
Multimedia Synchronization The process of maintaining the temporal order of one or several media
streams
Relations relevant for audio and video exchange:
Intra-stream synchronization: Voice samples and video frames need
to arrive in time at the receiver before display or playout time to
maintain the continuity of playback.
Inter-stream synchronizationis needed to present voice and video
samples in a certain relation.
Spatial synchronization:Participants in a multimedia conference
should receive audio-visual data at the same time although
geographically distributed.
Life & Synthetic synchronization
7/31/2019 Ms Synchronisation
5/38
E0262 MIS - Multimedia Pla back S stems
Basic Synchronization Issues
Content Relations: define a dependencies of media data
units on some data
Spatial Relations: usually as layout relationships define the
space used for the presentation of a media data units on
an output device at a certain point of time in a multimedia
presentation.
Temporal Relations: define the temporal dependencies
between the more than one media stream data.
7/31/2019 Ms Synchronisation
6/38
E0262 MIS - Multimedia Pla back S stems
Sources of Asynchrony Different delays:Assumption of independent network connections
imposes different delays.
Network jitter:asynchronous data transfer destroys synchrony.
End-system jitter:packetizing and depacketizing of media data
units with different size due to encoding introduces jitter as well as
passing media units through the lower protocol layers.
Clock driftbetween the clocks in the servers and in the client is
present because we do not assume global clocks.
Change of the average delay:the synchronization scheme has to
be adaptive with respect to a change of the average delay.
Server drop outs:due to process scheduling are a realistic
assumption when using non-real-time operating systems.
7/31/2019 Ms Synchronisation
7/38E0262 MIS - Multimedia Pla back S stems
Types of Synchronization Time based synchronization
To synchronize and to disseminate multimedia data using time as a
major parameter
Synchronization is achieved by mean of electronic time signals
generated ad hoc by devices directly connected to a primary time
standard and transmitted by radio or by cable.
Time signals classes: video or sound signals (called right time
signals) for low accuracy manual synchronization, digital signals for
automatic synchronization with medium-high accuracy
Passive synchronization techniques: Do not use a time signalgenerated ad hoc, instead it is used a signal generated for other
purpose
7/31/2019 Ms Synchronisation
8/38E0262 MIS - Multimedia Pla back S stems
Types of Synchronization Delay based synchronization
Schemes compensate for these delays by computing well-definedstarting times for each stream server of the application.
Allow to initiate the synchronized playback of a media stream that is
composed of several multimedia streams.
Jitter based synchronization
jitter is experienced by media units traveling from the source to the
destination.
To smoothen out jitter, elastic buffers are required. These schemes guarantee a smooth playback of the stream
very low buffer requirements.
7/31/2019 Ms Synchronisation
9/38E0262 MIS - Multimedia Pla back S stems
Intra-Stream Synchronization
Audio and video streams are isochronous in nature.
processing and network delay jitter (i.e., the variance indelay);
variations in rates of recording and playback; and
unreliable transmission of stream data units. Solutions:
buffer monitoring
feedback technique global clock
7/31/2019 Ms Synchronisation
10/38E0262 MIS - Multimedia Pla back S stems
Inter-Stream Synchronization
A temporal relationship may exist at the source
between multiple continuous streams
multiplexing of streams
aggregation in one data structure
global clocks synchronization marker
synchronization channel
feedback technique
7/31/2019 Ms Synchronisation
11/38E0262 MIS - Multimedia Pla back S stems
Spatial Synchronization
all participants in the conference receive the audio
and video data at the same time, to maintain a fairconference
Based on global clocks synchronization channel or
feedback techniques as proposed for inter-stream
synchronization.
When global clocks are available, mechanisms
based on these clocks can achieve the most
accurate spatial synchronization
7/31/2019 Ms Synchronisation
12/38E0262 MIS - Multimedia Pla back S stems
Synchronization Handlers
roles : to identify the stakeholders involved, as well as the
prohibition and policy concepts.
Roles of these components in handling synchronization
Service provider
Network provider
End-user roles
7/31/2019 Ms Synchronisation
13/38E0262 MIS - Multimedia Pla back S stems
Service Provider Intra-stream synchronization: SP manipulates incoming
audio/video streams so that outgoing streams are within the
10 ms jitter boundary;
Inter-stream synchronization: SP will manipulate incoming
audio/video streams so that related outgoing audio and
video streams are within the -20 ms and +40 ms range;
Spatial synchronization: SP is responsible for ensuring that
outgoing audio/video streams are played out simultaneously
at the multiple users within the 0.25 s boundary.
7/31/2019 Ms Synchronisation
14/38E0262 MIS - Multimedia Pla back S stems
Network Provider Compulsory network service:a transportation service with a
deterministic service is provided
Statistical reliable network service:a transportation service
with a certain percentage of QoS violations is provided.
Best effort network service:the request from a client for a
certain transport service is evaluated against the current
network traffic.
7/31/2019 Ms Synchronisation
15/38E0262 MIS - Multimedia Pla back S stems
End-User With respect to the display of audio and video it is important
that the following requirements to be met:
lip-synchronization is a well known requirement and should
be in the -20 ms to +40 ms range;
audio or video jitter should be within the range of 10 ms;
loss of video frames or audio samples is tolerable when less
than 1% of the total sent;
spatial synchronization should be in the range of -0.25 s to
+0.25 seconds.
7/31/2019 Ms Synchronisation
16/38
E0262 MIS - Multimedia Pla back S stems
End-User
compulsory end-user service:in this case the
synchronization requirements must be met. statistical reliable end-user service:a certain percentage of
violations of the synchronization requirements is allowed.
best effort end-user service:possibilities to full the
synchronization requirements are based on currentprocessing and storage activities.
User policies are often application dependent.
7/31/2019 Ms Synchronisation
17/38
E0262 MIS - Multimedia Pla back S stems
Synchronization Methods Startup Synchronization
Constant delay and zero jitter first nmedia units
where tiindicates expected arrival time of the MDU iat the
client-site and t0
is the playout start time of the Io
th stream
Evaluation phase:round trip delays for each of the stream ofthe application are calculated, while
Synchronization phase:the starting time for each server is
calculated and transmitted back to the servers.
7/31/2019 Ms Synchronisation
18/38
E0262 MIS - Multimedia Pla back S stems
Synchronization Methods Startup Synchronization
Beginning of first phase
Max of the round trip delay
for all nsubstreams
Second phase
Difference between
the arrival timesof arbitrary media units
7/31/2019 Ms Synchronisation
19/38
E0262 MIS - Multimedia Pla back S stems
Example of Startup Synchronization
7/31/2019 Ms Synchronisation
20/38
E0262 MIS - Multimedia Pla back S stems
Example of Startup Synchronization
7/31/2019 Ms Synchronisation
21/38
E0262 MIS - Multimedia Pla back S stems
Intra-Stream Synchronization
Jitter based Intra-Stream Synchronization
Worst case estimate for
error and buffer req.
Interdependencies:
7/31/2019 Ms Synchronisation
22/38
E0262 MIS - Multimedia Pla back S stems
Inter-Stream Synchronization Inter-stream synchronization example that shows temporal
relations in a multimedia presentation including audio, video,
animation, and picture frames
7/31/2019 Ms Synchronisation
23/38
E0262 MIS - Multimedia Pla back S stems
Inter-Stream Synchronizations Live Synchronization:Maintains the temporal relations as
they existed during the capturing process
Synthetic Synchronization: Temporal relations are artificiallyspecified.
S h i i R f M d l
7/31/2019 Ms Synchronisation
24/38
E0262 MIS - Multimedia Pla back S stems
Synchronization Reference Model
7/31/2019 Ms Synchronisation
25/38
E0262 MIS - Multimedia Pla back S stems
7/31/2019 Ms Synchronisation
26/38
E0262 MIS - Multimedia Pla back S stems
S h i ti M d l
7/31/2019 Ms Synchronisation
27/38
E0262 MIS - Multimedia Pla back S stems
Synchronization Models The Temporal Synchronization Model
7/31/2019 Ms Synchronisation
28/38
E0262 MIS - Multimedia Pla back S stems
Synchronization Models
7/31/2019 Ms Synchronisation
29/38
E0262 MIS - Multimedia Pla back S stems
Synchronization Models Unix Synchronization Model
7/31/2019 Ms Synchronisation
30/38
E0262 MIS - Multimedia Pla back S stems
7/31/2019 Ms Synchronisation
31/38
E0262 MIS - Multimedia Pla back S stems
Object Data Stream Synchronization Technique
7/31/2019 Ms Synchronisation
32/38
E0262 MIS - Multimedia Pla back S stems
Object Data Stream Synchronization Technique Synchronization point isa point held by all participating media streams
needing to be synchronized
Synchronization points defined on the composite stream specify the
places that synchronous presentation must be checked and maintained.
Let sidenote the maximum time interval that media mican skip and di
denote the current delay time interval with the presentation of media
stream mi. We then have the following situations:
Deadline based Scheduling for Sync
7/31/2019 Ms Synchronisation
33/38
E0262 MIS - Multimedia Pla back S stems
Deadline based Scheduling for Sync nMDUs in a multimedia document, set of mchannels in
network
Effective bandwidth rate cjper channel and transit delay j,
aggregate capacity is not sufficient to provide inter-stream and
intra-stream synchronization without pre-fetching.
If the ithMDU and playout deadline i
is scheduled for
transmission on thejthchannel at timej, according to some
scheduling policy , then its arrival time, ai, at the user site
becomes
Feedback Techniques for Sync
7/31/2019 Ms Synchronisation
34/38
E0262 MIS - Multimedia Pla back S stems
Feedback Techniques for Sync
The resynchronization policies are:
Conservative:This scheme reacts only when playout ofdifferent streams is guaranteed to be asynchronous,
Aggressive:In this approach the server reacts as soon as
there is even a slight chance that playback is asynchronous
Probabilistic:The server reacts on the average (assuming
the network delay distributions and playout rate variations
are known).
Virtual Local Time
7/31/2019 Ms Synchronisation
35/38
E0262 MIS - Multimedia Pla back S stems
Virtual Local Time
Virtual local time
Does not rely on a synchronized network clock, which is not
always feasible to procure on the Internet,
Can be implemented with a minimum overheads, and
Can produce a reasonably good quality even when the
network delay jitter is very high.
Different Playout Strategies
7/31/2019 Ms Synchronisation
36/38
E0262 MIS - Multimedia Pla back S stems
Different Playout Strategies
7/31/2019 Ms Synchronisation
37/38
E0262 MIS - Multimedia Pla back S stems
Synchronized multicast media streaming framework
7/31/2019 Ms Synchronisation
38/38
E0262 MIS - Multimedia Pla back S stems
y g