Top Banner
EE569 Digital Video Processin EE569 Digital Video Processin g g 1 Roadmap Roadmap Introduction Introduction Intra-frame coding Intra-frame coding Inter-frame coding Inter-frame coding Object-based and scalable video Object-based and scalable video coding coding * * Why object-based? Why object-based? motion segmentation, shape coding, R-D motion segmentation, shape coding, R-D optimization optimization scalability issues scalability issues Spatial/temporal/quality scalabilities Spatial/temporal/quality scalabilities
65

EE569 Digital Video Processing 1 Roadmap Introduction Intra-frame coding Inter-frame coding Object-based and scalable video coding* –Why object-based?

Dec 30, 2015

ReportDownload

Documents

  • RoadmapIntroductionIntra-frame coding Inter-frame codingObject-based and scalable video coding*Why object-based?motion segmentation, shape coding, R-D optimizationscalability issuesSpatial/temporal/quality scalabilities

    EE569 Digital Video Processing

  • Object-based Video CodingWaveform-based coding discussed so far uses a simple source model (e.g., H.261/263/264, MPEG-1/-2)Does not consider the semantic content (e.g. objects and their shape) of the videoObject-based video coding identifies objects (or regions) in a video and encodes them. Potential benefits may includeImproved coding efficiencyImproved visual quality (e.g., no blocking artifacts)Content descriptionContent-based interactivityAlso called content-dependent video codingThe buzz word for MPEG-4 but less successful than expected (so the important question is to understand why it does not work so well)

    EE569 Digital Video Processing

  • Essential Tasks in Object-based Video CodingObject/region segmentationSeparate pixels based on their color, texture, motion characteristicsClosely related to motion detection and segmentationIntrinsically ill-defined and desperate for a breakthrough2D shape modeling and codingNot all shapes are equally probableSubtle implications into video coding (hidden pitfalls)2D texture modeling and codingExtension of existing block-based MCP into region-basedDeformable textures (tradeoff between spatial and temporal prediction)

    EE569 Digital Video Processing

  • Object/Region SegmentationThe major challenge in content/object-based codingCommon approaches for segmentation in a still image: gray-level thresholding, clustering, edge detection, region growing, splitting and mergingObject segmentation in videoMotion information can be utilized, but how?Should we trust more on motion or spatial clues?

    EE569 Digital Video Processing

  • Motion-based SegmentationMotion-based segmentation: to segment an image using motion informationWe can first estimate the motion field and then segment the motion fieldHowever, estimation and segmentation are like two sides of the same coin +

    EE569 Digital Video Processing

  • A Mind-bothering ExampleFrame 1Frame 2It is easy to convince yourself that tree branches are moving,But how do we know the sky is still? What if it were also movingat the same speed (shouldnt we observe the same intensity patternsbecause sky is a smooth region)?

    EE569 Digital Video Processing

  • Implications into Video CodingTrue motion representation might be useful to computer vision and motion perception, but it is not indispensable in video codingThe fundamental reason lies in the relationship between motion representation and video coding: how to tolerate the uncertainty in motion?The same issue remains in object-based image coding: how to tolerate the uncertainty in shape? (we will discuss this in more detail later)

    EE569 Digital Video Processing

  • Simplified Segmentation: Change DetectionTo detect the changing parts in a video, from time ti to time tj , we compute a difference image and threshold the difference by T

    dij(x,y) can be further processed, e.g., to remove isolated 1s, or to group 1s that are close by to each otherf (x, y, tj)f (x, y, ti)

    EE569 Digital Video Processing

  • Change Detection: Pros and ConsSimple to implement; fastDetects all changesDetects even unwanted changesPositive and negative changes detected (occlusion)Difficult to quantify motionRequires a static reference frame

    EE569 Digital Video Processing

  • Change Detection: An ExampleMonitor the traffic

    EE569 Digital Video Processing

  • If without a static reference frameBackground extraction methodsAd-hoc median detector (your CA#6)To eliminate the impact of (small) moving objects, use the robust estimator approach to iteratively remove the outliersMore sophisticated approaches involve the modeling of background by mixture of Gaussian distributions and graph-cut based optimization

    EE569 Digital Video Processing

  • Simplified Segmentation: Global Motion EstimationPlanar homography (feature-based)Homogeneous coordinates Conditions for planar homographyHomography estimation from feature correspondenceHierarchical model-based GME (feature-less)Directly minimize an energy function (the MSE of MCP errors)Solve the optimization problem in a coarse-to-fine fashion (more robust and efficient)

    EE569 Digital Video Processing

  • Plane Homography

    EE569 Digital Video Processing

  • Model-based GMETarget function for minimizationSolution: Gauss-Newton methodwhereBergen, J. R., Anandan, P., Hanna, K. J., and Hingorani, R. Hierarchical Model-Based Motion Estimation. In Proc. of the Second European Conference on Computer Vision, pp. 237-252, 1992

    EE569 Digital Video Processing

  • Multi-resolution GME

    EE569 Digital Video Processing

  • Numerical Example

    EE569 Digital Video Processing

  • Summary for Change Detection and Global Motion EstimationMotion segmentation becomes relatively easier to solve when either camera is still or background objects belong to a planeLatest advances include a joint motion segmentation and estimation using level-set methods (PDE-based formulation)

    Mansouri, A.-R.; Konrad, J., "Multiple motion segmentation with level sets," Image Processing, IEEE Transactions on , vol.12, no.2, pp. 201-220, Feb 2003

    EE569 Digital Video Processing

  • 2-D Shape Modeling and CodingBitmap coding: a binary map specifying whether or not a pixel belongs to an objectA special case of the general alpha-map

    Contour coding: code only the contour of the object or the regionChain codesPolygon approximationSpline approximation

    EE569 Digital Video Processing

  • Image Matting (Soft segmentation)Not for coding but for interactive editing

    EE569 Digital Video Processing

  • 2-D Texture Modeling and Coding*Shape-adaptive DCT

    Shape-adaptive wavelet transform

    EE569 Digital Video Processing

  • RoadmapIntroductionIntra-frame coding Review of JPEGInter-frame codingConditional Replenishment (CR)Motion Compensated Prediction (MCP)Scalable video coding3D subband/wavelet coding and recent trend

    EE569 Digital Video Processing

  • Scalable vs. MulticastWhat is scalable coding?MulticastScalable codingforeman.yuvforeman128k.codforeman256k.codforeman512k.codforeman1024k.codforeman.yuvforeman.cod1024512256128

    EE569 Digital Video Processing

  • Spatial scalability

    EE569 Digital Video Processing

  • Temporal scalabilityFrame 0,1,2,3,4,5,Frame 0,2,4,6,8,Frame 0,4,8,12,30Hz15Hz7.5Hz

    101110101000110100

    EE569 Digital Video Processing

  • SNR (Rate) scalabilityPSNRavg=30dBPSNRavg=35dBPSNRavg=40dBPSNRi: PSNR of frame i

    101110101000110100

    EE569 Digital Video Processing

  • Scalability via Bit-Plane CodingA=(a0+a12+a222+ +a727)Least Significant Bit (LSB)Most Significant Bit (MSB)ExampleA=129 sign=+,a0a1a2 a7=10000001 sign=-, a0a1a2 a7=00110011 A=-(4+8+64+128)=-204sign bit

    EE569 Digital Video Processing

  • Why DPCM Bad for Scalability?Base layerEnhancement Layer 1Enhancement Layer 2IbasePPPIenh1Ienh2123Frame numberPPPPPPsuffer from drifting problemsuffer from coding efficiency loss

    EE569 Digital Video Processing

  • Fine Granular Scalability (FGS)H.264 with/without FGS optionForeman sequence (5fps)~2dB gap

    EE569 Digital Video Processing

  • 3D Wavelet/Subband Codingtxy2D spatial WT+1D temporal WT

    EE569 Digital Video Processing

  • Wavelet Video CoderTemporal Wavelet TransformSpatial Wavelet TransformLLLLLHLHLHOriginal video framesEmbeddedQuantization &Entropy Coding[Taubman & Zakhor, 1994] [Ohm, 1994] [Choi & Woods, 1999] [Hsiang & Woods, VCIP 99] . . . and others

    EE569 Digital Video Processing

  • Motion-Adaptive 3D Wavelet TransformRecall Haar transformMotion-adaptive Haar transformW,W-1: forward and backward motion vectorlifting-based implementation

    EE569 Digital Video Processing

  • Lifting

    EE569 Digital Video Processing

  • MC Wavelet Coding vs. H.264/AVC2.01.81.61.41.21.00.80.60.40.236343230282624222038Luminance PSNR (dB)bit-rate (Mbps)ScalableMC 5/3 WaveletNon-scalableH.264/AVCSequence: Mobile CIF

    H.264/AVC high complexity RD control CABAC PBBPBBP . . . 5 prev/3 future reference frames data courtesy of M. Flierl[Taubman & Secker, VCIP 2003] courtesy D. Taubman

    EE569 Digital Video Processing

  • Wavelet Synthesis with Lossy Motion VectorMC WaveletTransformMotionEstimatorEmbeddedEncodingEmbeddedEncodingDecoderDecoderInverseWaveletTransformVideo inVideo out[Taubman & Secker, ICIP03]MinimizeJ=D+lRMinimizeJ=D+lR

    EE569 Digital Video Processing

  • R-D Performance with Lossy Motion Vector[Taubman & Secker, VCIP 2003] courtesy D. Taubman

    EE569 Digital Video Processing

  • Surprising Success of ITU-T Rec. H.263??What H.263 was developed for . . . Analog videophone. . . and what is was used for.

    EE569 Digital Video Processing

  • What is Streaming Video?AccessSWData pathDomain ADomain BDomain CInternetAccessSWSourceReceiver 2Receiver 1Download mode: no delay bound Streaming mode: delay boundcnn.comRealPlayer

    EE569 Digital Video Processing

  • OutlineChallenges for quality video transportAn architecture for video streamingVideo compressionApplication-layer QoS controlContinuous media distribution servicesStreaming serverMedia synchronization mechanismsProtocols for streaming mediaSummary

    EE569 Digital Video Processing

  • Time-varying Available Bandwidth Data pathDomain ADomain BAccessSWSourceReceiver56 kb/sR>=56 kb/sR
  • Time-varying Delay Data pathDomain ADomain BAccessSWSourceReceiver56 kb/scnn.comRealPlayerDelayed packets regarded as lost

    EE569 Digital Video Processing

  • Effect of Packet LossData pathDomain ADomain BAccessSWSourceReceiverNo packet lossLoss of packetsNo retransmission

    EE569 Digital Video Processing

  • Unicast vs. MulticastUnicastMulticastPros and cons?

    EE569 Digital Video Processing

  • Heterogeneity For MulticastDomain ADomain BDomain CInternetSourceReceiver 1Receiver 2

    AccessSWGatewayEthernetTelephonenetworksReceiver 364 kb/s1 Mb/s256 kb/sNetwork heterogeneityReceiver heterogeneityWhat Quality?What Quality?

    EE569 Digital Video Processing

  • OutlineChallenges for quality video transportAn architecture for video streamingVideo compressionApplication-layer QoS controlContinuous media distribution servicesStreaming serverMedia synchronization mechanismsProtocols for streaming mediaSummary

    EE569 Digital Video Processing

  • Architecture for Video Streaming

    EE569 Digital Video Processing

  • Video CompressionLayered CoderLayer 0Layer 1Layer 21 Mb/s256 kb/s64 kb/sLayered video encoding/decoding. D denotes the decoder.

    EE569 Digital Video Processing

  • Application of Layered VideoDomain ADomain BDomain CInternetSourceReceiver 1Receiver 2

    AccessSWGatewayEthernetTelephonenetworksReceiver 364 kb/s1 Mb/s256 kb/sIP multicast

    EE569 Digital Video Processing

  • Application-layer QoS ControlCongestion control (using rate control): Source-based, requiresrate-adaptive compression or rate shapingReceiver-basedHybridError control:Forward error correction (FEC)RetransmissionError resilient compressionError concealment

    EE569 Digital Video Processing

  • Congestion Control Window-based vs. rate control (pros and cons?)Window-based controlRate control

    EE569 Digital Video Processing

  • Source-based Rate Control

    EE569 Digital Video Processing

  • Video MulticastHow to extend source-based rate control to multicast?Limitation of source-based rate control in multicastTrade-off between bandwidth efficiency and service flexibility

    EE569 Digital Video Processing

  • Receiver-based Rate ControlDomain ADomain BDomain CInternetSourceReceiver 1Receiver 2

    AccessSWGatewayEthernetTelephonenetworksReceiver 364 kb/s1 Mb/s256 kb/sIP multicast for layered video

    EE569 Digital Video Processing

  • Error ControlFECChannel codingSource coding-based FECJoint source/channel codingDelay-constrained retransmissionError resilient compressionError concealment

    EE569 Digital Video Processing

  • Channel Coding

    EE569 Digital Video Processing

  • Delay-constrained Retransmission

    EE569 Digital Video Processing

  • OutlineChallenges for quality video transportAn architecture for video streamingVideo compressionApplication-layer QoS controlContinuous media distribution servicesStreaming serverMedia synchronization mechanismsProtocols for streaming mediaSummary

    EE569 Digital Video Processing

  • EE569 Digital Video Processing

  • Continuous Media Distribution ServicesContent replication (caching & mirroring)Network filtering/shaping/thinningApplication-level multicast (overlay networks)

    EE569 Digital Video Processing

  • CachingWhat is caching? Why using caching? WWW means World Wide Wait?Pros and cons?

    EE569 Digital Video Processing

  • OutlineChallenges for quality video transportAn architecture for video streamingVideo compressionApplication-layer QoS controlContinuous media distribution servicesStreaming serverMedia synchronization mechanismsProtocols for streaming mediaSummary

    EE569 Digital Video Processing

  • Streaming ServerDifferent from a web serverTiming constraintsVideo-cassette-recorder (VCR) functions (e.g., fast forward/backward, random access, and pause/resume).Design of streaming serversReal-time operating systemSpecial disk scheduling schemes

    EE569 Digital Video Processing

  • Media SynchronizationWhy media synchronization?Example: lip-synchronization (video/audio)

    EE569 Digital Video Processing

  • Protocols for Streaming VideoNetwork-layer protocol: Internet Protocol (IP) Transport protocol: Lower layer: UDP & TCPUpper layer: Real-time Transport Protocol (RTP) & Real-Time Control Protocol (RTCP)Session control protocol: Real-Time Streaming Protocol (RTSP): RealPlayerSession Initiation Protocol (SIP): Microsoft Windows MediaPlayer; Internet telephony

    EE569 Digital Video Processing

  • Protocol Stacks

    EE569 Digital Video Processing

  • SummaryChallenges for quality video transportTime-varying available bandwidthTime-varying delayPacket lossAn architecture for video streamingVideo compressionApplication-layer QoS controlContinuous media distribution servicesStreaming serverMedia synchronization mechanismsProtocols for streaming media

    EE569 Digital Video Processing

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.