Efficient Tiling For Video Analytics Maureen Daum, Brandon Haynes, Amrita Mazumdar, Magda Balazinska, Alvin Cheung 1 Paul G. Allen School of Computer Science & Engineering, University of Washington 1 Department of Electrical Engineering and Computer Sciences, University of California, Berkeley Motivation Query: Run license plate detection on all cars. Decode the entire frame • Easy to store as encoded video • Decode many irrelevant pixels Decode only the car pixels • Difficult to store as encoded video • Decode only relevant pixels Use tiling to decode only the region of the frame that contains car pixels • Easy to store as encoded video • Decode few irrelevant pixels Decode car pixels Run license plate detector Strategy • Split up video frames into independently decodable regions called “tiles” • Set the tile layout using one of the following approaches: Approach 1: Uniform tiles Approach 2: Non-uniform tiles around objects 2.1: Large tiles around groups of objects 2.2: Small tiles around individual objects • Set the layout for a group of frames and update periodically • Speed up queries by only decoding the tiles that contain pixels for a given query Preliminary Results • Run queries on videos from the Netflix public data set 2 to decode pixels for particular object types (e.g. “person”, “car”) • Compare uniform tile layouts to layouts picked based on the locations of pixels being decoded • Study the effect of updating the custom layouts after different durations 2 https://github.com/Netflix/vmaf/blob/master/resource/doc/datasets.md Example video frame from UADetrac: http://detrac-db.rit.albany.edu Uniform tiles Tiles around the object being queried Tiles around an object other than the query object Effect of tiling on decode time Observations • Custom tile layouts reduce decoding time • Tile layouts optimized for pixels different from the ones being queried can hurt performance Positions in frames 1-3 Tile 0 Tile 1 Tile 2 Tile 3 Layout using large tiles This work is supported by the NSF through award CCF-1703051 Video storage and indexing for efficient query processing. Tile 0 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Layout using small tiles Tile 0 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Layout using uniform tiles Acknowledgements Effect of tiling on quality and storage size Observations • Custom tile layouts generally have better quality than uniform tiles (PSNR above 40 is considered lossless) • Custom tile layouts sometimes lead to larger storage sizes. The size of the tiles depends on how they are encoded Effect on PSNR Effect on storage size
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Efficient Tiling For Video AnalyticsMaureen Daum, Brandon Haynes, Amrita Mazumdar, Magda Balazinska, Alvin Cheung1Paul G. Allen School of Computer Science & Engineering, University of Washington1Department of Electrical Engineering and Computer Sciences, University of California, Berkeley
Motivation
Query: Run license plate detection on all cars.
Decode the entire frame• Easy to store as encoded video• Decode many irrelevant pixels
Decode only the car pixels• Difficult to store as encoded video• Decode only relevant pixels
Use tiling to decode only the region of the frame that contains car pixels• Easy to store as encoded video• Decode few irrelevant pixels
Decode car pixels Run license plate detector
Strategy• Split up video frames into independently decodable regions called “tiles”• Set the tile layout using one of the following approaches:
Approach 1: Uniform tilesApproach 2: Non-uniform tiles around objects
2.1: Large tiles around groups of objects2.2: Small tiles around individual objects
• Set the layout for a group of frames and update periodically• Speed up queries by only decoding the tiles that contain pixels for a
given query
Preliminary Results• Run queries on videos from the Netflix public data set2 to
decode pixels for particular object types (e.g. “person”, “car”)• Compare uniform tile layouts to layouts picked based on the
locations of pixels being decoded• Study the effect of updating the custom layouts after different
durations
2https://github.com/Netflix/vmaf/blob/master/resource/doc/datasets.md Example video frame from UADetrac: http://detrac-db.rit.albany.edu
Uniform tiles Tiles around the object being queried Tiles around an object other than the query object
Effect of tiling on decode time
Observations • Custom tile layouts reduce decoding time• Tile layouts optimized for pixels different from the ones being queried can hurt
performance
Positions in frames 1-3
Tile 0 Tile 1
Tile 2 Tile 3
Layout using large tiles
This work is supported by the NSF through award CCF-1703051
Video storage and indexing for efficient query processing.
Tile 0 Tile 1
Tile 2
Tile 3 Tile 4
Tile 5
Layout using small tiles
Tile 0 Tile 1 Tile 2
Tile 3 Tile 4 Tile 5
Layout using uniform tiles
Acknowledgements
Effect of tiling on quality and storage size
Observations • Custom tile layouts generally have better quality than uniform tiles (PSNR above 40 is
considered lossless)• Custom tile layouts sometimes lead to larger storage sizes. The size of the tiles
depends on how they are encoded
Effect on PSNR Effect on storage size
Related Documents
