Compression & Sampling - Howard Besser's Site

7/13/11

1

DigitalImaging—Compression&Sampling

Averyshortintroduc?on

HowardBesserNYUMovingImageArchiving&Preserva?onhHp://besser.tsoa.nyu.edu/howard/Talks/hHp://www.nyu.edu/?sch/preserva?on/

1Besser‐SOIMA2011‐Compression&Sampling

WhatisDigitalEncoding?‐

•  WhyDigitalisdifferentthanAnalog– DigitalAudioasexample

•  CompressionforAudioandS?llImages– FramesandFrameRate,BitRates,ChromaSubsampling

•  Theprocessofdigitalphotography•  DigitalCompressioncoding

•  Wrappers

•  (notdiscussingcopyphotography)2Besser‐SOIMA2011‐Compression&Sampling

Besser‐ICCROMRIO,6/8/073

SamplingSoundWaveoverTime


Frequency : every for seconds

DigitalSamplePointsonAnalogWave

DigitalAudioSamplingconvertfromcon?nuoustodiscrete

Besser‐SOIMA2011‐Compression&Sampling 5 Besser‐ICCROMRIO,6/8/076

Digi?za?on,samplingevery4seconds,1bit

Sample Time

Voltage Signal (in base 2)

0 4.3 1 4 6 1 8 6.5 1 12 1.5 0 16 4.2 1 20 4 1

Resolution

7/13/11

2


AudioDigi?za?onsebngsdependonseveralfactors

•  Thequalityontherecording,andthequalityyouneedforyourusers– Whenyouneedtocapturesoundsthatarelesscomplex(shorterandlesscomplexsounds),youcanloweryoursignalresolu?onand/orsamplingrate(thehumanvoiceonaninterviewrequireslesssignalresolu?on/fidelitythanmostmusic

•  Digi?za?onsebngsforspeecharerela?velylowunlessyouintendtocapturebackgroundsoundsaswell(backgroundmusicwhispers,gunshots,…)


Resolu'on

•  Humanspeechcanbeeffec?velyreproducedatarateof5.5kHz(kilo‐Hertz)

•  5500samples/sec

•  Mostnaturalworldsoundsandmediumfidelitymusiccanbereproducedat11kHzwithacceptablelossesoffidelity

•  11000samples/sec.

•  highfidelitymusicatCDaudioqualitythesamplingratemustbeabsoluteminimumof44.1kHz,givingaplaybackrateof22kHz,whichisjustabovethelimitofhumanhearing,butmostprofessionalandacademicorganiza?onsrecommendaboutdoublethat(96kHz).Butfor44.1kHZ:

•  44100samples/sec.•  22000samples/secisjustabovethelimitofhumanhearing


Quan'za'on

•  Thenumberofdis?nctsoundlevelsthatcanberepresentedisdeterminedbythenumberofbytesusedtostorethequan?za?onvalue.

•  themostcommonquan?za?onstrategyuses16bits(2bytes).Ex:11111111

•  216=65536discretelevels

Audio:Pulse‐CodeModula?on(PCM)

•  Samplingrate(numberofsamplespersecond)– 48kHz,96kHz

•  Bit‐depth(rangeofpossiblevaluesforeachsample)– 16bit,24bit

Besser‐SOIMA2011‐Compression&Sampling 10


Standards/BestPrac?cesforSoundDigi?za?on

•  Aremanagedbyprofessionalcommuni?eswithcommoninterests–  Interna?onalOralHistoryAssocia?onhHp://www.iohanet.org/resources/technical.html

–  Interna?onalAssocia?onofSoundandAudio‐VisualArchives(IASAA)hHp://www.iasa‐web.org/audio‐preserva?on‐tc04

–  SoundDirec,ons:BestPrac,cesforAudioPreserva,onh7p://www.dlib.indiana.edu/projects/sounddirec,ons/papersPresent/index.shtml

–  UniversityLibraries,…

S?llImageEncoding


7/13/11

3

FilmvsCCDArray


CCDforcapture


•  fromhHp://micro.magnet.fsu.edu/primer/digitalimaging/concepts/fullframe.html

Pixels


Pixels


Resolu?on

•  dpi,ppi,spi(scannervs.camera)


Example:50‐800ppi


7/13/11

4

Bit‐depth

•  8‐bit,24‐bit,36‐bit


Bit‐depthexample


Full‐colorspectrumvs8‐bits


RGBColor


Compressionforcapturingcon?nuous‐tonescenes

•  JPEG/JFIF– Predic?vecoding– DCT‐


Compressionforcapturingcon?nuous‐tonescenes

•  JPEG/JFIF– Predic?vecoding– DCT‐


7/13/11

5

ChromaSubSampling/DCTencoding

•  Mostly,eyeismoresensi?vetoluminencethantochroma

•  Divideimageintomatrix(8x8cellblocks;moreblocksforH.264[butalsoincludesub‐blocks])

•  Separateluminence(Y)fromChroma(Cb,Cr)

•  Encodeluminenceforeachindividualcell,butencodechromaforgroupsofcells(chromasubsampling)


Encodemoredataforluminencethanchroma

•  4:4:4isfullysampled

•  4:2:2useshalfthechromasamplesoneachchromachannel

•  4:2:0uses¼thechromasamplesoneachchromachannel

‐ImagefromTonyGladvinGeorge


DCTCodingar?facts


Ar?factsbecomelessvisibleaswezoomout


Ar?factsbecomelessvisibleaswezoomout


FileFormats

•  TIFF•  JFIF(JPEG)•  GIF•  J2K•  Raw•  DNG(Adobe)


7/13/11

6

Differentfileformats


OtherAr?facts:Aliasing

•  Aliasingduetohighfrequencyclearedupwithlow‐passfilteringbeforedigi?zing

‐hHp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/sab/report.htmlBesser‐SOIMA2011‐Compression&Sampling 32

CameraMetadata

•  XMP(Adobe’seXtensibleMetadataPlaworm)•  IPTC(Interna?onalPressTelecommunica?onsCouncil)–  hHp://www.iptc.org/site/Photo_Metadata/Embedded_Metadata_Manifesto_%282011%29

•  EXIF(ExchangableImageFileFormat)‐


EXIFmetadataexample


DigitalVideoCompression:Codecs

•  theschemeusedforcompressingandencodingthematerial.Usuallyrequiresdecodingsoywareorhardwaretoplayit.–  Hardwaredecodingindigitalvideotapeplayers–  Usuallysoywaredecodinginfile‐baseddigitalvideo

•  withwebbrowsers:mightbebuiltintobrowser(jpg)orrequirebrowserplug‐in(flv)

•  Otherwise,requiresplayersoywareorhardwarecard(J2K)•  Encodingisusuallyasymetrical

–  Takeslotsof?me&resourcestoencode

– …butdecodingcanbedonein(near)real?meonstandardwkst


MovingImageDigitalFiles

•  Eachframe(25‐30fps)isessen?allystoredsimilarlytoas?lldigitalimageframe(intermsofrepresenta?onandcompression)

•  Compressedformatssaveonstoragespacebynotsavingeverysingleframe– Maystoreonlythesmalldifferencesfromthepreviousframe

– Maystoreperiodicfullframesandpredict/interpolatebtwnthestoredframes

7/13/11

7

DigitalVideoCompression

•  Spa?alimagecompression(intraframe)•  Temporalmo?oncompression(interframe)

•  Andmostcompressionschemesgiveyouarangeofchoicesinhowheavilytocompress


Differentcompressionlevels(CarlFleischhauer)


Temporalmo?oncompression(interframe)‐basics


Interframecompression‐‐moredetails

•  I‐Frame(keyframe)—encodedass?llimage•  P‐Frame/Delta‐Frame—differencesfrompreviousframe

•  B‐Frame—a“virtual”framethataveragesdifferencebtwnframesbeforeandayer

‐imagefromPeHeriAimonenBesser‐SOIMA2011‐Compression&Sampling 40


Intra‐FrameCompression InterframeCompression

•  I‐Frame(orkeyframe)isonlyframethatdoesn’tdependuponotherframestodisplayit,butittakesupthelargestamountofstorage/bandwidth– NeedI‐Frametorecoverfromproblems

•  IntervalsbtwnI‐Frames(4,6,8,…,150)canbespecifiedinH.264orFLVformats


7/13/11

8

EncodingAr?facts&RelatedIssues

•  Ifevenonebitgoesbad,onecanlosesignalun?lnextkey‐frame(because,withinter‐framecompression,alltheframesbeforethenextkey‐frameonlycontainpar?alinforma?on)

•  Browsingwillonlyfindkey‐frames

•  Canseeablockingeffectofchromasubsamplingaroundthe?meoflumainterference‐


CodingAr?facts

"JensHenneman”,PerformanceArtpiecebyFredArmisen)careofPeterOleksik.720x980HDvideo,compressedusingMPEG2codec(23.98fps@2mbit/s),withonepassCBR


CodingAr?factsfollowcolorlightflash


CodingAr?factsfollowcolorlightflash


Tamiment‐35mmtelecinedtoDigibeta,thentranscodeduncompressedintocomputer.Then,filewascompressedusingMPEG2codec(3.7mbit/s)


AspectRa?oissuesTerryGilliam’sBrazil,digi?zed4:3VHSusingaDV50codecs?ll

displayedwithbadaspectra?o


7/13/11

9


VideoDigi?za?on&Transfer Framesize&rate

•  Framesize‐‐pixelarrayplusra?o(4:3,16:9)•  Framerate(29.97fps,24fps,…)

•  Somedigitalcompression&storageformatsrequireaspecificframesize


BitRate

•  Datatransferrate‐‐Amountofdatapersecond(comingacrossanetwork,orfromdisk/tapetoscreen)

•  UsuallymeasuredinMb/sec(8Mbps,30Mbps,50Mbps,200Mbps)– oyennotedas30Mbps(uncompressed),or8Mbps(compressed)


• AVI Microsoft format • MOV Quicktime format • MPEG compressed standardized format • RM Real Media compressed format • ASF Microsoft streaming format • DV25 25 Mbits/sec, fixed compression, 4:1:1 color

encoding, 3.6 MB per sec, 60 min. mini-DV cassette has 13 GB of storage • DV50 50 Mbits/sec, 4:2:2 or 3:3:1 color encoding • DV100 100 Mbits/sec, to be used for HDTV

production

From Steve Puglia from Adobe’s “A Digital Video Primer” and http://www.cit.cornell.edu/atc/materials/dig/videoformats.shtml

FileFormats

JimWheeler’slistofDigitalVideoFormats


MPEG

•  MPEG2‐‐currentlymostcommonlyusedfordistribu?on,butdifficulttouseforproduc?on

•  MPEG4‐‐usedwithQuickTimeandWindowsMediafiles

•  MPEG7&21‐‐metadatastandards,notusedforcompression

7/13/11

10

Encodingformats

•  MPEG‐1,op?mizedforCD‐ROMat1.5Mb/sec,1992,usedforVCDs

•  MPEG‐2,designedfor10Mb/sec,1994,usedforDVDsandmanybroadcasttvformats

•  MPEG‐4,madeforspeedsasslowas64Kb/sec,canseparateouts?lls/audio/video,goodforanima?onandstreaming

•  H.264(MPEG‐4part10),usedforBlu‐Rayandsomebroadcasttv

•  J2k(mo?onJPEG2000)usedinDigitalCinema(DCDM)•  WMV

•  .mov•  Flash(SWF),Real,…


Encodingformatgeneralproper?es

•  FrameIntegrity‐‐DPX2andJ2kencodeindividualframesasseparatefilesordis?ncten??es

•  Mostversa?le(varietyofimplementa?onchoices)‐‐Quick?meandMPEG‐4


Transcoding

•  Canalwaysbedonewithenoughhorsepower•  Butsomequalitycanbelost,dependinguponcompression,movingbtwncolorspaces(RGBandY’CbCr),etc.


Streaming

•  Latencyisprimaryissue(whichsome?mesnecessitatescaching)

•  FormatsincludeFlash(SWF7),Real,MPEG‐4


Wrappers

•  Forwrappingboththe“essence”andthemetadataaboutitintothesamecontainer

•  Wrapperscaneitherhaveeverythingembeddedwithinasinglefile,ortheycanhavepointerstofilesoutsidethewrapper

•  Wrappersinclude:–  MXF(broadcas?ngworld,canwrapanything)

–  AAF(videoedi?ngworld,canwrapanything)–  METS(libraryworld,canwrapanything)–  Quick?me(canwrap.mov,DV,MPEG‐2/4,J2K,Flash)

–  AVI(canwrapDV,J2K,DivX,mp3)

–  ASF(proprietaryMicrosoy)wrapperforWMVBesser‐SOIMA2011‐Compression&Sampling 59 Besser‐ICCROMRIO,6/8/0760

DigitalCinema

•  24fps•  2K(2,000X1,500)perframefortheatricaldistribu?on

•  4K(4,000X3,000)perframeformasterfile(36M/frameuncompressed)

7/13/11

11

Howmuchstoragedoyouneed?

•  1hourvideo•  640X480•  24bitscolor•  25fps•  =83Gbytes


Administra?ngDigitalVideoover?me

•  Preserva?onofdigitalvideofilesshouldfollowstandardrepositoryprac?cesforpreservingalltypesofdigitalcontent–  OAIScompliantrepositoryarchitecture

– Metadatawrappers(includingpreserva?onmetadatasuchasPREMISthatdocumentseachchangeorreformabngorreinstalla?onortransforma?on)

–  Periodicmonitoringofcheck‐sums–  RegularRefreshing– Migra?onand/orEmula?onstrategies


ImportantGuide(1/2)


ImportantGuide(2/2)


Re/Sources

•  Besser,Howard.Talks,hHp://besser.tsoa.nyu.edu/howard/Talks/

•  Besser,Howard.Introduc?ontoImaging,hHp://www.geHy.edu/research/publica?ons/electronic_publica?ons/introimages/index.html

•  AICGuidetoDigitalPhotographyandConserva?onDocumenta?on,2ndEdi?on,hHp://www.conserva?on‐us.org/index.cfm?fuseac?on=Store.viewProduct&product_ID=19276&prod_cat_ID=13

•  Fleischhauer,CarlandCarolineArms,SustainabilityofDigitalFormats,PlanningforLibraryofCongressCollec?ons:ContentCategories‐MovingImages,hHp://www.digitalpreserva?on.gov/formats/content/s?ll.shtml

•  IMAP/EAIPreserva?onGuidehHp://www.eai.org/resourceguide/preserva?on.html

•  MIAPWebsitehHp://www.nyu.edu/?sch/preserva?on/


Compression & Sampling - Howard Besser's Site

Documents