Bigger Hard Drive Jamie Lean

Film Archive headquarters, 84 Taranaki St, Wellington

Wednesday, 4 April 2012

Film Archive cafe and reception


Film Archive Medialibrary


The New Zealand Film Archive

120,000 titles300,000 items

40 + staff

Collect - acquisition, accessioning, cataloguing, research and commercial access

Protect - storage, preservation, digitisation

Connect - screenings, exhibitions and website


How will we store our films?

How long will they last?

Film Archive storage at Plimmerton



MWA Flashscan HD for 8mm, Super8mm and 9.5mm



First 1 GB hard drive, theIBM 3380 introduced June 1980

weight - 250kg, capacity 2.52GB


Generation total in TB

LTO 3 164

LTO 4 889

LTO 5 189

Server 40

total 1282

Film Archive data storage

LTO is the acronym for Linear Tape-Open, an open standard for digital data tape


UNIVAC I, March 1951 stored data using liquid mercury,the next model used magnetic tape


IBM729V from early 1960’s


Film was Easy (fear of the Black Box)


Last Film Search,Dunedin 1998


Analog tape formats

1" Type B video tape (Robert Bosch GmbH)1" Type C videotape (Ampex, Marconi and Sony)2" Quadruplex videotape (Ampex)2" Helical Scan Videotape (Rank Cintel)Betacam (Sony)Betacam SP (Sony)Betamax (Sony)S-VHS (JVC) (1987)W-VHS (JVC) (1994)U-matic 3/4" (Sony)VCR, VCR-LP, SVRVERA (BBC experimental format ca. 1958)VHS (JVC)VHS-C (JVC)Video 2000 (Philips)

Digital tape formats

Betacam IMX (Sony)D-VHS (JVC)D-TheaterD1 (Sony)D2 (Sony)D3D5 HDDigital-S D9 (JVC)Digital Betacam (Sony)Digital8 (Sony)DVHDVProHD (JVC)MicroMVMiniDV

Videotape formats


http://en.wikipedia.org/wiki/1_inch_type_B_videotape

http://en.wikipedia.org/wiki/1_inch_type_B_videotape

http://en.wikipedia.org/wiki/Robert_Bosch_GmbH

http://en.wikipedia.org/wiki/Robert_Bosch_GmbH

http://en.wikipedia.org/wiki/1%22_Type_C_videotape

http://en.wikipedia.org/wiki/1%22_Type_C_videotape

http://en.wikipedia.org/wiki/Ampex


http://en.wikipedia.org/wiki/Marconi

http://en.wikipedia.org/wiki/Marconi

http://en.wikipedia.org/wiki/Sony


http://en.wikipedia.org/wiki/2%22_Quadruplex_videotape

http://en.wikipedia.org/wiki/2%22_Quadruplex_videotape



http://en.wikipedia.org/w/index.php?title=2%22_Helical_Scan_Videotape&action=edit&redlink=1

http://en.wikipedia.org/w/index.php?title=2%22_Helical_Scan_Videotape&action=edit&redlink=1

http://en.wikipedia.org/w/index.php?title=Rank_Cintel&action=edit&redlink=1

http://en.wikipedia.org/w/index.php?title=Rank_Cintel&action=edit&redlink=1

http://en.wikipedia.org/wiki/Betacam

http://en.wikipedia.org/wiki/Betacam

http://en.wikipedia.org/wiki/Betacam_SP

http://en.wikipedia.org/wiki/Betacam_SP

http://en.wikipedia.org/wiki/Betamax

http://en.wikipedia.org/wiki/Betamax

http://en.wikipedia.org/wiki/S-VHS

http://en.wikipedia.org/wiki/S-VHS

http://en.wikipedia.org/wiki/JVC


http://en.wikipedia.org/wiki/W-VHS

http://en.wikipedia.org/wiki/W-VHS



http://en.wikipedia.org/wiki/U-matic

http://en.wikipedia.org/wiki/U-matic



http://en.wikipedia.org/wiki/Video_Cassette_Recording

http://en.wikipedia.org/wiki/Video_Cassette_Recording

http://en.wikipedia.org/wiki/VERA_videotape_format

http://en.wikipedia.org/wiki/VERA_videotape_format

http://en.wikipedia.org/wiki/BBC

http://en.wikipedia.org/wiki/BBC

http://en.wikipedia.org/wiki/VHS

http://en.wikipedia.org/wiki/VHS



http://en.wikipedia.org/wiki/VHS-C

http://en.wikipedia.org/wiki/VHS-C



http://en.wikipedia.org/wiki/Video_2000

http://en.wikipedia.org/wiki/Video_2000

http://en.wikipedia.org/wiki/Philips

http://en.wikipedia.org/wiki/Philips

http://en.wikipedia.org/wiki/Betacam_IMX

http://en.wikipedia.org/wiki/Betacam_IMX



http://en.wikipedia.org/wiki/D-VHS

http://en.wikipedia.org/wiki/D-VHS



http://en.wikipedia.org/wiki/D-Theater

http://en.wikipedia.org/wiki/D-Theater

http://en.wikipedia.org/wiki/D1_(Sony)

http://en.wikipedia.org/wiki/D1_(Sony)



http://en.wikipedia.org/wiki/D2_(video_format)

http://en.wikipedia.org/wiki/D2_(video_format)



http://en.wikipedia.org/wiki/D3_(video)

http://en.wikipedia.org/wiki/D3_(video)

http://en.wikipedia.org/wiki/D5_HD

http://en.wikipedia.org/wiki/D5_HD

http://en.wikipedia.org/wiki/Digital-S

http://en.wikipedia.org/wiki/Digital-S



http://en.wikipedia.org/wiki/Digital_Betacam

http://en.wikipedia.org/wiki/Digital_Betacam



http://en.wikipedia.org/wiki/Digital8

http://en.wikipedia.org/wiki/Digital8



http://en.wikipedia.org/wiki/DV


http://en.wikipedia.org/wiki/HDV

http://en.wikipedia.org/wiki/HDV

http://en.wikipedia.org/wiki/ProHD

http://en.wikipedia.org/wiki/ProHD



http://en.wikipedia.org/wiki/MicroMV

http://en.wikipedia.org/wiki/MicroMV

http://en.wikipedia.org/wiki/MiniDV

http://en.wikipedia.org/wiki/MiniDV

Lossless compression

AlparyAnimation (qtrle) - QuickTime, FFmpegArithYuvAVIzlib - LCL (VfW codec), FFmpeg

! ! CamStudio GZIP/LZODirac lossless - libdirac, FFmpeg (decoder only)FastCodecFraps - FFmpeg (decoder only)FFV1 - FFmpegH264 lossless - x264 (encoder only), FFmpeg Huffyuv - FFmpegJPEG 2000 lossless

Video codecs (from wikipedia)

Lagarith - Lagarith Codec (VfW codec), FFmpeg

LOCO - FFmpeg (decoder only)LZOMSU Lossless Video CodecPNG - CorePNG, FFmpeg

! ! SheerVideoSnow lossless - FFmpegTechSmith Screen Capture Codec (TSCC)

- FFmpeg Ut Video[5] - FFmpeg (decoder only)YULSZMBV (Zip Motion Block Video) Codec

- FFmpegZRLE used by VNC


http://en.wikipedia.org/wiki/Video_compression#Lossless_compression

http://en.wikipedia.org/wiki/Video_compression#Lossless_compression

http://en.wikipedia.org/wiki/Animation_codec

http://en.wikipedia.org/wiki/Animation_codec

http://en.wikipedia.org/wiki/Dirac_(codec)


http://en.wikipedia.org/wiki/FFV1

http://en.wikipedia.org/wiki/FFV1

http://en.wikipedia.org/wiki/H264

http://en.wikipedia.org/wiki/H264

http://en.wikipedia.org/wiki/X264


http://en.wikipedia.org/wiki/Huffyuv

http://en.wikipedia.org/wiki/Huffyuv

http://en.wikipedia.org/wiki/JPEG_2000


http://en.wikipedia.org/wiki/Lagarith

http://en.wikipedia.org/wiki/Lagarith

http://en.wikipedia.org/wiki/MSU_Lossless_Video_Codec

http://en.wikipedia.org/wiki/MSU_Lossless_Video_Codec

http://en.wikipedia.org/wiki/Portable_Network_Graphics

http://en.wikipedia.org/wiki/Portable_Network_Graphics

http://en.wikipedia.org/wiki/SheerVideo

http://en.wikipedia.org/wiki/SheerVideo

http://en.wikipedia.org/wiki/Snow_(codec)


http://en.wikipedia.org/wiki/YULS

http://en.wikipedia.org/wiki/YULS

http://wiki.multimedia.cx/index.php?title=DosBox_Capture_Codec

http://wiki.multimedia.cx/index.php?title=DosBox_Capture_Codec

http://en.wikipedia.org/wiki/VNC

http://en.wikipedia.org/wiki/VNC

Lossy compression

Audio Video Standard (AVS) OpenAVS - FFmpeg Blackbird FORscene video codecCineformCinepak - FFmpeg (decoder only)Dirac - Schrödinger, dirac-research, FFmpeg DV - FFmpegFirebird [1] Original FORscene video codecH.261 - FFmpeg H.261 (libavcodec)MPEG-1 Part 2 (MPEG-1 Video)

Cinema Craft EncoderElecard MPEG-1 Decoder/EncoderFFmpegLigos LSX MPEG-1MainConcept MPEG-1TMPGEnc

! ! H.262/MPEG-2 Part 2 (MPEG-2 Video)Canopus ProCoderCinema Craft EncoderElecard MPEG-2 Video Decoder/Encoder

FFmpeg InterVideo Video DecoderLigos LSX MPEG-2MainConcept MPEG-2TMPGEnc

! ! H.263 - FFmpeg H.263 (libavcodec)(decoder only)

MPEG-4 Part 2 (MPEG-4 Advanced Simple Profile)

3ivxDivXElecard MPEG-4 Decoder/EncoderlibavcodecHDX4Nero DigitalXvid

More video codecs


http://en.wikipedia.org/wiki/Audio_Video_Standard

http://en.wikipedia.org/wiki/Audio_Video_Standard

http://en.wikipedia.org/w/index.php?title=OpenAVS&action=edit&redlink=1

http://en.wikipedia.org/w/index.php?title=OpenAVS&action=edit&redlink=1

http://en.wikipedia.org/wiki/FORscene#Blackbird

http://en.wikipedia.org/wiki/FORscene#Blackbird

http://www.cineform.com/

http://www.cineform.com/

http://en.wikipedia.org/wiki/Cinepak

http://en.wikipedia.org/wiki/Cinepak





http://en.wikipedia.org/wiki/FORscene#Codecs

http://en.wikipedia.org/wiki/FORscene#Codecs

http://www.forbidden.co.uk/demos/live/

http://www.forbidden.co.uk/demos/live/

http://en.wikipedia.org/wiki/H.261


http://en.wikipedia.org/wiki/FFmpeg


http://en.wikipedia.org/wiki/Libavcodec


http://en.wikipedia.org/wiki/MPEG-1

http://en.wikipedia.org/wiki/MPEG-1

http://en.wikipedia.org/wiki/Cinema_Craft_Encoder


http://en.wikipedia.org/wiki/Elecard_Group




http://en.wikipedia.org/wiki/MainConcept


http://en.wikipedia.org/wiki/TMPGEnc


http://en.wikipedia.org/wiki/H.262/MPEG-2_Part_2

http://en.wikipedia.org/wiki/H.262/MPEG-2_Part_2

http://en.wikipedia.org/wiki/Canopus_Corporation

http://en.wikipedia.org/wiki/Canopus_Corporation

http://en.wikipedia.org/wiki/ProCoder

http://en.wikipedia.org/wiki/ProCoder







http://en.wikipedia.org/wiki/InterVideo

http://en.wikipedia.org/wiki/InterVideo









http://en.wikipedia.org/wiki/MPEG-4_Part_2

http://en.wikipedia.org/wiki/MPEG-4_Part_2

http://en.wikipedia.org/wiki/3ivx

http://en.wikipedia.org/wiki/3ivx

http://en.wikipedia.org/wiki/DivX

http://en.wikipedia.org/wiki/DivX





http://en.wikipedia.org/wiki/HDX4

http://en.wikipedia.org/wiki/HDX4

http://en.wikipedia.org/wiki/Nero_Digital


http://en.wikipedia.org/wiki/Xvid

http://en.wikipedia.org/wiki/Xvid

Lossy compression (continued)Indeo 3/4/5 - FFmpeg (decoder only)MJPEG

FFmpegMorgan Multimedia M-JPEGPegasus PICVideo M-JPEGPICVideo

JPEG 2000 intra frame video codec H.264/MPEG-4 AVC or MPEG-4 Part 10,

CoreAVCElecard AVC Decoder/EncoderMainConceptNero DigitalQuickTime H.264Sorenson AVC Pro codec, Vanguard Software Solutionsx264 (encoder only)FFmpeg (decoder only

OMS VideoOn2 Technologies TrueMotion VP3/VP4, VP5, VP6, VP7, VP8; ! under the name The Duck Corporation: TrueMotion S, ! TrueMotion 2, FFmpeg (decoder only)

PixletApple ProRes 422 - FFmpegRealVideo - FFmpegSnow Wavelet CodecSorenson Video, Sorenson Spark - FFmpegTarkinTheora - FFmpeg (decoding only), libtheoraVC-1 (SMPTE standard, subset of Windows Media Video)

FFmpeg (decoder only)VC-2 SMPTE standard (a.k.a. Dirac Pro)

Schrödinger, dirac-researchVC-3 SMPTE standard - Avid DNxHD, FFmpegWindows Media Video (WMV) WAX


http://en.wikipedia.org/wiki/Indeo

http://en.wikipedia.org/wiki/Indeo

http://en.wikipedia.org/wiki/MJPEG

http://en.wikipedia.org/wiki/MJPEG





http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC

http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC

http://en.wikipedia.org/wiki/CoreAVC

http://en.wikipedia.org/wiki/CoreAVC







http://en.wikipedia.org/wiki/QuickTime

http://en.wikipedia.org/wiki/QuickTime

http://en.wikipedia.org/w/index.php?title=Sorenson_AVC_Pro_codec&action=edit&redlink=1

http://en.wikipedia.org/w/index.php?title=Sorenson_AVC_Pro_codec&action=edit&redlink=1

http://en.wikipedia.org/w/index.php?title=Vanguard_Software_Solutions&action=edit&redlink=1

http://en.wikipedia.org/w/index.php?title=Vanguard_Software_Solutions&action=edit&redlink=1



http://en.wikipedia.org/wiki/OMS_Video

http://en.wikipedia.org/wiki/OMS_Video

http://en.wikipedia.org/wiki/On2_Technologies

http://en.wikipedia.org/wiki/On2_Technologies

http://en.wikipedia.org/wiki/VP3










http://en.wikipedia.org/wiki/TrueMotion_S

http://en.wikipedia.org/wiki/TrueMotion_S

http://en.wikipedia.org/wiki/Pixlet

http://en.wikipedia.org/wiki/Pixlet

http://en.wikipedia.org/wiki/ProRes_422

http://en.wikipedia.org/wiki/ProRes_422

http://en.wikipedia.org/wiki/RealVideo

http://en.wikipedia.org/wiki/RealVideo



http://en.wikipedia.org/wiki/Sorenson_Video

http://en.wikipedia.org/wiki/Sorenson_Video

http://en.wikipedia.org/wiki/Sorenson_Spark

http://en.wikipedia.org/wiki/Sorenson_Spark

http://en.wikipedia.org/wiki/Tarkin_(codec)

http://en.wikipedia.org/wiki/Tarkin_(codec)

http://en.wikipedia.org/wiki/Theora

http://en.wikipedia.org/wiki/Theora

http://en.wikipedia.org/wiki/VC-1


http://en.wikipedia.org/wiki/Windows_Media_Video








http://en.wikipedia.org/wiki/DNxHD

http://en.wikipedia.org/wiki/DNxHD






JPEG2000

Created in 2000 by the Joint Photographic Experts Group

- Flexible codestream, scalable compression- provides for both Lossless and Lossy compression- complex encoders/decoders- ISO standard- Digital Cinema standard- initially slow to gain acceptance- licensed but coding system available free


[New Zealand infantry] - 1914 recently scanned at Weta Digital


Upstream, John Ford, 1928


IBM 1401 Data Processing System from 1959. From the left: 1402 Card Read-Punch, 1401 Processing Unit, 1403 Printer.


http://en.wikipedia.org/wiki/IBM_1402




earthquake fire tornado tsunami arson vandalism theft litigation

ignorance war depression indifference

Risks


Obsolescence - hardware & software have a finite lifetime

Data Integrity - latent errors, ingest errors, transfer errors

Single Point Loss - single copy vulnerable to machine error, human error or natural disaster

Mono Culture - singular technologies can suffer

Compression - mathematically lossless, perceptually lossless, lossy compression

Technical Risks


Jedi Archives, Coruscant. Star Wars. Episode II


Keanu Reeves as Johnny Mnemonic

1995





Dreamtime (or how Goodbye Pork Pie becomes a legend)





The Automated Archive no more boring tasks in your paperless office

Digital deposit, accessioning and cataloguing through metadata and crowd sourcing

But getting everything digitised is hard on humans

The Distributed Archive

safety in numbersCompact storage and many copies means less risk of loss

More reliant on complex technology and material can still burn and drown

The Everything for Everyone Archive

quantity vs qualityWe can keep everything and have it instantly available

We may lose control of the content and will that devalue it?


[email protected]

www.filmarchive.org.nz

The Digital Archive promises a bright future

We just have to keep up with it


mailto:[email protected]




Bigger Hard Drive Jamie Lean

Technology

vfw codec

ffmpeg

dirac

vhs

schrdinger

4 april 2012

accessioning

ampex