Implementing a media archive - lessons learned

FOUNDED

21 Dec. ‘12

Not (only) a Broadcast

Archive

CULTURAL HERITAGE

We don’t own

collections

We provide services towards

50+ org’s

BROADCAST CITY

ARCHIVES

SERVICE PROVIDER

STRONGER TOGETHER

Collectie Huis Van Alijn | © Huis Van Alijn

DIGITISATION ARCHIVING DISSEMINATION

DIGITISATION

HERITAGE + MEDIA

HERITAGE

MEDIA

Type materiaal/formaat

# DRAGERS

TOTAAL

UREN

TOTAAL

# dragers aandeel %

# dragers aandeel %

Film 74.173 26.709 11% 89%

Video analog 150.389 228.773 26% 74%

Video digital 225.578 156.189 5% 95%

Audio analogue 165.883 78.370 36% 64%

Audio digital 34.870 18.112 54% 46%

TOTAL 650.893 508.153 21% 79%

INVENTORY ANALOGUE CARRIERS

FIRST DIGITIZATION WAVE

ALL DIGITIZATION WAVES

UP  TO    450  TB/MONTH  

ALL DIGITIZATION WAVES

UP  TO    450  TB/MONTH  

WOI  

Evergem's Yzerblad: komt uit de loopgrachten als ’t past (S.l. 1917 – 1918) Collectie Erfgoed- bibliotheek Hendrik Conscience | © Vlaamse Erfgoedbibliotheek – Foto: Stefan Tavernier

WW I Newspaper

project

ARCHIVING

Slide 14

Scalable, redundant preservation system

Slide 15

MAM system

DISSEMINATION

QUID PRO QUO

CP’s

Education

Libraries

Research

TARGET AUDIENCE

EDUCATION

Slide 20

FROM ARCHIVE

TO CLASSROOM

ARCHIVE SYSTEM PROCUREMENT / INSTALLATION

BUDGET 2013-2014

11.8 mio EURO

TIME CONSTRAINT

TIME CONSTRAINT Evaluation

after 18 months

START JAN 1, 2013

(staff: 1)

Get and keep all customers

on board Start all

services within 1 year

VIAA NOW TEAM

ARCHIVE 4 people

14 FTE

March 2013 -

Greenfield!

lessons learned

May 2013 -

Greenfield!

Public Procurement

surveys towards users

UX Designed according

to their needs

Creates buy-in!

requirements gathering

workhops involving users

Users co-wrote our RFP

PRIOR NOTICE

Transparent Procurement Proces

DETAILED  ALLOTMENT  REPORT  (debrief)  

EVALUATION JURY

Working with a jury

•  Jury consisted of •  Stakeholders (future users)•  International experts•  VIAA staff

•  Why?•  A balanced answer / evaluation•  Jury members add weight & credibility•  Again: user buy-in!

3 EU tenders - Timeline of procurement•  June 2013 : prior notice•  August 2 : Candidates invited•  September 12 : Candidates selected•  September 13 : RFP published•  October 17 : Quotes received•  November, 6 : Allotment •  November 25 : Final allotment•  April 2014 : MAM in production

Slide 34

Three copies (2 MAM’s)

Archive system

Slide 36

Zeticon MediaHaven

Main MAM services

mul@-­‐tenant  MAM  SYSTEM  

1.  IMPORT  WORKFLOWS  &  INGEST  REPORTING  

3.  EXPORT  WORKFLOW  (OAI-­‐PMH)  ,  EXIT  PATH  

2.  MANAGEMENT      -­‐  Transcoding  (ffmpeg)      -­‐  Par@al  re@eve      -­‐  Metadata      -­‐  Time-­‐based  Annota@on      -­‐  Search  (~SOLR)      -­‐  Manual  QC      -­‐  Storage  Management    

Slide 38

Processing capacity & ramp-up

Processing capacity: => up until 13 TB / day (and still scaling)

WORKFLOWS

Practical MAM use

•  We work along the lines of OAIS for defining•  Ingest process (SIP definition)•  Long term preservation (AIP)•  Dissemination (DIP)

•  Definition in a service agreement•  Practical agreement between VIAA and CP•  Usually for one collection (e.g. audio digitization)

Ingest workflow from digitisation

•  We use PREMIS for provenance•  Every step in the process is recorded

digitization firm, CP, VIAA, …  

registration, carrier inspection, digitisation, encoding, …  

Each having a time and outcome + notes  

Analogue carrier or digital equivalent  

Complete flow => more than 1 MAM

1. Registration of the carrier / AMS

AMS  func@ons:  •  Analogue  carrier  

registra@on  •  PID  crea@on  •  Support  the  

logis@cs  process  •  Technical  

characteris@cs  •  Metadata  from  

digi@za@on  

Persistent identifier creation

Persistent identifier creation

1  Analogue  carrier  =  1  Intellectual  En@ty  =  1  PID  

Persistent identifier

The  PID  is  the  key  for  keeping  track  of  all  events  that  have  an  impact  on  the  digital  object.      We  monitor  the  complete    lifecycle  of  the  digital  object,  from  registra7on  to  dissemina7on  

2. Ingest validation

SIP  contains  •  Essence  (MXF)  •  Metadata  (XML)  •  QC  Report  (XML)  

2a. Transfer validation

2b. SIP validation

2c. Storage

All  metadata  generated  during  the  ingest  process  is  stored  as  PREMIS.  

MAM workflow-Ingest monitoring using PREMIS

Ingest monitoring using PREMIS

Error handling

Error handling (Tableau report)

Step 3: Archived

How  big  is  the  VIAA  archive?  Which  mime  types  do  we  have  in  the  archive?  How  many  items  do  we  archive  /  CP?  

grappige  foto  van  VIAA  team  en  dashboard?  

Items Archived

●  539,6  TB  ●  47.273  items  

Size of the VIAA archive

•  539  TB  •  50.000  items  

TB archived per CP

Step 4: Reuse / interaction

Collectie Huis Van Alijn © Huis Van Alijn

Road Ahead

Collectie Huis Van Alijn © Huis Van Alijn

Road ahead

•  Ingest of born digital material•  Complex and many (50+) data sources•  Need for a ‘SIP creator’•  Realisation through integration with an

enterprise service bus•  Should we be a TDR?

•  Builing along the lines, looking into certification

CONCLUSIONS

Collectie Huis Van Alijn © Huis Van Alijn

CONCLUSION•  MAM system

•  Took a while to understand business needs•  Found a very flexible partner in Zeticon

•  Perfect MAM system? •  Flexibility through well documented API’s•  Pluggable!•  Interface: Usability – HTML5•  Standards: support for and keep up with –

PREMIS, RDF, DC:TERMS, OAI-PMH, …

THANKS!

Collectie Huis Van Alijn © Huis Van Alijn