Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Post on 03-Jan-2016
35 Views
Preview:
DESCRIPTION
Transcript
Building An Use Case Implementation With Building An Use Case Implementation With Denmark’s Broadcasting Archive Of Radio And Television Denmark’s Broadcasting Archive Of Radio And Television (BART)(BART)
Researching Fedora To Serve As Central Repository For Denmark’s State And University Library
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Stephan Drescher, std@statsbiblioteket.dk
picture 1 Analog Antenna Set
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
That’s where it all started …In an attic…
Options for System
• complete outsourcing of taskcomplete outsourcing of task• from shelffrom shelf• taylormade systemtaylormade system• own developmentown development
– around black box around black box professional equipmentprofessional equipment
– high-end consumer + Linux
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Nature of the data harvest process
• 24/7/36524/7/365• single shotsingle shot• large quantities of data (220 GB a day)large quantities of data (220 GB a day)• several independent input sourcesseveral independent input sources• no unique identifier in or between no unique identifier in or between
sourcessources• different MIME types different MIME types • data needs to be evaluated and data needs to be evaluated and
eventually corrected after 48 hourseventually corrected after 48 hours• automatically ingested into repository
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
figure 1 Where will be FedoraFedora within Bart’s resource workflow?
TV/FM Analog SignalTV/FM Digital Signal
Footage
DescriptiveTechnicalLogistical
Metadata
Data Capture
Encoded StreamsImagesTrailers
Footage
DCMETSRDF
FOXML
Metadata
Ingester
Data Collection
Requirements...
Footage
Requirements...
Metadata
Migrate...
Dissemminator Index and Search
Data Preserve
Time MachineA/V Clippings
Footage
RelationCollection
Metadata
Disseminator DRMUser Groups
Distribute
BART
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Hauppauge WinTV
Hauppauge WinTV
Hauppauge WinTV
DR1
DR2
TV2
Hauppauge WinTV
Hauppauge WinTV
Hauppauge WinTV
DR1
DR2
TV2
Hauppauge WinTV
Hauppauge WinTV
Hauppauge WinTV
DR P1
DR P2
DR P3
PC #1
PC #1’
PC #2
EMCmass-
storage
lokaldisk
lokaldisk
lokaldisk
Hauppauge WinTV
Hauppauge WinTV
Hauppauge WinTV
DR P1
DR P2
DR P3
PC #2’
lokaldisk
figure 2 Setup Capture
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Encoder setup
• 3 x MPEG1/2 hardware encoder = 3 x MPEG1/2 hardware encoder = control quality and amount of datacontrol quality and amount of data
• PCM output = lossless audio, but PCM output = lossless audio, but down-sampleddown-sampled
• both FM Radio and TV frequency both FM Radio and TV frequency
tuning on a single card = flexible tuning on a single card = flexible fallback configurationfallback configuration
• parallel setup =parallel setup =
backup from original signal sourcebackup from original signal source
picture 2 Encoder And Sampler
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
digital resources factory… basement
Mass storage specs
• 80 - 100 TB a year (average 220 GB 80 - 100 TB a year (average 220 GB a day)a day)
• Backup on tapeBackup on tape
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
figure 3 Mass Storage
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
The cooled place…
figure 3 Signal coverage
KildeKanalkode Frekvens Format
GB/dag Start End
Landsd¾kkende tv-kanaler:DR 1 antenne 8 196.25 MPEG-2 54.8 08:00 03:00DR 2 kabel 9 203.25 MPEG-2 42.0 14:00 02:00TV2 Danmark, inkl. TV2 Æstjylland antenne 26 511.25 MPEG-2 60.0 06:00 04:00TV2 Zulu kabel 67 839.25 MPEG-1 13.5 06:45 04:30TV2 Charlie kabel 40 623.25 MPEG-1 8.1 12:45 02:00dk4 kabel 63 807.25 MPEG-1 16.2 00:00 00:00Regionale tv-kanaler:TV2 Lorry antenne (Ballerup)53 727.25 MPEG-1 3.4 11:00 12:30
18:10 23:30TV2 Syd antenne 30 543.25 MPEG-1 3.4 11:00 12:30
18:10 23:30Landsd¾kkende radiokanaler:P1 antenne 88.1 PCM 1.8 00:00 00:00P2 (Musik & Kultur) antenne 103.0 PCM 1.8 00:00 00:00P3 antenne 91.7 PCM 1.8 00:00 00:00DR Klassisk kabel 100.8 PCM 1.8 00:00 00:00Radio2 antenne 94.6 PCM 1.8 00:00 00:00Sky Radio antenne 88.6 PCM 1.8 00:00 00:00(reserveret) antenne / kabel PCM 1.8Regionale radiokanaler:DR P4 Kæbenhavns Radio antenne (Ballerup) 96.5 PCM 0.6 06:00 00:00DR P4 Radio Syd (ikke mulig) PCM 0.6 06:00 00:00DR P4 Æstjyllands Radio antenne 95.5 PCM 1.4 06:00 00:00
total (uden uge 46) 216.7
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Create recordings order• Failover definitionsFailover definitions• Channel definitionsChannel definitions• Virtual encoder definitionVirtual encoder definition• Compression definitionsCompression definitions• Source definitionsSource definitions• Physical encoder definitionPhysical encoder definition• Valid Compression per physical encoder typeValid Compression per physical encoder type• Encoder mappingsEncoder mappings
– Virtual encoderVirtual encoder– Physical encoderPhysical encoder
• Abstract schedule definitionAbstract schedule definition• Query definitions (where to get start/stop info)Query definitions (where to get start/stop info)
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Metadata
• Descriptive metadata Descriptive metadata – program informationprogram information– creator, author --> Dublin Corecreator, author --> Dublin Core
• Technical metadataTechnical metadata– encoding formatencoding format– create/modified dates --> METScreate/modified dates --> METS
• Logistical data Logistical data – addressesaddresses– Path to footage e.g. --> METSPath to footage e.g. --> METS
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Collect metadata
• Ritzau Media Service, (MSSQL Ritzau Media Service, (MSSQL database)database)
• DR 1, (XML stream)DR 1, (XML stream)• Logfiles recording, (filesystem)Logfiles recording, (filesystem)• Logfiles storage, migration Logfiles storage, migration
(filesystem)(filesystem)
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Automatic harvest and ingest system
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Scheduler application
• ModulesModules– BouncerBouncer– HarvesterHarvester– LogSnifferLogSniffer– RecorderRecorder– TransformerTransformer– InspectorInspector– IngesterIngester– MigratorMigrator– SyncronizerSyncronizer
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
e.g. Virtual encoder definition in Recorder module
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
figure 4 Main Class Table
Bouncer Access to system and modules
Login, registration
Dispatcher
Harvester Collects metadata
Harvest Definition
Syncronizer,LogSniffer
LogSniffer Knows how to parse log files
List of name-value
Recorder
Recorder Schedules encoder
Recording Definition
Syncronizer
Transformer Arrange dataInto different formats, locations
e.g. From SQLResultSets toXML, METS, HTML
METSWriter,Connector
Inspector Data quality control
Checksum, Scene detection
LogSniffer
Ingester Feed repository
Upload fileXML stream
Transformer,Fedora
Migrator Move repository
Download fileXML Stream
Fedora,Transformer
Syncronizer Tune a task Mirror DatabasesCorrect Data
Transformer
Researching Fedora, Fedora Meeting Copenhagen 09/28/05
Thank you!
top related