Archive eXchange Format AXF An Open Format for Universal Content Transport, Storage and Long Term Preservation Brian Campanotti CTO Front Porch Digital What is AXF? • AXF is a universal format for the encapsulation, storage, transport and preservation of file assets • AXF is like an advanced ZIP which encapsulates any number of files, metadata and a ubiquitous file system • AXF packages can exist on block based storage as well as any file system on any media type • AXF is the first open format targeted at the storage, preservation and transport of file based assets
14
Embed
Archive eXchange Format AXF · Archive eXchange Format • AXF is a file collection “wrapper” which can encapsulate any number of files of any type and size ! • AXF does not
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Archive eXchange Format AXF
An Open Format for Universal Content Transport,!Storage and Long Term Preservation!
!Brian Campanotti!
CTO!Front Porch Digital!
What is AXF? • AXF is a universal format for the encapsulation,
storage, transport and preservation of file assets !
• AXF is like an advanced ZIP which encapsulates any number of files, metadata and a ubiquitous file system!
• AXF packages can exist on block based storage as well as any file system on any media type !
• AXF is the first open format targeted at the storage, preservation and transport of file based assets!
Origins of AXF • Developed based on the demanding needs of the
media and entertainment industry!
o Very large files and file collections!o Complex and important file relationships!o Fundamental connection between assets and metadata!o Randomly accessible at every levels!o Valuable assets which must be preserved!
• Although developed in the M&E space, the benefit of AXF is easily extended to any file based application !
• AXF is currently at the heart of various storage, preservation and cloud transport applications today!
Where Did We Come From? • Proprietary systems are rampant in M&E!o Many data formats on media!o Numerous interface protocols!o Same system type is required for recovery!o Media migration is not easy !
• Ever increasing danger of orphaned archives!
• High costs of implementation and operation!o Individualized system integration requirements!o Transfer costs resulting from inability to interchange
media!
User Demands • Simplified movement of material between different storage
and archive systems!
• Flexibility in changing storage vendors, technology and middleware solutions!
• Limitless scale (number of files and size) including the ability to span across media!
• Universally accessible leveraging stand alone utilities!
• Support for all media types – now and into the future!
• Highly resilient with in-built redundancy!
• Universal despite the storage or transport technology!
AXF Technology
Storage and Preservation • We must take steps to ensure long term accessibility
to valuable file-based assets stored in digital archives in all industries!
• Key goals of an ideal storage format!o Ensure long term accessibility!o Self-describing assets and self-describing media!o Encapsulation to maintain important metadata/file
relationships!o Scalability for any number of elements of any size and type!o Standardized regardless of storage media technology!o Transportability and compatibility between systems!o Preservation (OAIS) features such as fixity, provenance,
etc.!o Streaming support to enable on-the-fly processing!
• What choices are there?!
Tape ARchive Format • Tape ARchive (TAR) format has been around for
many decades!
• There is no true universal TAR implementation but many “customized” ones !
• TAR is a legacy format which cannot support many of the core functionality required in the M&E space !
• TAR does not address many of the storage format goals we have outlined !
• Do we have any other choices?!
Linear Tape File System • The Linear Tape File System (LTFS) is a simple file system
for linear data tape !
• LTFS makes data tapes appear as “removable storage”!
• There are no standards bodies which currently endorse LTFS although some work is underway within SNIA!
• Unfortunately LTFS has some significant limitations with respect to the goals previously outlined !
• LTFS is very useful for the physical “transport” of content but not for long term storage or preservation !
• But why?!
Linear Tape File System • LTFS offers no media encapsulation and relies on simple
folder hierarchies to form important asset relationships!
• LTFS does not scale well due to lack of support for spanning across storage media – a significant problem in M&E!
• LTFS only supports modern data tape technologies and is not applicable to any other storage technologies!
• Neither LTFS nor TAR achieve 100% of our long term storage and preservation goals!
• What other choices are there?!
Archive eXchange Format • AXF is a file collection “wrapper” which can
encapsulate any number of files of any type and size !
• AXF does not overlap with MXF or other “media wrapper” approaches as these are simply files inside AXF objects!
• AXF brings the same level of universal transport and interoperability to storage as MXF has for media !
• It was designed to support all storage technologies – now and into the future !
• It is IT-centric and not tied to media applications alone!
What is an Object Store? • Key to media and entertainment asset storage and
preservation is the concept of an Object Store !
• An Object Store encapsulates all related files (essence, metadata, ancillary files, etc.) into a single package maintaining critical asset relationships !
• CSM solutions are Object Stores by definition (HSM systems are NOT!)!
• This concept becomes even more important when we extend the CSM functionality into the cloud and asset transport!
• Block alignment for increased resiliency!• Provenance records track the entire lifecycle of object!• On-the-fly fixity verification on all files and structures!• Spanning across media for limitless scale (eg. data tape spanning)!• Random access to all metadata and files!• Fully contained AXF indices enable quick re-indexing!• Self-describing objects and media allow quick transport between systems!
• On-the-fly encapsulation and de-encapsulation!• AXF objects are re-aligned to the block/chunk size of target storage devices!• Provenance metadata automatically tracks all actions on the AXF object!• On-the-fly fixity verification on all files and structures!• Self-describing objects and media allow quick transport between systems!• Important file and metadata relationships are always maintained!