‘Archiving and managing a million or more data files on BiG Grid’ Peter Doorn, Data Archiving and Networked Services (DANS) With Jan Just Keijser (NIKHEF) BiG Grid & Beyond, Amsterdam, 26/9/2012
Nov 17, 2014
‘Archiving and managing a million or more data files on BiG Grid’
Peter Doorn, Data Archiving and Networked Services (DANS)With Jan Just Keijser (NIKHEF)BiG Grid & Beyond, Amsterdam, 26/9/2012
Contents
Promises and ideas at the kick-off of BiG Grid in 2007: what became of them?
In NL SSH in UK, DE, ESFRI
Two sub-projects of BiG Grid with DANS Analyzing and visualizing big humanities data (briefly)
Archiving and managing a million or so humanities files
Beyond BiG Grid: next requirements and challenges for the future of SSH research and infrastructure
An example of analysis of Big Social Science Data (GPS traces) from Italy
Challenges for data infrastructure
From the original Big Grid proposal:
“BIG GRID is crucial to the success and continuity of many Dutch research communities, covering important areas such as life sciences, astronomy, particle physics, meteorology, and climate research, water management, to name just a few.
However, the very nature of the new infrastructure, a multidimensional collaboration enabler and accelerator, allows for direct participation of also social sciences, humanities, and even addresses communities in administrative domains, like digital academic repositories.” Pro
mise
s… P
rom
ises…
ESFRI projects in SSH about grid
CESSDA: grid technologies for facilitating the merging of distributed data sources
DARIAH: grid services for an open semantic architecture
facilitating arts and humanities research need for ‘easy’ interfaces for humanities scholars,
services need to be usable without the complexities of the grid infrastructure
CLARIN: grid technology for access to guidance and advice through distributed knowledge centres access to repositories of data with standardized descriptions, processing
tools ready to operate on standardized data
Prom
ises…
Pro
mise
s…
Tools for processing, analysing, annotating, editing and publishing text data
• Grid-enabled workbench to process, analyse, annotate, edit and publish XML-encoded textual data for academic research
• Connect to the D-Grid Integration Platform (DGI) via TextGrid-specific middleware components
• Demonstrate the efficiency of the grid-enabled tools in the areas publishing, processing, retrieval, and linking
• Semantic TextGrid: semantic methods for processing text assets, and for interweaving texts and dictionaries
Prom
ises…
Pro
mise
s…
Germany: Textgrid
But a
lso re
sults
…
TextGrid VRE: Repository + Lab
But a
lso re
sults
…
UK: e-Social Science
“The National Centre for e-Social Science (NCeSS) investigates how innovative and powerful computer-based infrastructure and tools, developed under the UK e-Science programme, can benefit the social science research community”
Examples of grid-projects: Mixed Media Grid (MiMeG): generate tools and techniques for social scientists to analyse audio-visual qualitative data and related materials collaboratively
SABRE software has been specifically designed for the statistical analysis of multi-process random effect response data, using parallel processing Pro
mise
s… P
rom
ises…
UK e-Social Science discontinued…
Is no
mor
e…
Dutch example from humanities
Subject: organization of knowledge Comparison of designed classification system
(UDC) with a socially grown knowledge system (Wikipedia)
Multidisciplinary research group, including DANS researcher Andrea Scharnhorst
Big data set (dump of Wikipedia: 2,8 TB) Mine the data to extract the page and category link changes over time
Create complex visualizations Computational support by BiG Grid team: Tom
Visser, Coen Schrijvers and Ammar Benabadelkader
Archiving experiments since 2007
Grid middleware not very suitable for our archiving purposes
Use case: How can you be sure that what you store on the grid is valid?
Giving proof of data integrity is a requirement of ISO standard 16363 for trusted digital archives
Advantages of grid storage: Fast access to grid worker node Hierarchical storage manager: eg. efficient automated backup procedures
Shared facility is efficient and economically attractive
Large numbers of datasets and files
> 23,000 data sets in DANS archives Every data set consists of 1+ data files, sometimes 1000+ Most data sets are small (98% < 1 Gb) For example, the entire population census of 1960 (>11
million records) fits on one CD-ROM (< 700 Mb) Total number of files >1 million Total storage volume ca. 70 Tb Long processing times with large numbers of datasets and
files Management operations on the whole archive: slow and
problematic on normal servers Mass conversions (e.g. thumbnails of images) Data integrity control (checksums) Compressing the data
Copying of the whole archive to the grid is not trivial
Datasets in DANS EASY (Sept. 2012)
1,8% of datasets > 2 GB2,8% of datasets > 1 GB
23,560 datasets 1,693,413 files
The experiment
Experiment with five digital archives (not in EASY), containing a total 290,341 files, grouped over a total of 1695 'tar' files of 5 GB each (c. 8.5 TB)
Carried out by Jan Just Keijser (Nikhef) Three-phase workflow
DANS Workflow phase 1:• Create checksums• Create tarballs (.tar files)• Upload tarballs to the grid
1) md5sum
2) tar
3) Upload gridstorage
DANS Workflow phase 2:• Download .tar file• Compress it to a .tar.gz file• Upload compressed tarball
3) Upload
gridstorage
Worker Node
1) Download
2) Compress
4) Compare
gridstorage
Worker Node
1) Download
2) Unpack
DANS Workflow phase 3:• Download .tar.gz file• Unpack it• Calculate checksums• Send checksums back and compare
3) md5sum
Results
The tool works One checksum mismatch detected: disk failure on grid worker node!
SSH: big data challenges
Data generated by people tend to be small Data generated by social processes (Twitter, Facebook), transactions (financial), administrations and by devices (GSM, GPS) tend to be big
More analytical projects of big data in SSH (but few in NL)
Millions of digitized books (“Culturomics”)
Sentiment analysis of twitter feeds to predict markets and economic trends
Traffic flows using GPS
An example from Italy
GPS traces17K private carsone week of ordinary mobility200K trips (trajectories)Milan, Italy
From presentation by Dino PedreschiPisa
Data donated by OCTO Telematics
Where is traffic concentrated between midnight and 2 a.m.? (red = most intense)
Where is traffic concentrated between 6 p.m. and 8 p.m.?
Select only trips that start in the city centre (orange) and move to North-West
Where is people between 6pm and 8pm of Wednesday, April 4th?
Where is people between 8pm and 10pm of Wednesday, April 4th? (high density spot appeared)
Where is people between 10pm and midnight of Wednesday, April 4th? (The dense spot disappeared. What happened?)
Focus on the high-density spot: Centered on the parking lots of the stadium, a football match took place there...
SSH Research beyond Big Grid
Acceptance of grid technology by SSH community is low and slow: “my laptop has enough processing power”
Grid is still perceived as “complicated” Researchers are not aware of:
data management issues the research potential of “Big SSH Data”
Demonstrator projects are still needed: Social scientists need to focus more on the analytical potential of “Big Social Data”
“Culturomics” in humanities DANS can help to make that accessible, although we
are not only driven by data, but also by… demand!
Archiving beyond BiG Grid
Storage capacity: joining forces with other parties: 3TU Data Centre, National Coalition for Digital Preservation (NCDD with Royal Library, National Archives, Institute for Sound and Vision, museum sector), Roadmap projects
Archiving is more than storage: archival management requires repeated operations on masses of files, many small, but also big (e.g. audio/visual)
Set of procedures to support archival management Continuity of grid infrastructure is prerequisite Is cloud the answer?
Public cloud is not without risk Costs are not yet attractive enough Private community cloud is attractive