The Atlas Petabyte Datastore A grid enabled, networked data storage system: CrystalGrid Workshop 15 th Sept 2004 David Corney. [email protected].

The Atlas Petabyte Datastore

A grid enabled, networked data storage system:

CrystalGrid Workshop 15th Sept 2004

David Corney. [email protected]

Data Store Overview

• General purpose, multi user, data archive. • In use over 20 years. Four major upgrades.• Current capacity 1PB – largest (non dedicated) multi user

system in UK academia?• Grid Interfaces:

– SE (Storage Element) – will be SRM compliant– SRB interface (Storage Resource Broker)

STK 9310

Switch_1 Switch_2

RS6000 RS6000RS6000 RS6000

fsc0 fsc1 fsc1fsc0

9940B 9940B 9940B 9940B 9940B 9940B 9940B 9940B

1 2 3 4 5 6 7 8

11 14 11 1415

fsc1fsc0fsc1fsc0

12 13 12 13 15

rmt1 rmt4rmt3rmt2rmt5-8 rmt5-8rmt5-8rmt5-8

A A A A A A A A

STK 9310 “Powder Horn”

Gbit network

1.2TB 1.2TB 1.2TB 1.2TB

flfstk

tapeserv

Farm Server

flfsys(+libflf)

user

flfscan

data transfer (libvtp)

catalogue data

STK tape drive

cellmgr

Catalogue Server (brian)

flfdoexp(+libflf)

flfdoback(+libflf)

datastore (script)

Robot Server (buxton)

ACSLS

API

control info(mount/dismount)

data

data

Tape Robot

flfsys user commands (sysreq)

SE

recycling (+libflf)

read

read

read

Atlas Datastore Architecture

28 Feb 03 - 2 B Strong

SSI

CSI

flfsys farm commands (sysreq)

LMU

flfsys admin commands(sysreq)

administrators

flfaio

flfaio

flfaio

IBM tape drive

flfqryoff(copy of

flfsyscode)

Backupcatalogue

stats

flfsys tapecommands

(sysreq)

servesys

pathtape

long name(sysreq)

short name(sysreq)

frontend

backendPathtape Server (rusty)

(sysreq)

importexport

flfsys import/export commands (sysreq)

libvtp User Node

I/E Server(dylan)

?

Copy BCopy C

ACSLS

cache disk

Copy A

vtp

vtp

user program

tape

(sysreq)

Strategy -

• De-couple user and application from storage media. • Upgrades and media migration occur “behind the scenes”• High resilience - very few Single Point Failures• High reliability high, availability (99.9986% in 2003)• Constant environmental monitoring linked to alarm/call out• Lifetime data integrity checks hardware and software• Fire safe and off-site backups; Tested disaster recovery

procedures; media migration, recycling• Easy to exploit (endless) new technology• Technology watch to monitor future technology path

Robot History

• M860 – 110GB

• STK 4400– 1.2Tbytes

• IBM 3494– 30Tbytes

• STK 9310– 1Pbyte

Hardware upgrade - completed Jun 2003

• STK 9310 “Powderhorn” with 6000 slots (1.2Pbytes)• 4 IBM 3590 B drives now phased out

– 10 Gbyte native– 10 Mbyte/s transfer

• 8 New STK 9940B drives– 200 Gbyte native– 30Mbytes/sec/drive transfer – 240Mbyte/sec theoretical maximum bandwidth

• 4 RS6000 Data servers (+ 4 “others”)• 1Gbit networking (Expected to become 10Gbit by 2005)• Data Migration to new media completed ~ Feb 2004

Users

• Particle Physics Community (LHC: CMS, Atlas, LHcb,….)• ISIS, British Atmospheric Data Centre• EISCAT (Radar research)• National Earth Observation Data Centre• World Data Centre, BITD• Central Laser Facility• Diamond…• National Crystallography Service, Southampton University, • WASP, VIRGO Consortium• Integrative Biology,• Others…

Interfaces

• “Light weight” interfaces:– Client server configuration:

• “tape” command for many platforms• Virtual Tape Protocol (VTP)• Fortran and C callable library

• “Heavy weight” interfaces:– SRB interface– SE interface developed for EDG/GRIDPP/GRIDPP2…

SRB-ADS architecture

SRB MCATDatabase

SRB MCATServer

SRB ADSServer

SRBClient

SRB DiskServer (Local Server)

Atlas Data Store SRB ADS Server

SRB-ISIS server instance

SRB-BADC server instance

SRB-CCLRC server instance

Port 5600

Port 5601

Port 5602

Adding Interfaces for ADS

ADS CentralCatalog Serverflfsys

ADS FarmServer

ADS FarmServer

ADS FarmServer

SRB01Server

SRB02Server

ADS FarmServer

VTP InterfaceSRB Interface

LCGServers

SE Interface

SRBUsers

SEUsers

VTPUsers(tape)

Logical Resource for Containers

ADS-cacheresource

ADS-taperesource

ADS-logical-resource

Sput –c <container-name> <data-file>

Ssyncont

Copies container from cache to tape

The Storage Element (SE) mass storage interface

The Storage Element (SE)• A component of European Data Grid (EDG/EGEE) middleware

developed by CCLRC’s e-Science & PPD departments

• Uniform Grid Interface which enables a standard protocol for mass data transfer across the grid, between the many diverse Mass Storage Systems, including:

– Atlas Petabyte Data Store

– CASTOR

– ENSTOR

– HPSS

– Others…

SE Deployment

CERN – Castor and disk

UAB Barcelona – Castor

RAL – Atlas DataStore and Disk

ESA/ESRIN – disk

CC-IN2P3 – HPSS

INFN / CNAF – disk

FZK Karlsruhe – disk

Questions?

Digital Curation Centre

• Joint collaboration between CCLRC, UKOLN, and Edinburgh and Glasgow Universities.

• Provide advice, support, research and Development into aspects of Digital Curation for the UK HE community

• Funded jointly by JISC and EPSRC - £1m/year for three years initially. Feb 2004- 2007

• Establish collaboration with industrial partners…

Objectives

• Vibrant research programme

– addressing the wider issues of digital curation

• Collaborative Associates Network of Data Organisations

– strong links across existing community of practice

– engagement with curators (individuals & organisations)

• Services

– to evaluate tools, methods, standards and policies

– a repository of tools and technical information

• ‘Virtuous circle’

– expertise, experience & requirement feed into the DCC research programme

Digital Curation Centre - Organisation

Service Operations Group

management & governance

Industryresearch collaborators

standards bodies

users: communities of practice

U. of EdinburghU. of Glasgow

UKOLN(Bath)

NDCC/NeSC focus & physical presence

curation organisations e.g. DPC

JISC & Research Councils

Management BoardAdvisory

Group

Collaborative Associates Network of DataOrganisations

CCLRC

Steering & Policy Committee

Research Co-ordination Committee

CCLRC’s role within the DCC

• Standards watch

• Standards definition and publication

• Tools watch

• Tools selection and certification

• Registry of metadata standards

• Metadata research

DCC role in Certification

• DCC will help to create

– Standards against which to perform audit and certification• OAIS Reference Model and follow-on work

– Processes for accreditation and certification• Work in Digital Repository Certification Task Force

– Organisation(s) to perform accreditation and certification

ADS SRB Interface for CMS

SRBMCATserver

SRBADS

Server

ADS ServerFarm

Oracle Database Cluster

ADSTape Robot

System

ADSPathtape

server

File System

File System

SRBClient

SRBFile

Server

SRBClient

SRBFile

Server

CSF disk serverAt RAL

CMS nodeExternal to RAL

SRBClient,

External