Karlsruhe, April 27th, 2009 DESY IT 1 Computing Services for large Experiments in Physics Volker Gülzow April 27th, 2009
Karlsruhe, April 27th, 2009 DESY IT 1
Computing Services for large Experiments in Physics
Volker Gülzow
April 27th, 2009
Karlsruhe, April 27th, 2009 Gülzow DESY IT 2
Outline
Computing for the LHC Experiments Open the Grid to other fields Change of landscape Conclusion
Karlsruhe, April 27th, 2009 Gülzow DESY IT 3
Computing for the physics
Simulation Experiments
Karlsruhe, April 27th, 2009 Gülzow DESY IT 4
Large Experiments do need Computing TDR‘s
Karlsruhe, April 27th, 2009 Gülzow DESY IT 5
The Eventflow
Rate
[Hz]
RAW
[MB]
ESDrDST
RECO[MB]
AOD
[kB]
MonteCarlo
[MB/evt]
MonteCarlo
% of real
ALICE HI 100 12.5 2.5 250 300 100ALICE pp 100 1 0.04 4 0.4 100ATLAS 200 1.6 0.5 100 2 20CMS 150 1.5 0.25 50 2 100LHCb 2000 0.025 0.025 0.5 20
107 seconds/year pp from 2009 on ~109 events/experiment106 seconds/year heavy ion
Karlsruhe, April 27th, 2009 DESY IT 6
GridKA
Each layer:Specialised for certain tasks
e.g. T2:
-Analysis
-User access
-AOD storage
Karlsruhe, April 27th, 2009 Gülzow DESY IT 7
GRID
MIDDLEWARE
Visualizing
Supercomputer, PC-Cluster
Data Storage, Sensors, Experiments
Internet, Networks
Desktop
Mobile Access
The Grid Dream
Karlsruhe, April 27th, 2009 Gülzow DESY IT 8
Grid Computing
Grid Computing is about virtualization of global resources.
• It is about transparent access to globally distributed resources such as data and compute cycles
• A Grid infrastructure consists of services to access resources and (of course) of the resources itself
Opposite to distributed computing, Grid resources are not centrally controlled
Hence it is mandatory to use standard, open, general-purpose protocols and interfaces
A Grid must deliver nontrivial qualities of services
• In general Grid infrastructures are generic; without any dependencies of the applications / experiments
Karlsruhe, April 27th, 2009 Gülzow DESY IT 9
Grid Types
• Data Grids:• Provisioning of transparent access to data which can be
physically distributed within Virtual Organizations (VO)• Computational Grids:
• allow for large-scale compute resource sharing within Virtual Organizations (VO)
• Information Grids:• Provisioning of information and data exchange, using well
defined standards and web services• Collaborative Grids:
• allow networking people from various locations
There is not „THE Grid“
Karlsruhe, April 27th, 2009 Gülzow DESY IT 10
Grid Infrastructure and T2
Site3Site2
outputUI
LFC
WN WN WNWN
SEGR
ISCE GR
IS
BDII
JDL RBWMS
Batch
LFC
VOMSRBCESE
/etc/grid-security/grid-mapfile
DESY
ssh
$HOME/.globus/
certs
GIIS
HMS
world
R-GMA
VO exclusive
Karlsruhe, April 27th, 2009 Gülzow DESY IT 11
The Storage Element basic characteristics
LAN data access method/protocol for POSIX style file access
Has a information provider system, which can be queried externally
Has a wide area data access capability – “FTP”and “Web”-style
Has a Storage Management Interface (SRM), for accessibility and communication with other SE and/or data management systems
Karlsruhe, April 27th, 2009 Gülzow DESY IT 12
OSMHPSS
SRM SRM
Virtual Storage LayerSRM
SRM ClientTransfer Protocol Negotiation
Data Transfer
Storage Element
EnstoreTsmCastor
SRM
Jasmine
SRMSRM
Desy GridKarlsruhe
Fermi San Diego CERN Jefferson
dCache.ORG
gsiFTP(ssl) HTTP
Grid Storage Fabric Abstraction
dCap
GRID Middleware
Karlsruhe, April 27th, 2009 Gülzow DESY IT 13
dCache - a few ‘what it is’ lines
distributed cache system to optimize data access on tertiary storage
distributed Peta Byte Disk Store with single rooted file-system namespace, providing posix like and wide area access protocols
Grid Storage Element coming with standard data access protocols, Information Provider Protocols and Storage Resource Manager
Karlsruhe, April 27th, 2009 Gülzow
dCac
he.O
RGdC
ache
.ORG
dCache topology
dCache is a collaboration between DESY, FERMILab and the Nordic Data Grid Facility (NDGF) (Leader: Patrick Fuhrmann, DESY).
Beside providing core function of dCache, DESY is managing the project infrastructure
Code repository, Web, Wiki, code download pages Ticket system and e-mail support lists Regression test suite Customer relations, workshops, tutorials
dCache is distributed by gLite and US Open Science Grid Virtual Data Tool (VDT)
dCache is supported by d-grid, the HGF “Physics at the Terascale” and the US Open Science Grid (OSG).
dCache will hold the largest share of LHC data outside CERN within the next years.
dCache is in production at 7 of 11 LHC Tier 1 centres and more than 40 Tier 2's.
dCache.ORG
gLite
Karlsruhe, April 27th, 2009 Gülzow DESY IT 15
Karlsruhe, April 27th, 2009 Gülzow DESY IT 16
Issues being a Grid Site
• The local installation is operated in a global environment• There is always day light somewhere on the globe• Core Grid services are used everywhere (VOMS, LFC)
• One common infrastructure for multiple VOs of multiple disciplines• Different groups want different things• Computing models differ fundamentally• Use and user patterns differ• Software requirements differ
• User support is a big issue• Not scalable• Underestimated• Has a huge social factor
Karlsruhe, April 27th, 2009 Gülzow DESY IT 17
Karlsruhe, April 27th, 2009 Gülzow DESY IT 18
Reliability of the DESY Tier 2
march-09 Dec-08 jan-09 feb-09
Karlsruhe, April 27th, 2009 Gülzow DESY IT 19
Pushing the Grid: The D-Grid Initiative
National Grid Initiative BMBF-funded with > 100 Mio € Up to now 3 Calls 2 strategic lines: Communities & Integration
(DGI) All major Grid Labs are in DGI Many many communities Work in progress: sustainable infrastructure
Karlsruhe, April 27th, 2009 Gülzow DESY IT 20
Pushing Grid Technology
Generic vs Application driven strategy: Generic: Find/develope generic tools,
blueprints and generic strategies for many communities
Application driven: help communities from the neighbourhood to get started with the grid (-> my preferred approach)
Why? „You need to speak the language of the partner and have to have a feeling for the problem“
Karlsruhe, April 27th, 2009 Gülzow DESY IT 21
Research with Photons
Light Source DORIS @ DESY
Karlsruhe, April 27th, 2009 Gülzow DESY IT 22
Get the Grid to the „Photons“
Differences to HEP Short experiments Many experiments Data Volumes/Year: FDE ~ 3 PB, MID ~ 1.8 PB (1
MPIX CCD) Data stored in small files Data compression: open Many beamlines, different experiments Lifetime of data unclear Opportunistic users The „mental“ problem
Karlsruhe, April 27th, 2009 Gülzow DESY IT 23
Methods are different
Karlsruhe, April 27th, 2009 Gülzow DESY IT 24
Grids for „Photons“
Certificates for user access Computing for analysis under debate (~1000
„MPI-“cores / beamline) Storage: Access via Grid Tools Short term analysis Long term archiving
-> Concept: To start joint door opener projects between research groups and IT, it needs missionery effort
Karlsruhe, April 27th, 2009 Gülzow DESY IT 25
Everything on the grid?
Grid and the Tier model well suited for• Global & coordinated tasks
Analysis• Local & uncoordinated, chaotic data access
Strong need for additional resources for interactive use but with Grid-technology
Access of interactive resources to the SE
Karlsruhe, April 27th, 2009 Gülzow DESY IT 26
The landscape has changed
Virtualization Clouds Energy Networks and international collaborations
Karlsruhe, April 27th, 2009 Gülzow DESY IT 27
What is virtualization?
technique for hiding physical characteristics of computing resources from the way in which other systems, applications, or end users interact with those resources. • making a single physical resource appear to function as multiple logical
resources– server, operating system, application, storage device
• making multiple physical resources appear as a single logical resource.– storage devices or servers
In detail different technologies: bare metal full virtualization, paravirtualization etc.
It’s a concept to make use out of multicore systems Vendor provide HW-support
Karlsruhe, April 27th, 2009 Gülzow DESY IT 28
Can we profit from virtualization?
Yes, definitely, we can
Pro: We will have multiple core systems, which we can efficiently
use We can run legacy applications We can share resources between large experiments Gives portability of the codeCon: Security issues, what about OS patches? Lasy-factor „we don‘t want to port our application“ Support and maintainance costs
Karlsruhe, April 27th, 2009 Gülzow DESY IT 29
Simple Idea:
Setup huge facility and profit from:• Economies of scale (hardware, management,
operation)• Efficient resource scheduling for high utilisation• Tax-efficient locations• Cheap energy, cheap labor• Use virtualization techniques
Clouds
Karlsruhe, April 27th, 2009 Gülzow DESY IT 30
Google had 36 data centers in 2008**
WLCG had 134
BUT
One Google Data Center is estimated to cost ~$600M
An order of magnitude more than the new centre being planned at CERN
Google’s data center at the Dalles on the Columbia river
**source: royal.pingdom.com
Different Approaches
Karlsruhe, April 27th, 2009 Gülzow DESY IT 31
Microsoft’s data center in Quincy, WA • 44K m2 - 10 rooms each with up to 30K servers and storage for 6 trillion
photos Yahoo, Amazon, IBM -- also building giant data centers These major companies are expecting to build new markets for utility
computing (clouds), software as a service (Google Apps), as well as absorbing the expansion of traditional computing services...
Clouds
Karlsruhe, April 27th, 2009 Gülzow DESY IT 32
Clouds vs Grids
Clouds aim at efficient sharing of the hardware• low-level execution environment, Isolation between users• Operated as a homogeneous, single-management domain• Straight-forward I/O and storage• Expose only a high-level view of the environment -
scheduling, data placement, performance issues are hidden from the application and the user
Grids aim at collaboration• Add your resources to the community, but retain management
control• Expose topology – location of storage, availability of
resources• Choice of tools to hide the complexity from the user,
Karlsruhe, April 27th, 2009 Gülzow DESY IT 33
Clouds vs. Grids
Both need complex middleware to function• Grids had a problem in trying to provide a universal high-
functionality environment (OS, data management, ....), with intersecting collaborations and a naturally competitive environment
• Clouds have an advantage in offering a simpler base environment, leaving much of the functionality to the application - where universal solutions are not necessary – and what they do have to provide can be decided within a single management hierarchy
• Both have problems with licenceing As the names suggest –
the grids are transparent and the clouds are opaque
Karlsruhe, April 27th, 2009 Gülzow DESY IT 34
“Clouds” are just starting, will they survive these day’s? What to do, if not? Are there standards? Arn‘t we running a cloud ourselves already? We have to look on costs and potentially have to be ready to
integrate clouds May be use clouds as „overload resource“ And also learning from the innovative technology that is appearing
stimulated by the cloud market• to manage, operate and supply these giant facilities with
data• to package applications to use (heterogeneous) clouds
Clouds for scientific environment
Karlsruhe, April 27th, 2009 Gülzow DESY IT 35
Energy Energy cost of a computing infrastructure are not negletable Look for way‘s to save energy What helps? New processor technology (eg Nehalem) New peripherals concepts Efficient cooling concepts Efficient service provisioning concepts (eg virtualization) Power management and monitoring …
Karlsruhe, April 27th, 2009 Gülzow DESY IT 36
DOE Energy Datacenter Efficiency Initiative
www.energystar.gov
Karlsruhe, April 27th, 2009 Gülzow DESY IT 37
Networks & Mobility
We do have excellent connectivity in parts of the world But we have „digital divide“ We have large user communities accessing our resources at
any time Travelling scientists: We are close to having good bandwidth data connections
almost everywhere we go And we already have a powerful high capacity computer in the backpack This is where end-user analysis is going to be done The physicist’s notebook must be integrated with the experiment
environment, the physics data, and the grid resources Without burdening the notebook or its user The grid environment is too complex to be extended to the notebook
Karlsruhe, April 27th, 2009 Gülzow DESY IT 38
Role of a Scientific Computer Centre
Provide excellent operational service Have flexible and open concepts Know How source Own but adjusted scientific agenda Active partner in scientific collaborations Driving force for IT-innovation Solution (and SW) development Member in international projects
Karlsruhe, April 27th, 2009 Gülzow DESY IT 39
Conclusion
Computing is a major component in large physics experiments
Computing TDR‘s are definitely needed very early Grid‘s are a possible good solution Clouds will not replace the Grid but we can learn
and merge Virtualization is a evident concept In general there are excellent opportunities for vital
scientific Computer Centres
Karlsruhe, April 27th, 2009 Gülzow DESY IT 40
Backup slides
Karlsruhe, April 27th, 2009 Gülzow DESY IT 41
Risks
Requirements of the experiments are not fully settled
Member of a multisite chain, you depend on others There is often money around for invest, not for
personell Operation of huge installations is often
underestimated Financing ist not always long term secured Huge software effort is needed for following the
technology
Karlsruhe, April 27th, 2009 Gülzow DESY IT 42
Summary Grids are all about sharing.
• groups distributed around the world can pool their computing resources• large centres and small centres can all contribute• users everywhere can get equal access to data and
Grids are also flexible • place the computing facilities in the most effective and efficient places• exploiting funding wherever it is provided
HEP and others have shown that• grids can support computational and storage resources on a massive scale• that can be operated around the clock• running hundreds of thousands of jobs every day
The grid model has stimulated high energy physics to organise its computing• in a widely distributed way• building a collaboration involving directly a large fraction of the LHC members and their
institutes
This will be the workhorse for production data handling for many yearsand as such must be maintained and developed through the first waves of data taking
Karlsruhe, April 27th, 2009 Gülzow DESY IT 43
But – the landscape has changed dramatically over the past decade
The Web, the Internet, powerful PCs, broadband to the home, …• have stimulated the development of new applications that generate a
massive demand for computing remote from the user• …. that is being met by giant, efficient facilities deployed around the world• .... and creates a market for new technologies capable of operating on a
scale equivalent to that of HEP Whether or not commercial clouds become cost-effective for HEP data handling
is only a financial and funding-agency issueBUT Exploiting the associated technologies is an obligation
Could there be a revolution here for physics analysis?
Karlsruhe, April 27th, 2009 Gülzow DESY IT 44
The MoU for WLCG
See: http://lcg.web.cern.ch/LCG/mou.htm Stating rules, Tier 1&2 centres&responsibilities&
pledges, 11 Tier 1‘s, ~ 120 Tier 2‘s
Karlsruhe, April 27th, 2009 Gülzow DESY IT 45
The German Science Net XWIN
Karlsruhe, April 27th, 2009 Gülzow DESY IT 46
Different tasks: Different requirements
MC Production• Event Generation: no I; small O; little CPU• Detector Simulation: small I; large O & CPU
Event Reconstruction/Reprocessing• Reprocessing: full I; full O; large CPU• Selections: large I; large O; large CPU
Analysis• Usually: large I; small O; little CPU• Performed by many users, many times!• LHC StartUp phase: Short turn-around
Coordinated &global tasks
Uncoordinated, chaotic & local tasks
Karlsruhe, April 27th, 2009 Gülzow DESY IT 47
SE + surroundings
File
Transfer
ServiceFTS DM App
SRM SRM
SE SEFTS Channel
FTP connections
CE
LANAccess
Information System
Karlsruhe, April 27th, 2009 Gülzow DESY IT 48
The Grid Installation @ DESY, a complex system DESY’s obligation to WLCG (Tier 2 for Atlas, CMS, LHCb) The NAF for the Helmholtz Alliance (grid, interactive, batch),
seamless, site independent Support of the HERA analysis Hosting & support of various Virtual Organisations like ILC,
Icecube, ILDG etc. (total 15 VO’s) with a complete Grid infrastructure (example: running 7 Workload Mgmt Sys)
DESY’s obligation to D-Grid DESY’s obligation to EGEE Federation with RWTH Aachen (CMS) Federation with Uni Göttingen (Atlas) Support and cooperation with the Experiments
Karlsruhe, April 27th, 2009 Gülzow DESY IT 49
Storage & Computing Resources
• Zeuthen: – ~500 Cores -> 700 kSI2k for Tier2 & NAF Grid & ILDG &
Icecube– 460 TB disk for dCache, Lustre
• Hamburg:– ~2000 Cores -> 3000+ kSI2k for Tier 2 & NAF Grid & Hera &
ILC & others– 900 TB disk for dCache, Lustre – 450 TB for NAF dCache
• Total: ~ 3700+ kSI2k for the Grid, • NAF: interactive/batch (Zeuthen&Hamburg)
– 868 Cores ->1200 kSI2k for NAF interactive/batch
Karlsruhe, April 27th, 2009 Gülzow DESY IT 50
Karlsruhe, April 27th, 2009 Gülzow DESY IT 51
links
http://lcg.web.cern.ch/LCG/ http://naf.desy.de/ http://www.sun.com/software/products/lustre/ http://gridmap.cern.ch
Karlsruhe, April 27th, 2009 Gülzow DESY IT 52
EGEE …
Objectives:
The EGEE project brings together experts from more than 50
countries with the common aim of building on recent advances in
Grid technology and developing a service Grid infrastructure which is
available to scientists 24 hours-a-day.
The project provides researchers in academia and business with
access to a production level Grid infrastructure, independent of their
geographic location. The EGEE project also focuses on attracting a
wide range of new users to the Grid.
Because of its needs and its tradition and because of its *simple* use
cases, HEP has become the pilot application for the Grid (in EGEE).
Karlsruhe, April 27th, 2009 Gülzow DESY IT 53
NAF: Schematic basic layout
Interactive
Proof?
Batch-Type(SGE)
gsissh
grid-submit
qsub
??
AFS
Parallel Cluster FS
(Lustre)
SRM (dCache)DedicatedSpace in DESY T2 space
Desy Tier 2 (Batch)grid-submit
AFS/Kerberos
?? SRM ??
Grid-ftp /
srmcp
scp
NAFInterconnect
Karlsruhe, April 27th, 2009 Gülzow DESY IT 54
sketch of dCache components
tertiarystorage
Pool Nodes
namespaceservice
NFS v2/3
dCache Core
FTP
local access:- dCap- xrootd- http
SRMworld
put/get
local client
loca
l clie
nt
Karlsruhe, April 27th, 2009 Gülzow DESY IT 55
Virtualization on the Worker Nodes
Surprising idea: Virtualization costs performance, but many benefits:• More OS types and flavors can be supported, also old OS on new
hardware possible• Each jobs runs in his own OS instance, does not affect other jobs:
security through encapsulation• Separation of local and grid environment/users• Desktop harvesting?• Each job might get a clean system at start: No trojans• Buy a general purpose cluster, and use it for many different
purposes• Job migration and checkpointing: Interesting for MPI and very long
jobs• Distributed administration: Local admin installs VMM, generic
Virtual Machine provided by user or third party
Karlsruhe, April 27th, 2009 Gülzow DESY IT 56
Energy
www.koomey.com