Tony Doyle - University of Glasgow UK UK title.open ( ); revolution title.open ( ); revolution {execute}; {execute}; LHC Computing Challenge LHC Computing Challenge Methodology? Methodology? H H ierarchical ierarchical I I nformation in a nformation in a G G lobal lobal G G rid rid S S upernet upernet Aspiration? Aspiration? HIGGS HIGGS DataGRID-UK DataGRID-UK Aspiration? Aspiration? ALL ALL Data Intensive Computation Data Intensive Computation Teamwork
31
Embed
UK Tony Doyle - University of Glasgow title.open ( ); revolution {execute}; LHC Computing Challenge Methodology? Hierarchical Information in a Global Grid.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
HHierarchical ierarchical IInformation in a nformation in a GGlobal lobal GGrid rid SSupernetupernet
Aspiration?Aspiration?
HIGGSHIGGS
DataGRID-UKDataGRID-UK
Aspiration?Aspiration?
ALLALL Data Intensive Computation Data Intensive Computation Teamwork
Tony Doyle - University of Glasgow
UKUK
OutlineOutline
Starting PointStarting Point The LHC The LHC
Computing Computing ChallengeChallenge
Data HierarchyData Hierarchy DataGRIDDataGRID Analysis Analysis
ArchitecturesArchitectures
GRID Data GRID Data ManagementManagement
Industrial Industrial PartnershipPartnership
Regional CentresRegional Centres Today’s WorldToday’s World Tomorrow’s WorldTomorrow’s World SummarySummary
Tony Doyle - University of Glasgow
UKUK
Starting PointStarting Point
Tony Doyle - University of Glasgow
UKUK
Starting PointStarting Point
“Current technology would not be able to scale data to such an extent, which is where the teams at Glasgow and Edinburgh Universities come in.The funding awarded will enable the scientists to prototype a Scottish Computing Centre which could develop the computing technology and infrastructure needed to cope with the high levels of data produced in Geneva, allowing the data to be processed, transported, stored and mined. Once scaled down, the data will be distributed for analysis by thousands of scientists around the world. The project will involve participation from Glasgow University's Physics & Astronomy and Computing Science departments, Edinburgh University's Physics & Astronomy department and the Edinburgh Parallel Computing Centre, and is funded by the Scottish Higher Education Funding Council's (SHEFC Joint Research Equipment Initiative). It is hoped that the computing technology developed during the project will have wider applications in the future, with possible uses in astronomy, computing science and genomics observation, as well as providing generic technology and software for the next generation Internet.”
Tony Doyle - University of Glasgow
UKUK
The LHC Computing ChallengeThe LHC Computing Challenge
Detector for ALICE experiment
Detector forLHCb experiment
Tony Doyle - University of Glasgow
UKUK
A Physics EventA Physics Event
Gated electronics response from a proton-proton collisionGated electronics response from a proton-proton collision Raw data: hit addresses, digitally converted charges and timesRaw data: hit addresses, digitally converted charges and times Marked by a unique code:Marked by a unique code:
Proton bunch crossing number, RF bucket Event number
Collected, Processed, Analyzed, Archived….Collected, Processed, Analyzed, Archived…. Variety of data objects become associated Event “migrates” through analysis chain:
may be reprocessed; selected for various analyses; replicated to various locations.
Example analysis scenario:Example analysis scenario: Physicist issues a query from Athena for a Monte Carlo dataset
Issues: How expressive is this query? What is the nature of the query: declarative Creating new queries and language
Algorithms are already available in local shared libraries
An Athena service consults an ATLAS Virtual Data Catalog
Consider possibilities:Consider possibilities: TAG file exists on local machine (e.g. Glasgow)
Analyze it
ESD file exists in a remote store (e.g. Edinburgh) Access relevant event files, then analyze that
RAW File no longer exists (e.g. RAL) Regenerate, re-reconstruct, re-analyze !!! GRID Data
Management
Tony Doyle - University of Glasgow
UKUK
GlobusGlobus
Tony Doyle - University of Glasgow
UKUK
GlobusGlobus
DataGRIDToolKit
Tony Doyle - University of Glasgow
UKUK
GRID Data ManagementGRID Data Management
Goal: develop middle-ware infrastructure to manage petabyte-scale data
Replica Manager
Data Mover
Data Accessor
Storage Manager
Castor HPSS
Data Locator
Meta Data Manager
Local Filesystem
Query Optimisation &Access Pattern Manag.
Secure Region
High Level Services
Medium Level Services
Core ServicesService levels reasonably well defined
Identify Key AreasWithin Software
Structure
Tony Doyle - University of Glasgow
UKUK
5 areas for development5 areas for development Data Accessor - hides specific storage system requirements.
Mass Storage Management group. Replication - improves access by wide-area caching. Globus
toolkit offers sockets and a communication library, Nexus. Meta Data Management - data catalogues, monitoring
information (e.g. access pattern), grid configuration information, policies. MySQL over Lightweight Directory Access Protocol (LDAP) being investigated.
Security - ensuring consistent levels of security for data and meta data.
Query optimisation - “cost” minimisation based on response time and throughput Monitoring Services group.
Identifiable UKContributions
RAL
Identifying Key AreasIdentifying Key Areas
RAL
Tony Doyle - University of Glasgow
UKUK
AstroGridAstroGrid
WP1 PROJECT MANAGEMENT
WP2 REQUIREMENTS ANALYSIS : existing functionality and future requirements; community consultation
WP3 SYSTEM ARCHITECTURES: benchmark and implement
WP4 GRID-ENABLE CURRENT PACKAGES : implement and test performance
WP5 DATABASE SYSTEMS : requirements analysis and implementation; scalable federation tools.
WP6 DATA MINING ALGORITHMS : requirements analysis, development and implementation
WP7 BROWSER APPLICATIONS : requirements analysis and software development
WP8 VISUALISATION : concepts and requirements analysis, software development.
WP9 INFORMATION DISCOVERY : concepts and requirements analysis, software development
WP10 FEDERATION OF KEY CURRENT DATASETS : e.g.. SuperCOSMOS, INT-WFS, 2MASS, FIRST, 2dF
WP11 FEDERATION OF NEXT GENERATION OPTICAL-IR DATASETS : esp. Sloan, WFCAM
WP12 FEDERATION of HIGH ENERGY ASTROPHYSICS DATASETS : esp. Chandra, XMM
WP13 FEDERATION of SPACE PLASMA and SOLAR DATASETS : esp. SOHO, Cluster, IMAGE
WP14 COLLABORATIVE DEVELOPMENT OF VISTA, VST, and TERAPIX PIPELINES
WP15 COLLABORATION PROGRAMME WITH INTERNATIONAL PARTNERS
WP16 COLLABORATION PROGRAMME WITH OTHER DISCIPLINES
Testbed = Learning by ExampleTestbed = Learning by Example
+Cloning
SRIF Expansion
= expansion of open source ideas
“GRID Culture”
Tony Doyle - University of Glasgow
UKUK
missionmission to accelerate the exploitation of simulation by to accelerate the exploitation of simulation by industry, commerce and academia industry, commerce and academia
45 staff, £2.5M turnover - externally funded45 staff, £2.5M turnover - externally funded solve business problems - not sell technologysolve business problems - not sell technology
PartnershipImportant
Tony Doyle - University of Glasgow
UKUK
Industrial PartnershipIndustrial Partnership
pingping
service
ping
monitor
WAN
LAN
Adoption of OPENIndustry Standards
+OO Methods
Industry ResearchCouncil Inspiration:
Data-IntensiveComputation
Tony Doyle - University of Glasgow
UKUK
Regional CentresRegional Centres
SRIF Infrastructure
Grid Data Management
SecurityMonitoring
Networking
Local Perspective:Consolidate
Research Computing
Optimisation of Number of Nodes?4-5?
Relative size dependent on funding dynamics
Global Perspective:V. Basic Grid Skeleton
Regional Expertise Model?
Tony Doyle - University of Glasgow
UKUK
Today’s WorldToday’s World
Istituto Trentino Di Cultura
Helsinki Institute of Physics
Science Research Council
SARA
Tony Doyle - University of Glasgow
UKUK
Tomorrow’s WorldTomorrow’s World
CR2
AC12
AC13
AC14
Istituto Trentino Di
CulturaHelsinki Institute of
PhysicsScience Research Council
AC7
AC8
AC9
AC10
AC11
CR3
AC15
AC16
AC17
CR4
SARA
AC18
AC19
CR5
AC20
AC21
CR6
CO
Tony Doyle - University of Glasgow
UKUK
SummarySummary
General Engagement (£=OK)General Engagement (£=OK) Mutual Interest (Mutual Interest (ScotGRIDScotGRID
Example)Example) Emphasis on Emphasis on
DataGrid Core Development (e.g. Grid Data Management)
“CERN” lead + Unique UK Identity Extension of Open Source Idea “Grid
Culture” = Academia + Industry Multidisciplinary Approach =
University + Regional Basis Use of Existing Structures (e.g. EPCC,