Grid-related High Performance Middleware and Laboratories Dr. Carl Kesselman Director Center for Grid Technologies
Grid-related High Performance Middleware and Laboratories
Dr. Carl KesselmanDirectorCenter for Grid Technologies
EO Grid Middleware
How do we solve problems? Communities committed to common goals
- Virtual organizations
Teams with heterogeneous members & capabilities
Distributed geographically and politically- No location/organization possesses all required skills
and resources
Adapt as a function of the situation- Adjust membership, reallocate responsibilities,
renegotiate resources
EO Grid Middleware
The Grid Vision
“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”- On-demand, ubiquitous access to computing, data,
and services
- New capabilities constructed dynamically and transparently from distributed services
“When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances”
(George Gilder)
EO Grid Middleware
A Little History(U.S. Perspective)
Early 90s- Gigabit testbeds, metacomputing
Mid to late 90s- Early experiments (e.g., I-WAY), software projects (e.g.,
Globus), application experiments 2001
- Major application communities emerging- Major infrastructure deployments are underway- Rich technology base has been constructed- Global Grid Forum: >1000 people on mailing lists, 192
orgs at last meeting, 28 countries
EO Grid Middleware
Selected Major Grid Projects
Name URL & Sponsors FocusAccess Grid www.mcs.anl.gov/FL/
accessgrid; DOE, NSFCreate & deploy group collaboration systems using commodity technologies
BlueGrid IBM Grid testbed linking IBM laboratories
DISCOM www.cs.sandia.gov/discomDOE Defense Programs
Create operational Grid providing access to resources at three U.S. DOE weapons laboratories
DOE Science Grid
sciencegrid.org
DOE Office of Science
Create operational Grid providing access to resources & applications at U.S. DOE science laboratories & partner universities
Earth System Grid (ESG)
earthsystemgrid.orgDOE Office of Science
Delivery and analysis of large climate model datasets for the climate research community
European Union (EU) DataGrid
eu-datagrid.org
European Union
Create & apply an operational grid for applications in high energy physics, environmental science, bioinformatics
g
g
g
g
g
g
EO Grid Middleware
Selected Major Grid Projects
Name URL/Sponsor FocusEuroGrid, Grid Interoperability (GRIP)
eurogrid.org
European Union
Create technologies for remote access to supercomputer resources & simulation codes; in GRIP, integrate with Globus
Fusion Collaboratory fusiongrid.org
DOE Off. Science
Create a national computational collaboratory for fusion research
Globus Project globus.org
DARPA, DOE, NSF, NASA, Msoft
Research on Grid technologies; development and support of Globus Toolkit; application and deployment
GridLab gridlab.org
European Union
Grid technologies and applications
GridPP gridpp.ac.uk
U.K. eScience
Create & apply an operational grid within the U.K. for particle physics research
Grid Research Integration Dev. & Support Center
grids-center.org
NSF
Integration, deployment, support of the NSF Middleware Infrastructure for research & education
g
g
g
g
g
g
EO Grid Middleware
Selected Major Grid Projects
Name URL/Sponsor FocusGrid Application Dev. Software
hipersoft.rice.edu/grads; NSF
Research into program development technologies for Grid applications
Grid Physics Network
griphyn.org
NSF
Technology R&D for data analysis in physics expts: ATLAS, CMS, LIGO, SDSS
Information Power Grid
ipg.nasa.gov
NASA
Create and apply a production Grid for aerosciences and other NASA missions
International Virtual Data Grid Laboratory
ivdgl.org
NSF
Create international Data Grid to enable large-scale experimentation on Grid technologies & applications
Network for Earthquake Eng. Simulation Grid
neesgrid.org
NSF
Create and apply a production Grid for earthquake engineering
Particle Physics Data Grid
ppdg.net
DOE Science
Create and apply production Grids for data analysis in high energy and nuclear physics experiments
g
g
g
g
g
g
EO Grid Middleware
Selected Major Grid Projects
Name URL/Sponsor FocusTeraGrid teragrid.org
NSF
U.S. science infrastructure linking four major resource sites at 40 Gb/s
UK Grid Support Center
grid-support.ac.uk
U.K. eScience
Support center for Grid projects within the U.K.
Unicore BMBFT Technologies for remote access to supercomputers
SCEC www.scec.org
Nsf
Integrated geophysics modeling
g
gNew
Also many technology R&D projects: e.g., Condor, NetSolve, Ninf, NWS
See also www.gridforum.org
EO Grid Middleware
The Grid World: Current Status
Dozens of major Grid projects in scientific & technical computing/research & education
Considerable consensus on key concepts and technologies- Open source Globus Toolkit™ a de facto standard for
major protocols & services
- Far from complete or perfect, but out there, evolving rapidly, and large tool/user base
Industrial interest emerging rapidly Opportunity: convergence of eScience and
eBusiness requirements & technologies
EO Grid Middleware
Layered Grid Architecture
Application
Fabric“Controlling things locally”: Access to, & control of, resources
Connectivity“Talking to things”: communication (Internet protocols) & security
Resource“Sharing single resources”: negotiating access, controlling use
Collective“Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services
InternetTransport
Application
Link
Inte
rnet P
roto
col
Arch
itectu
re
Globus Toolkit
Globus Toolkit is the source of many of the protocols described in “Grid architecture”
Adopted by almost all major Grid projects worldwide as a source of infrastructure
Open source, open architecture framework encourages community development
Active R&D program continues to move technology forward
Developers at ANL, USC/ISI, NCSA, LBNL, and other institutions
www.globus.org
Globus ToolkitComponents Include … Core protocols and services
- Grid Security Infrastructure
- Grid Resource Access & Management
- MDS information & monitoring
- GridFTP data access & transfer
Other services- Community Authorization Service
- DUROC co-allocation service
Other Data Grid technologies- Replica catalog, replica management service
EO Grid Middleware
User
Userprocess #1
Proxy
Authenticate & create proxy
credential
GSI(Grid
Security Infrastruc-
ture)
Gatekeeper(factory)
Reliable remote
invocation
GRAM(Grid Resource Allocation & Management)
Reporter(registry +discovery)
Userprocess #2Proxy #2
Create process Register
The Globus Toolkit in One Slide Grid protocols (GSI, GRAM, …) enable resource sharing within
virtual orgs; toolkit provides reference implementation ( = Globus Toolkit services)
Protocols (and APIs) enable other tools and services for membership, discovery, data mgmt, workflow, …
Other service(e.g. GridFTP)
Other GSI-authenticated remote service
requests
GIIS: GridInformationIndex Server (discovery)
MDS-2(Meta Directory Service)
Soft stateregistration;
enquiry
EO Grid Middleware
Globus Toolkit Structure
GRAM MDS
GSI
GridFTP MDS
GSI
???
GSI
Reliable invocationSoft state
management
Notification
ComputeResource
DataResource
Other Serviceor Application
Jobmanager
Jobmanager
Lots of good mechanisms, but (with the exception of GSI) not that easilyincorporated into other systems
Service naming
EO Grid Middleware
NSF Middleware Initiative
NSF Funded Project to build national middleware infrastructure- USC/ISI, SDSC, U. Wisc., ANL, NCSA, I2
Software Integration (NMI Software Releases)- Interoperability
- Testing
- Install, Configure, Manage University Campus Infrastructure Integration
- Campus Authentication / GSI
- Enterprise Directories / GSI and MDS Use NMI as Teragrid Baseline
- Specialize for Teragrid unique aspects (e.g. Viz resources)
EO Grid Middleware
NMI-R1 Software Components
Globus Toolkit Condor-G Network Weather Service KX.509 / KCA Certificate Profile Maker Pubcookie Grid Packaging Tools
EO Grid Middleware
U.S. GRIDS Center
GRIDS = Grid Research, Integration, Deployment, & Support
NSF-funded center to provide- State-of-the-art middleware infrastructure to support
national-scale collaborative science and engineering
- Integration platform for experimental middleware technologies
ISI, NCSA, SDSC, UC, UW + commercial partners
www.grids-center.org
EO Grid Middleware
Network for Earthquake Eng. Simulation
NEESgrid: national infrastructure to couple earthquake engineers with experimental facilities, databases, computers, & each other
On-demand access to experiments, data streams, computing, archives, collaboration
www.neesgrid.org: Argonne, Michigan, NCSA, UIUC, USC
EO Grid Middleware
SCEC Modeling Environment
Knowledge Base
OntologiesCurated taxonomies,
Relations & constraints
Pathway ModelsPathway templates,
Models of simulation codes
Code Repositories
Data & SimulationProductsData Collections
FSM
RDM
AWM
SRM
Storage
GRIDPathway Execution
Policy, Data ingest, Repository access
Grid ServicesCompute & storage management, Security
DIGITALLIBRARIES
Navigation &Queries
Versioning,Topic maps
MediatedCollectionsFederated
access
KNOWLEDGEACQUISITION
Acquisition InterfacesDialog planning,
Pathway constructionstrategies
Pathway AssemblyTemplate instantiation,
Resource selection,Constraint checking
KNOWLEDGE REPRESENTATION & REASONINGKnowledge Server
Knowledge base access, InferenceTranslation Services
Syntactic & semantic translation
Pathway Instantiations
Computing
Users
EO Grid Middleware
Data Intensive Physical Sciences
High energy & nuclear physics- Including new experiments at CERN
Gravity wave searches- LIGO, GEO, VIRGO
Time-dependent 3-D systems (simulation, data)- Earth Observation, climate modeling
- Geophysics, earthquake modeling
- Fluids, aerodynamic design
- Pollutant dispersal scenarios
Astronomy: Digital sky surveys
EO Grid Middleware
National Virtual Observatory
Xray (ROSAT) theme
Change scale
Change theme
http://virtualsky.org/fromCaltech CACRCaltech AstronomyMicrosoft Research
Optical (DPOSS)
Coma cluster
Virtual Sky has140,000,000 tiles
140 Gbyte
EO Grid Middleware
Grid Physics Network (GriPhyN)
Enabling R&D for advanced data grid systems, focusing in particular on Virtual Data concept
Virtual Data ToolsRequest Planning and
Scheduling ToolsRequest Execution Management Tools
Transforms
Distributed resources(code, storage,computers, and network)
Resource Management
Services
Resource Management
Services
Security and Policy
Services
Security and Policy
Services
Other Grid Services
Other Grid Services
Interactive User Tools
Production Team
Individual Investigator Other Users
Raw data source
ATLASCMSLIGOSDSS
www.griphyn.org; see also www.ppdg.net, www.eu-datagrid.org
EO Grid Middleware
Data Grids for High Energy Physics
Tier2 Centre ~1 TIPS
Online System
Offline Processor Farm
~20 TIPS
CERN Computer Centre
FermiLab ~4 TIPSFrance Regional Centre
Italy Regional Centre
Germany Regional Centre
InstituteInstituteInstituteInstitute ~0.25TIPS
Physicist workstations
~100 MBytes/sec
~100 MBytes/sec
~622 Mbits/sec
~1 MBytes/sec
There is a “bunch crossing” every 25 nsecs.
There are 100 “triggers” per second
Each triggered event is ~1 MByte in size
Physicists work on analysis “channels”.
Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server
Physics data cache
~PBytes/sec
~622 Mbits/sec or Air Freight (deprecated)
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
Caltech ~1 TIPS
~622 Mbits/sec
Tier 0Tier 0
Tier 1Tier 1
Tier 2Tier 2
Tier 4Tier 4
1 TIPS is approximately 25,000
SpecInt95 equivalents
Image courtesy Harvey Newman, Caltech
EO Grid Middleware
LaserInterferometricGravitational waveObservatory
Listening to Collisions ofBlack Holes andNeutron Stars
EO Grid Middleware
LIGO Hardware
EO Grid Middleware
Grid LIGO Architecture
Clientseg Web, Script, Agent
Clientseg Web, Script, Agent
Text requestText request Request Manager
Request Manager
GriPhyN LDAS
Gatekeeper(GRAM)
Gatekeeper(GRAM)
Science AlgorithmsSoftware Collaboratory
Parallel ComputingGridFTPGridFTP
Local Disk
Data
HPSS
GridFTPGridFTP
Replica CatalogReplica ManagementTransformation CatalogVirtual Data Catalog
Virtual Data Request
Data Movement
Globus RPC
other LDASCondor jobs
EO Grid Middleware
iVDGL: A Global Grid Laboratory
International Virtual-Data Grid Laboratory- A global Grid laboratory (US, Europe, Asia, South America, …)- A place to conduct Data Grid tests “at scale”- A mechanism to create common Grid infrastructure- A laboratory for other disciplines to perform Data Grid tests- A focus of outreach efforts to small institutions
U.S. part funded by NSF (2001-2006)- $13.7M (NSF) + $2M (matching)
“We propose to create, operate and evaluate, over asustained period of time, an international researchlaboratory for data-intensive science.”
From NSF proposal, 2001
EO Grid Middleware
iVDGL Components Computing resources
- 2 Tier1 laboratory sites (funded elsewhere)- 7 Tier2 university sites software integration- 3 Tier3 university sites outreach effort
Networks- USA (TeraGrid, Internet2, ESNET), Europe (Géant, …)- Transatlantic (DataTAG), Transpacific, AMPATH?, …
Grid Operations Center (GOC)- Joint work with TeraGrid on GOC development
Computer Science support teams- Support, test, upgrade GriPhyN Virtual Data Toolkit
Education and Outreach Coordination, management
EO Grid Middleware
iVDGL Components (cont.)
High level of coordination with DataTAG- Transatlantic research network (2.5 Gb/s) connecting
EU & US
Current partners- TeraGrid, EU DataGrid, EU projects, Japan, Australia
Experiments/labs requesting participation- ALICE, CMS-HI, D0, BaBar, BTEV, PDC (Sweden)
EO Grid Middleware
Initial US-iVDGL Data Grid
Tier1 (FNAL)Proto-Tier2Tier3 university
UCSDFlorida
Wisconsin
FermilabBNL
Indiana
BU
Other sites to be added in
2002
SKC
Brownsville
Hampton
PSU
JHUCaltech
EO Grid Middleware
iVDGL Map (2002-2003)
Tier0/1 facility
Tier2 facility
10 Gbps link
2.5 Gbps link
622 Mbps link
Other link
Tier3 facility
DataTAG
Surfnet
LaterBrazilChile?PakistanRussiaChina
EO Grid Middleware
The TeraGrid:26
24
8
4 HPSS
5
HPSS
HPSS UniTree
External Networks
External Networks
External Networks
External Networks
Site Resources Site Resources
Site ResourcesSite ResourcesNCSA/PACI8 TF240 TB
SDSC4.1 TF225 TB
Caltech Argonne
EO Grid Middleware
Summary Grid infrastructure is becoming widespread
- Major deployment based on common technology
- Significant new deployment activities
Consensus building mechanisms in place- Global Grid Forum (www.gridforum.org)
Industrial buy in starting- IBM, Entropia, more to come
EO Grid Middleware
For More Information
Book (Morgan Kaufman)- www.mkp.com/grids
Globus- www.globus.org
- “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”
GRIDS Center- www.grids-center.org
Grid Forum- www.gridforum.org