Everything you always wanted to know about the Grid and
never dared to ask
Tony Hey and Geoffrey Fox
Outline• Lecture 1: Origins of the Grid – The Past to
the Present (TH)• Lecture 2: Web Services, Globus, OGSA and
the Architecture of the Grid (GF)• Lecture 3: Data Grids, Computing Grids and
P2P Grids (GF)• Lecture 4: Grid Functionalities – Metadata,
Workflow and Portals (GF)• Lecture 5: The Future of the Grid - e-Science
to e-Business (TH)
Lecture 1
Origins of the Grid –
The Past to the Present
[Grid Computing Book:
Chs 1,2,3,4,5,6,36]
Lecture 5
The Future of the Grid –
e-Science to e-Business
[Grid Computing Book:
Chs 38, 39, 40,41,42,43]
Lecture 5
1. e-Science Research and the Future of Scientific Research
2. Computer Science Research Issues
3. A Business Case for the Grid
4. Concluding Remarks
e-Science and the Future of Scientific Research
‘e-Science will change the dynamic of the way science is undertaken.’
John Taylor, 2001
Integrated e-Science Environment
Framework for distributed scientific computing and experimentation
local remote
Computers
local remote
Data storage
local remote
Experiments
Grid services middleware
Computing Grid
service
Data discovery
Grid service
Data visualisation Grid service
“Problem Solving Environments” Domain-specific application interfaces for scientists
Authentication
AuthorisationAccounting
Experiment control Grid
service
e-Science Examples
• Particle Physics
• Virtual Observatories
• e-Engineering
• e-Chemistry
• Bioinformatics
• High-Throughput Applications
• e-Health
In flight data
Airline
Maintenance Centre
Ground Station
Global Networkeg: SITA
Internet, e-mail, pager
DS&S Engine Health Center
Data centre
DAME Project
Comb-e-Chem
Structures DB
Properties DB
Simulation andcalculation
Structure + Properties Knowledge + Prediction
Combinatorial Chemistry
• Parallel synthetic approach– create hundreds of materials
– screen properties to find those that fit the bill
• Typically requires several passes– find chemical structure of the best
candidates
– create new batches of similar materials for subsequent passes
• Leads to explosive growth in:– volume of data generated
– potential to exploit this data
MeOH EtOH PrOH BuOH
R1COOH
R2COOH
R3COOH
R4COOH
same reaction sequencefor all combinations
Mon
itor
& A
naly
sis
Dat
a
Inte
rfac
e to
Gri
d
AArray production of different chemical speciesrray production of different chemical species
Mass Spec
x-ray
Raman
Well plate with typically 96 or 384 cells
Str
uctu
re a
nd p
rope
rtie
s an
alys
is
databases
High throughputsystems
Librarysynthesis
100,000’s compounds at a time analysis100,000’s compounds at a time analysisProduces huge amounts of complex dataProduces huge amounts of complex data
Users Users Users
Experiment Expert
Data & control links
Access Grid links
ExperimentRemote (Dark) Laboratory
Remote equipment, multiple users, few experts
Model for National crystallographic Service NCSModel for National crystallographic Service NCS
NCS Workflow
X-Ray e-LaboratoryStructuresDatabase
ComputationService
Send sample Send sample material to material to
NCS serviceNCS service
Search materials database Search materials database and predict properties using and predict properties using
Grid computationsGrid computations
Download full Download full data on materials data on materials
of interestof interest
Collaborate in e-Lab Collaborate in e-Lab experiment and experiment and obtain structureobtain structure
NCS Portal Access
NCS Experimental Services
NCS Lab Service
Samples andSchedules
StatusMonitor
Collaboratory Interface Data Access Interface
Proxy Proxy Proxy
ControlGUI
(FilteredVNC)
Chat AudioRaw
ImagesResultsAccess
StructAccess
Schedules
SampleManage-
ment
ScheduleManage-
ment
Raw Data(Files)
ProcessedData (DB)
StructureData (DB)
Admin
Auth
Scheduling Expt Control HKL Calculation Struct Calc
UI
Middleware
Backend
Control
myGrid Project
• Imminent ‘deluge’ of data
• Highly heterogeneous• Highly complex and
inter-related• Convergence of data
and literature archives
myGrid: Generic Technologies
1. Database access from the Grid
2. Process enactment on the Grid
3. Personalisation services
4. Metadata services
5. Development of Agent Services
Ultimate goal is to put Grid Services together
with Ontologies to develop ‘Semantic Grid’
Workflow
• Know how.• Associate base resources with derived data.• Keep, describe, find, compare, protect,
share.• Repeat/reuse/re-enact• Specialise/Customise/Personalise• Evolution – notification, knowledge• Quality & best practice
– Need the workflows to be effective good experimental practice.
1
2
3
4
Personalisation
• Dynamic creation of personal data sets
• Personal views over repositories• Personalisation of workflows • Personal notification • Annotation of datasets and
workflows• Personalisation of service
descriptions – ‘what I think the service does’
1
2
3
4
Provenance
• Who, what, where, why, when, how?• The traceability of knowledge as it is
evolves and as it is derived.• Identity – the Life Sciences ID• Lab Books, Methods in papers.• Immutable Metadata• Migration – travels with its data but
may not be stored with it.• Private vs Shared provenance records.• Ownership/credit
1
2
3
4
Using Distributed Resources
ScientificInformationScientific
InformationScientific Discovery
In Real Time
LiteratureLiterature
DatabasesDatabases
OperationalData
OperationalData
ImagesImages
InstrumentData
InstrumentData
Discovery Net Project
Real Time Integration
Dynamic ApplicationIntegration
Workflow Construction
Interactive Visual Analysis
Discovery Process Management• Workflow = Service Composition +
Discovery Pathway
• Towards a Standard Workflow Representation for Discovery Informatics: Discovery Process Markup Language (DPML):
– Discovery Pathway Construction: Recording and managing a collaboratively-built discovery process
– Distributed Service Composition: Components organsied by the workflow can be executing anywhere
– Discovery Pathway as Key Intellectual Property: Discovery Processes can be stored, reused, audited, refined and deployed in various forms D-Net Workflow for Genome Annotation :
16 services executing across Internet
Dynamic Integration Services• Dynamic Application Integration =
On-demand access and composition of remote analysis components
• Towards a Dynamic Component Integration:
– Knowledge Servers: allow users to register, locate and remotely execute components
– Execution Servers: allow users to control the execution of components distributed environments
– Easy Maintenance: New components can be added through a clean API
Text analysisText analysis
ClusteringClustering ClassificationClassification
Gene function perdition
Gene function perdition
Homology Search
Homology Search
Promoter PredictionPromoter
Prediction
D-NET APID-NET API
Case Study: SC2002 HPC Challenge
blastgenscan
RepeatMasker
grail
genscanE-PCR
Identify
Genes
Gene markers
tRNAs, rRNAs
Non-translatedRNAs
RegulatoryRegions
RepetitiveElements
SegmentalDuplication
SNPVariations
LiteratureReferences
…..
3D-PSSMblast
MotifSearch
PFAM
DSCpredator
InterPro
InterPro
SMARTSWISSPROT
Identify
FunctionalCharacteisation
Homologues
Domain 3-D Structure
Fold PredictionSecondary structure
LiteratureReferences
…..
ProteinsClassify into
Protein Families
Identify
OrganismChromosomes
Organism’sDNA
Relate
CellCycle
Metabolism
DrugsBiologicalProcess…..
Cell deathEmbryogenesis
LiteratureReferences
…..
Ontologies
PathwayMaps
GeneMapsAmiGO
GenNav
virtual chip
High ThroughputSequencers
Nucleotide-level Annotation
Protein-level Annotation
Process-level Annotation
NCBIEMBL
TIGR SNP
GO CSNDB
GKKEGG
15 DBs 21 Applications
D-Net based Global Collaborative Real- Time Genome Annotation
Genome Annotation
Nucleotide Annotation Workflows
How It Works
Download sequence
from Reference
Server
Save to Distributed Annotation
Server
InteractiveEditor &
Visualisation
Execute distributed annotation workflow
NCBIEMBL
TIGR SNP
InterPro
SMART
SWISSPROT
GO
KEGG
500 Web access1800 clicks200 copy/paste 3 weeks work in 1 workflow and few second execution
eDiamond Applications of SMF
Training and Differential Diagnosis “Find one like it”
Teleradiology and QC VirtualMammo
Epidemiology SMFcomputed breast density
?
Advanced CAD SMF-CAD workstation
Image guided interventions Images CourtesyDerek HillGuy’s Hospital
Image guided interventions (2)Images CourtesyGuy’s Hospital
Surgical verificationAccuracy of surgical placement against plan
• Surgeon plans on X-ray or CT, uses database of prostheses• Operation takes place using plan as guidance• Post operative X-ray evaluated for accuracy of placement• Data stored and used for short term assessment and long
term evaluation studies
Courtesy of Ian RevieDepuy International
• UK e-Science projects emphasize data federation and integration as much as computation
• Metadata and ontologies key to higher level Grid services
• e-Science projects will produce a deluge of scientific data that will need to be annotated and curated in scientific data ‘digital libraries’
Summary
Databases in the Grid
Computational Complexity
DataComplexity
OGSA – DAI Project
• Key middleware project for UK Program- Total Budget £3M (CP £1.5M)
• Three Centres involved: - Edinburgh, Manchester and Newcastle • Industrial partners:
- IBM US, IBM Hursley and Oracle UK
Goal is to develop high-quality data-centric middleware
OGSA – DAI Project
• Design Specification completed– Papers for GGF WG on Database Access and
Integration Services• Alpha versions delivered:
– Distributed Query Service– XML Database Interface– Relational Database Interface
• Beta versions by April 2003– Integrate with Globus GT3 release
e-Science and the Future of Scientific Research
‘e-Science will change the dynamic of the way science is undertaken.’
John Taylor, 2001 Need to break down the barriers between the Victorian
‘bastions’ of science – biology, chemistry, physics, …. Develop ‘permeable’ structures that promote rather than
hinder multidisciplinary collaboration Engage Computing Services and Libraries in developing a
new e-Science support service on Campus
e-Science and Computer Science
• The lesson of the Web
• The Semantic Grid
– The myGrid project
– The Discovery Net Project
• Computer Science Research and the Grid
Error 404: Page not found
‘If you want the Web to scale,
You must allow the links to fail’
Wendy Hall after Tim Berners-Lee
HTML as the ‘Fortran’ of Hypertext!
Semantic Web
Metadata & Ontologies• Metadata – computationally
accessible data about the services
• Ontologies – the shared and common understanding of a domain– A vocabulary of terms– Definition of what those terms
mean.– A shared understanding for
people and machines– Usually organised into a
taxonomy.
• Consistency — check if knowledge is meaningful
• Subsumption — structure knowledge, compute classification
• Equivalence — check if two classes denote same set of instances
• Instantiation — check if individual instance of class C
• Retrieval — retrieve set of individuals that instantiate C
Reasoning in DAML+OIL
Computer Science Challengesfrom e-Science
UK CS Team led by Tom Rodden identified 4 major research challenges arising from e-Science:
- Developing a Semantic Grid- Trusted Ubiquitous Systems- Rapid Customized Assembly of Services- Autonomic Computing
Towards a Semantic Grid
• Trace provenance from initial data to information and knowledge structures
• Techniques to allow scalable reasoning over uncertain/incomplete knowledge
• Tools for design, development and deployment of large-scale ontologies
• Support for semantic-directed knowledge discovery to complement data-mining
• Development of flexible network-based reasoning and decision support services
Trusted Ubiquitous Systems
• New theories to model, specify and analyse trust in distributed ubiquitous systems
• New quality of service and service-based models for ubiquitous systems
• New design guidelines and practices to enable the development of reusable trusted components
• New understanding of the practical engineering trade-offs required to realise trusted ubiquitous systems
Rapid Customised Assembly of Services
• New theories to describe and reason about semantics and behaviour of services and compositional effects
• Agent and service representations that promote adaptability and emergent, opportunistic and implicit arrangement of services
• New tools to support the discovery, composition and use of services based on high-level description of requirements
• Techniques to support directed automatic composition, decomposition and recomposition of services
Autonomic Computing
• Techniques to analyze, describe and reason about adaptive systems
• Management of semi-autonomous systems with policies, services and software agents
• Interoperability and reasoning across and between different autonomous domains
• Modeling and measurement of performance of QoS for autonomic structures
• Techniques to capture and represent history, context and environment
IBM Autonomic Computing Vision
Self-Healing Discover, diagnose, and react to disruptions
Self-Healing Discover, diagnose, and react to disruptions
Self-OptimizingMonitor and tune resources automatically
Self-OptimizingMonitor and tune resources automatically
Self-Protecting Anticipate, detect, identify, and protect against attacks from anywhere
Self-Protecting Anticipate, detect, identify, and protect against attacks from anywhere
Self-ConfiguringAdapt automatically to the dynamically changing environments
Self-ConfiguringAdapt automatically to the dynamically changing environments Self-
ConfiguringSelf-
Configuring
Self-Healing
Self-Healing
Self-Optimizing
Self-Optimizing
Self-Protecting
Self-Protecting
A Business Case for the Grid
• Total Cost of Ownership – TCO
• Value of Open Standards
• Industrial Applications
• Time to exploitation
• e-Utilities
Gateway
Hub Server Group
Local
Director
Network
Business LogicPresentation
WebSphereApplication
Server
NetscapeEnterprise
Server WebSphereApplication
Server
JDBC
HTTP
MQ
ProfileCapture
DatabaseServers
SecurityGateways
DB2SecurityServers
MQMQ
ApplicationLogging
GatewayLogging
MQ MQ
SecurityClient
SNA
SNA
SNA
Back-endSystems
ComplexIMSData
CICS
SysplexIMSData
SysplexIMSData
SysplexIMSData
TPF
Front-end for Web presence for financial services
Typical Financial Subsystem Configuration
Current IT EnvironmentDistributed, Heterogeneous, Complex
Gateway
Hub Server Group
Local
Director
Network
Business LogicPresentation
WebSphereApplication
Server
NetscapeEnterprise
Server WebSphereApplication
Server
JDBC
HTTP
MQ
ProfileCapture
DatabaseServers
SecurityGateways
DB2SecurityServers
MQMQ
ApplicationLogging
GatewayLogging
MQ MQ
SecurityClient
SNA
SNA
SNA
Back-endSystems
zSeries
ComplexIMSData
CICS
zSeries
SysplexIMSData
zSeries
SysplexIMSData
zSeries
SysplexIMSData
zSeries
TPF
Front-end for Web presence for financial services
Typical Financial Subsystem Configuration
Current IT EnvironmentDistributed, Heterogeneous, Complex
Complexity, TCO
Tech. Cost, Utilization
Server / Storage Utilization
52%N/AN/AStorage
2-5%5-10%30%Intel-based
<10%10-15%50-70%UNIX
60%70%85-100%Mainframes
24-hour Period Utilization
Prime-shift Utilization
Peak-hour Utilization
Source: IBM Scorpion White Paper: Simplifying the Corporate IT Infrastructure, 2000
Total Cost of Ownership: TCO
Integration32.0%
Hardware10.0%
Software12.0%
Personnel16.0%
Maintenance30.0%
IT Budgets
HardwareSoftwarePersonnelMaintenanceIntegration
32%
30%
Integration32.0%
Much More than Hardware and Software Costs
16%Personnel
16.0%
Grid Computing Sales Pitch
Storage
ApplicationsApplicationsProcessingProcessing
Operating System
DataData
I/O
Distributed Computing Over a Network, Using Open Standards to Enable Heterogeneous Operations
Grid Technology Enables
Increased Server Utilization Workload Management and Consolidation Reduced Cycle Times
Collaboration and Access to Data Federation of Data Global Distribution
Resilient/Highly Available Infrastructure Business Continuity Recovery and Failover
Supporting Heterogeneous Resources Through Open Standards….
Increased Server Utilization• Exploit distributed resources to provide
capacity for high-demand applications– Existing applications that cannot be run
effectively on a single processor
– New large scale application that provide strategic business advantages
• Reduce infrastructure cost associated with over-provisioned resources– Balance workload based on policies
– Optimize for cost or throughput
• Reduce the cost of manpower to manage and configure resources– Fewer resources to manage for the same
workload
Collaboration and Access to Data
• Enable collaboration across applications to integrate results – Leverage Distributed Data and Resources
• Support large multi-disciplinary collaborations– Link Business Processes
– Federation of Data
• Both within a single organization and between partners– Exploit Replication Services Across
Enterprises
Simulation
PricingDesign
Design
Design Analytics
The Value of Open Standards
Networking:The Internet
(TCP/IP)
Communications:e-mail
(pop3,SMTP,Mime)
Information:World-wide Web
(html, http, j2ee, xml)
Applications:Web Services
(SOAP, WSDL, UDDI)
Distributed Computing:Grid
(Globus -> OGSA)
Operating System: Linux
Sun and the Grid: ‘Grid Computing is one of the three next big
things for Sun and our customers’
Ed Zander, COO
Microsoft and the Grid: ‘The alignment of OGSA with XML Web
services is important because it will make Internet-scale, distributed Grid Computing possible’
Robert Wahbe, General Manager of Web Services
Grid Infrastructure
Industry Applications
DerivativesAnalysis
Statistical Analysis
Portfolio Risk
Analysis
Batch Throughput
Product Design
Process Simulation
FiniteElement Analysis
Failure Analysis
Cancer Research
Drug Discovery
Protein Folding
Protein Sequencing
Collaborative Research
Weather Analysis
HPC
Unique by Industry with Common Characteristics
Seismic Analysis
Reservoir Analysis
Bandwidth Consumption
Digital Rendering
Multiplayer Gaming
Primary Focus
Energy
Financial Services
Manufacturing
LS/
BioinformaticsTelco & Media
Gov’t & Education
Unlimited Numbers of Players
Distributed Artificial Intelligence
Multiple Concurrent Players
1,000 downloads of developer’s kit per week
Hot-swappable Components
Developers, Publishers, ESPs
Globalization Grid: Butterfly.net
HP, the Grid and e-Utilities
The Grid fabric for e-Utilities will be:
• Soft – malleable, multi-purpose
• Dynamic – resources will be constantly changing
• Federated – global structure not owned by any single authority
• Heterogeneous – from supercomputer clusters to PCs
John Manley, HP Labs
Timescales for Exploitation?• IBM see ‘early adopters’ of Grid technology
coming from pharmaceutical, engineering and petrochemical sectors
UK program confirms this picture (AstraZeneca, GSK, Merck, Pfizer, Rolls Royce, BAESystems, Schlumberger)
• IBM see Grid middleware being adopted by more mainstream commerce and industry in 2003/2004 timeframe
Status of the Grid• Today - ‘early adoption’ phase - just like the Web in
the early days– Industry now selling ‘IntraGrid’ solutions– Genuine Virtual Organisation ‘InterGrid’
middleware not yet mature• ‘Tomorrow’ - sophisticated combinations of services
to locate information, applications to process it, and computer systems to run them
Autonomic Middleware infrastructure capable of supporting Virtual Organisations, c-Commerce and e-Utilities will take time!
e-Government and the Grid
‘[The Grid] intends to make access to computing power, scientific data repositories and experimental facilities as easy as the Web makes access to information.’
Tony Blair, 2002
Acknowledgements
With thanks to:
Gerd Breiter, Phillipe Bricard, David Boyd,
Jens Jensen, Daron Green, Mike Brady,
Derek Hill, Carole Goble, Yike Guo,
Jeremy Frey, Bill Johnston, Ray Browne,
Jim Fleming, Anne Trefethen and many others