The Grid: The First 50 Years Ian Foster Argonne National Laboratory University of Chicago Carl Kesselman Information Sciences Institute University of Southern.

The Grid: The First 50 YearsIan Foster

Argonne National LaboratoryUniversity of Chicago

Carl Kesselman

Information Sciences InstituteUniversity of Southern California

Grid, Globus Toolkit, and OGSA

*

AbstractThe Internet and Web have had a major impact on society: by allowing us to discover and access information on a global scale, they have created entirely new businesses and brought new meaning to the term "surf." Yet simply being able to "show you stuff" and "know stuff" are ultimately unsatisfactory: we want to "do stuff"and increasingly, to "do stuff together" within distributed teams.This need has lead to the creation of the Grid, an infrastructure that enables us to share capabilities, integrating services and resources within and across enterprises, and allowing active collaborations across distributed, multi-organizational collaborations. Powered by on-demand access to computing, seamless access to data, and dynamic composition of distributed services, the Grid promises to enable fundamentally new ways of interacting with our information technology infrastructure, of doing business and practicing science. It represents perhaps the final step in the great disappearing act that will take computing out of our homes and machine rooms and into the fabric of society, where it will stand alongside telephone switches, power generators, and the other invisible technologies that drive the modern world.In this talk, we trace the history of Grid ideas, starting with the first days of the Internet in the early 1970s, proceeding to the work in the mid to late 1990s that created today's Grid, and finally looking some years into the future. We talk about accomplishments, opportunities, obstacles, and challenges--and, we hope, provide a compelling picture of the dynamic and open mix of ideas, technology, and community that are turning the Grid into reality.


*

Licklider (1960):Man-Computer Symbiosis is an expected development in cooperative interaction between men and electronic computers. The main aims are

to let computers facilitate formulative thinking as they now facilitate the solution of formulated problems, and

to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs.


*

Problem Solving in the 21st CenturyTeams organized around common goalsCommunities: Virtual organizationsWith diverse membership & capabilitiesHeterogeneity is a strength not a weaknessAnd geographic and political distributionNo location/organization possesses all required skills and resourcesMust adapt as a function of the situationAdjust membership, reallocate responsibilities, renegotiate resources


*

Context (1):Revolution in SciencePre-InternetTheorize &/or experiment, alone or in small teams; publish paperPost-InternetConstruct and mine large databases of observational or simulation dataDevelop simulations & analysesAccess specialized devices remotelyExchange information within distributed multidisciplinary teams


*

Context (2):Revolution in BusinessPre-InternetCentral data processing facilityPost-InternetEnterprise computing is highly distributed, heterogeneous, inter-enterprise (B2B)Business processes increasingly computing- & data-richOutsourcing becomes feasible => service providers of various sorts


*

The (Power) Grid:On-Demand Access to ElectricityTimeQuality, economies of scale


*

By Analogy, A Computing GridDecouple production and consumptionEnable on-demand accessAchieve economies of scaleEnhance consumer flexibilityEnable new devicesOn a variety of scalesDepartmentCampusEnterpriseInternet


*

Not Exactly a New Idea The time-sharing computer system can unite a group of investigators . one can conceive of such a facility as an intellectual public utility.Fernando Corbato and Robert Fano, 1966We will perhaps see the spread of computer utilities, which, like present electric and telephone utilities, will service individual homes and offices across the country.Len Kleinrock, 1967


*

But Things are Different Now


*

Computing isnt Really Like ElectricityI import electricity but must export dataComputing is not interchangeable but highly heterogeneous: data, sensors, services, This complicates things; but also means that the sum can be greater than the parts Real opportunity: Construct new capabilities dynamically from distributed servicesRaises fundamental questionsAchieving economies of scaleQuality of service across distributed servicesApplications that exploit synergies


*

New OpportunitiesDemand New Technology Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations

When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances (George Gilder)


*

Taking Sharing to the Next LevelSharing of communicationTelephones, mailing lists, collaboration toolsSharing of data and knowledgeWeb, semantic webWhat about the rest of the infrastructure?Services, computers, programs, sensors,


*

Existing Technologies are Helpful,but Not Complete Solutions Peer-to-peer technologiesLimited scope and mechanismsEnterprise-level distributed computingLimited cross-organizational supportDatabasesVertically integrated solutionsWeb servicesNot dynamicSemantic webLimited focus


*

Whats Missing is Support for Sharing & integration of resources, viaDiscoveryProvisioningAccess (computation, data, )Security PolicyFault toleranceManagementIn dynamic, scalable, multi-organizational settings


*

Enter the GridInfrastructure (middleware) for establishing, managing, and evolving multi-organizational federationsDynamic, autonomous, domain independentOn-demand, ubiquitous access to computing, data, and servicesMechanisms for creating and managing workflow within such federationsNew capabilities constructed dynamically and transparently from distributed servicesService-oriented, virtualization


*

Building the GridOpen source softwareGlobus Toolkit , UK OGSA DAI, Condor, Open standardsOGSA, other GGF, IETF, W3C standards, Open communitiesGlobal Grid Forum, Globus International, collaborative projects, Open infrastructureUK eScience, NSF Cyberinfrastructure, StarLight, AP-Grid,


*

Open SourceEncourage adoption of standards by reducing barriers to entryOvercome new technology Catch-22Enable broad Grid technology ecosystemKey to international cooperationKey to science-commerce partnershipJumpstart Grid industry and allow vendors to focus on value-addE.g., IBM, Avaki use GT3; Platform GlobusOpen source is industry friendly!


*

Globus Toolkit HistoryNASA beginsfunding Grid work, DOE adds supportThe Grid: Blueprint for a New ComputingInfrastructure publishedGT 1.0.0ReleasedEarly ApplicationSuccesses ReportedNSF & European CommissionInitiate Many New Grid ProjectsAnatomy of the GridPaper ReleasedSignificantCommercialInterest inGridsPhysiology of the GridPaper ReleasedGT 2.0ReleasedDoes not include downloads from: NMI, UK eScience, EU Datagrid, IBM, Platform, etc.


Chart4

0

0

0

0

0

0

40

40

40

40

40

40

50

50

50

50

50

50

50

50

50

75

75

75

75

75

75

75

100

100

100

150

200

350

350

400

425

450

498

1481

1385

2509

1448

1563

1506

1884

1801

1217

1896

3086

2916

2107

2587

1796

2254

11016

9897

14318

13050

6646

11270

8247

10607

19430

15900

11911

12004

16517

18790

26640

24371

20788

Downloads per Month from ftp.globus.org

Sheet1

Jan-990

Feb-990

Mar-990

Apr-990

May-990

Jun-990

Jul-9740

Aug-9740

Sep-9740

Oct-9740

Nov-9740

Dec-9740

Jan-9850

Feb-9850

Mar-9850

Apr-9850

May-9850

Jun-9850

Jul-9850

Aug-9850

Sep-9850

Oct-9875

Nov-9875

Dec-9875

Jan-9975

Feb-9975

Mar-9975

Apr-9975

May-99100

Jun-99100

Jul-99100

Aug-99150

Sep-99200

Oct-99350GT 1.0.0 released

Nov-99350

Dec-99400GT 1.1.1 released

Jan-00425

Feb-00450

Mar-00498GT 1.1.2 released

Apr-001481

May-001385

Jun-002509GT 1.1.3 released

Jul-001448

Aug-001563

Sep-001506GT 1.1.4 and MPICH-G2 released

Oct-001884

Nov-001801

Dec-001217

Jan-011896Anatomy of the Grid paper released

Feb-013086

Mar-012916

Apr-012107

May-012587

Jun-011796

Jul-012254

Aug-0111016

Sep-019897

Oct-0114318

Nov-0113050GT 2.0 beta released

Dec-016646

Jan-0211270

Feb-028247

Mar-0210607

Apr-0219430GT 2.0 released

May-0215900

Jun-0211911

Jul-0212004

Aug-0216517

Sep-0218790

Oct-0226640GT 2.2 released

Nov-0224371

Dec-0220788

Sheet1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Globus Toolkit Downloads

Sheet2

Sheet3

*

The Emergence ofOpen Grid StandardsIncreased functionality,standardizationCustomsolutions19901995200020052010


*

Grid CommunitiesGlobal Grid ForumStandards, information exchange, advocacy1000+ participants in tri-annual meetingsApplication communitiesE.g., physics, earthquake engineering, biomedical, etc.Software development and supportNSF Middleware Initiative, UK eScience, Globus Toolkit, EGEE,


*

Data Grids for High Energy PhysicsEnable international community of 1000s to access & analyze petabytes of dataHarness computing & storage worldwideVirtual data concepts: manage programs, data, workflowDistributed system management


*

myGrid(Goble, De Roure, Shadbolt, et al.)Imminent data deluge in bioinformaticsHeterogeneous, complex, and inter-related data sourcesIntegrated, community -wide treatment of data, literature, computational services


*

NEESgrid Earthquake Engineering CollaboratoryU.Nevada Renowww.neesgrid.org


Network for Earthquake Engineering Simulation

Field Equipment

Laboratory Equipment

Remote Users

Remote Users: (K-12 Faculty and Students)

High-Performance Network(s)

Instrumented Structures and Sites

Leading Edge Computation

Curated Data Repository

Laboratory Equipment (Faculty and Students)

Global Connections(fully developed FY 2005 FY 2014)

(Faculty, Students, Practitioners)

12/06/01 MRE Panel

*

Distributed Aircraft Maintenance Environment (Austin et al.)In flight dataAirlineMaintenance CentreGround StationGlobal Networkeg: SITAInternet, e-mail, pagerDS&S Engine Health CenterData centre


*

Industrial Perspective on Grids:A Wide Range of ApplicationsSources: IDC, 2000 and Bear Stearns- Internet 3.0 - 5/01 Analysis by SAIGrid Services Market Opportunity 2005Unique by Industry with Common Characteristics


*

Open InfrastructureBroadly deployed services in support of fundamental collaborative activitiesFormation & operation of virtual organizationsAuthentication, authorization, discovery, Services, software, and policies enabling on-demand access to critical resourcesComputers, databases, networks, storage, software services,Operational support for 24x7 availabilityIntegration with campus and commercial infrastructures


*

Open Infrastructure


*

Grid Communities & TechnologiesYesterdaySmall, static communities, primarily in scienceFocus on sharing of computing resourcesGlobus Toolkit as technology baseTodayLarger communities in science; early industryFocused on sharing of data and computingOpen Grid Services ArchitectureTomorrowLarge, dynamic, diverse communities that share a wide variety of services, resources, dataChallenging computer science research issues


*

Summary: The Grid Explained,via the BCS Lovelace Medal


Questions?


There is an increasing need in the military to support virtual organizations: human / agent communities assembled as needed out of existing resources in order to pursue common goals. Virtual organizations frequently must be created to carry out multinational peace keeping operations, joint force operations, or to respond to natural disasters.

Virtual organizations pose a range of problems, some of which are potentially amenable to technological solutions. Commanders need to locate the resources that the virtual organization requires. Elements of the VO may be part of multiple real organizations and may be distributed geographically, so proper coordination is paramount. The composition and function of the virtual organization must change as the situation changes. People may need help in adapting their work practices to meet the needs of the virtual organization.Climate Assumptions about the models

Science is unstable.

The Grid: The First 50 Years Ian Foster Argonne National Laboratory University of Chicago Carl Kesselman Information Sciences Institute University of Southern.

Documents

globus toolkit

todays grid

history of grid ideas

access information

distributed teams

seamless access

demand access

new meaning