ScotGRID:The Scottish LHC Computing Centre Summary of the ScotGRID Project Summary of the ScotGRID Project Phase2 of the ScotGRID Project Phase2 of the.

Georg Raff elt, Max-Planck-I nstitut für Physik, München, Germany CMB & Cosmology: Where Are We?, Ringberg Castle, Tegernsee, Germany, 2-6 December 2002

ScotGRID:The Scottish LHC Computing ScotGRID:The Scottish LHC Computing CentreCentre

Summary of the ScotGRID ProjectSummary of the ScotGRID Project

Phase2 of the ScotGRID ProjectPhase2 of the ScotGRID Project

Phase1 of the ScotGRID ProjectPhase1 of the ScotGRID Project

Introduction: ScotGRID GoalsIntroduction: ScotGRID Goals


The LHC Computing

Challenge for Scotland

2000: JREI Bid The JREI funds will make possible to commission and fully exercise a prototype LHC computing centre in

Scotland

The Centre would provide:

1. Technical service based for the grid services

2. DataStore to handle samples of data Analysis

3. Significant simulation production capability

4. Excellent network connection RAL / Regional sites

5. Support grid middle devel. with CERN/RAL

6. Support core software devel. within LHCb/ATLAS

7. Support user applications in other scientific areas

This will enable us to answer: Grid viable solution for LHC computing challenge,TDR Can a two-site Tier-2 centre be setup and operate effectively How can network topology between Ed,GL, RAL & CERN

This will enable us to answer: Grid viable solution for LHC computing challenge,TDR Can a two-site Tier-2 centre be setup and operate effectively How can network topology between Ed,GL, RAL & CERN

ScotGRID: The Spirit of the ProjectScotGRID: The Spirit of the Project

http://www.scotgrid.ac.uk/


2002 200520042003

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

Q1 Q2 Q3 Q4

Q1 Q2 Q3 Q4

2001

Q4

PrototypePrototypeTo To

ProductionProduction

PrototypePrototypeTo To

ProductionProductionDelivery of more Kit…

End of JREI fundingStart of ScotGRID-II

Start of GridPP-II

Links to other applications…

Production Tier-2 Site

Time-Line for a Production Tier-2 & beyondTime-Line for a Production Tier-2 & beyond

Completed: Phase1 Phase2

Future Upgrades ?

Georg Raff elt, Max-Planck-I nstitut für Physik, München, Germany CMB & Cosmology: Where Are We?, Ringberg Castle, Tegernsee, Germany, 2-6 December 2002Georg Raff elt, Max-Planck-I nsti tut f ür Physik, München, Ge rmany CMB & Co smology: Where Are We?, Ringberg Castle, Tegernsee , Ger many, 2-6 December 2002

ScotGRIDScotGRID-- PhasePhase-- 1:Completed Successfully1:Completed Successfully

HardwareHardware ApplicationsApplicationsMiddlewareMiddleware

ScotGRID Processing nodes at GlasgowScotGRID Processing nodes at Glasgow• 59 IBM X Series 330 dual 1 GHz Pentium III with 2GB memory • 2 IBM X Series 340 dual 1 GHz Pentium III with 2GB memory • 3 IBM X Series 340 dual 1 GHz Pentium III with 2GB memory • 1TB disk • LTO/Ultrium Tape Library • Cisco ethernet switches

ScotGRID Storage at EdinburghScotGRID Storage at Edinburgh• IBM X Series 370 PIII Xeon • 70 x 73.4 GB IBM FC Hot-Swap HDD


ScotGrid reached ~ 580,000th processing hour

Tier-2 Web-Based Monitoring

http://www.scotgrid.ac.uk/


ScotGrid accumulated statsScotGrid accumulated stats

Significant non-LHC application use

Q1Q2Q3


ScotGrid JREI project includes a mid-term hardware upgrade.

As part of GridPP planning, we need to upgrade from Prototype to Production Tier 2 status by 2004.

JREI funding left to be spent by June 2003:

Edinburgh £220k Glasgow £30k

£250k

ScotGRID Phase-2 : Just Started ScotGRID Phase-2 : Just Started


Upgrade Database Server in Edinburgh xSerier 440 (8 Way 1.9GHz Intel Xeon, 1MB L3 Cache, 32GB Memory, 36GB Internal Disk)

16 – 20 TB of Disk Storage in Edinburgh FAStT700 or FAStT900 (see next slide)

5 TB of Disk Storage in Glasgow Agreed to relocate existing FAStT500 from Edinburgh to Glasgow

Edge Servers for Edinburgh EDG SE:1 from Glasgow (x340) LCFG, CE: 2* x205, 1 x 1.8 GHz, 256MB, 40GB

Summary of ScotGRID UpgradeSummary of ScotGRID Upgrade

New Kit for Glasgow CDF & eDIRT Kit


Specifications IBM

FAStT700IBM

FAStT900

RAID Controllers Dual Dual

Max Cache Size 2 GB 2 GB

Host interface 1 or 2Gb FC 1 or 2 Gb FC

# Host Ports 8 8

Disk Drive Interface 1 or 2Gb FC 1 or 2 Gb FC

Max # of Drives 224 224

Max Raw Capacity (73 GB HDD) 33TB 33TB

Performance IOPS (cache reads) 110,000 148,000

Performance IOPS - read(random disk I/O) - write

40,0008,500

53,200*10,900*

Performance MBps - read(sequential disk I/O) - write

390240

772*530*

Max Partitions (with SW upgrades) 64 16/(64)

Max Logical Drives (with SW upgrades) 2048 2048

RAID Levels 0,1,3, 5, 10 0,1,3,5,10

20TB vs. 18TB

Storage SystemStorage System


Dual FastT900+ 24 TB

IBM@server: xSeries 440 8* Xeon

(1.9GHz) 32 GB, Scalable configuration

x340 dual PIII/1 GHz /2 GbyteRH7.3

x205 P4/1.8 GHz /256 MBRH7.3

x205 P4/1.8 GHz /256 MBRH7.3

Storage Nodes

CE

LCFG

ScotGRID-Edinburgh: Schematic ScotGRID-Edinburgh: Schematic

Fibre connectionGbps


The CDF kit has 1 rack with only UPS 1 rack with 10 Dell dual Xeon 2.4 GHz 4GB, 7.5 TB disk, UPS 1 rack with IBM x370 8 of 700 MHz Pentium III with 8GB, 1 TB via FAStT500, UPS 1 tape library

ScotGRID-Glasgow: CDF KitScotGRID-Glasgow: CDF Kit

IBM: http://www.xcat.org/CDF: http://www.scali.com/


The eDIKT kit has: http://www.edikt.org/ 4th rack on the right and has 6 IBM x335 (1U high) dual Xeon 2.4 GHz and 1.5 GB memory 2 IBM BladeCentres (7U high) each with

14 Blades x335 dual Xeon 2.4 GHz and 1.5 GB memory

ScotGRID-Glasgow: eDIKT KitScotGRID-Glasgow: eDIKT Kit


Possible ScotGRID ConfigPossible ScotGRID Config

Cluster

Worker

nodes

Front-end

Globus gatekeeper+ job manager

Globus GRIS

EDG Job GridManager

Information Providers: queues, jobs, users

PBS

Working Directories/session1/session2

...

GridFTP

StorageElement(GridFTP)

control connection

data connection

GlobusReplicaCatalog

GIIS

GIIS

GIIS

GIIS

Cluster

Cluster

Tier2 GateWay (Broker)

A

u

t

h

VirtualOrg.Catalog


Summary of Tier-2 ScotGRID ProjectSummary of Tier-2 ScotGRID Project

Have Successfully completed Phase-1

ScotGRID had been upgraded – Phase-2 eDIRT Kit, CDF Kit HPS + Larger DataStore

ScotGRID: will concentrate on commissioning the new hardware during the next few month. before going to EDG2.0

There will be an ad. for ScotGRID Technical Coordinator

We will have to start to think about restructing the ScotGRID team for production running – early next year

In the New Year plan to integrate the Durham CPU Farm


BackUp Slid

es


Internet VLAN

10.0.0.0 VLAN

100 Mbps

1000 Mbps

Masternode Storage Nodes Head Nodes

Compute Nodes

Campus Backbone

59 x330 dual PIII 1GHz/2 Gbyte

x340 dual PIII/1 GHz /2 Gbytex340 dual PIII/1 GHz /2 Gbytex340 dual PIII/1 GHz/0.5 Gbyte

ScotGRID-Glasgow: Schematic ScotGRID-Glasgow: Schematic


Disk Arrays(Total 4.6 Tb)Disk Arrays(Total 4.6 Tb)

FastT 500 Storage ControllerFastT 500 Storage Controller

Server (4*Pentium Xeon, 16Gb RAM)Server (4*Pentium Xeon, 16Gb RAM)

SRIF Network

ScotGRID-Edinburgh: Schematic ScotGRID-Edinburgh: Schematic

ScotGRID:The Scottish LHC Computing Centre Summary of the ScotGRID Project Summary of the ScotGRID Project Phase2 of the ScotGRID Project Phase2 of the.

Documents

esciencescotgrid phase

edikt kitcp violation

scotgrid team

scotgrid project

escienceinternet vlan10

esciencegeorg raffelt

tbstorage systemcp violation

esciencex340 dual piii1