R. Hughes-Jones Advanced Networks Workshop Montreal November 2000 Networking for the Particle Physics Community - a UK Perspective uIntroduction to the.

R. Hughes-Jones Advanced Networks Workshop Montreal November 2000

Networking for the Particle Physics Community - a UK Perspective

Introduction to the PPNCG Network TopologiesMonitoring ActivitiesPeering to the US and CanadaGrid Work SuperJANET4


Introduction to the PPNCGMembership includes HEP and Astronomy users

–Ensure the community has the required networking facilities–Monitor end-to-end performance–Investigate new network applications / technologies–Provide advice on kit / facilities– URL icfamon.rl.ac.uk/ppncg/title.html

Acts as an Interface between HEP and UKERNA–Collaborations: Multicast, H.323, QoS tests, SuperJANET4

Active Network Monitoring–Uses the ICFA monitoring tools and the PPNCG traceping and ftp –Monitor MANs / SuperJANET & the networks to CERN, DESY, US–Report problems to UKERNA


SuperJANET Connections in the UK

Bishop Grosseteste

Warwick

Lancaster

Trinity

Afan

Merthyr

CoventryTechnical College

Northern College

Leeds

Bristol

Londo

n

Manch

ester CWC

O-BEN

R-BEN

Nott-BEN

Le-BEN

Ncl-BEN

Ed-BEN

C-BEN

ULCC-BEN

Cf-BEN

Be-BEN

Gla-BEN

Ex-BEN

Bham-BEN

Bris-BEN

155 Mbps

34 Mbps

CWC

Soton-BEN

Rdg-BEN

BWEMAN

MidMAN

LMAN

EMMAN

NorMAN

YHMAN

SWAN

SWMAN

FaTMAN

AbMANClydeNET

HomertonHinxton Hall

ITE Monkswood

BAS

Anglia Poly

Cambridge

RAL

Brookes

Oxford

Portsmouth

BEN

PoP Site

MAP

Birkbeck

UCL

IOE

SOASRVC LSHTM

Southbank

LondonInstitute

LondonGuildhall

Imperial

Westminster

KingstonRoehampton

LBS

RCA

Birmingham

Westhill

Newman

Coventry

UCEBWolverhampton

AstonQMW UEL

Telehouse

Goldsmiths

ULCC

Swansea

Carmarthen

Lampeter

Soton

Nottingham

Leeds

GreenwichRavensbourne

St Andrews College

Glasgow

Exeter

Plymouth

Dundee

Cardiff

Bristol

Belfast

St MaryStranmill

BGS

Bath

Bath College

HEFCE

UWE

RobertGordon

AberdeenSt Andrews

Abertay

ULCC

NewportWCM

DSwansea Institute

Glamorgan

St Davids

Cardiff Institute

GlasgowCaledonia Strathclyde

RSAMD

GSA

Paisley

EaStMAN

Edinburgh

Napier Queen Margaret

Stirling Heriot

Watt

ECA

Sunderland

Northumbria

DurhamUNN

Teesside

Nott Trent

Derby

HuddersfieldSheffield

Linc & Humb

Sheffield Hallam

Leeds Metropolitan

Hull

Scarborough

Bradford

York

MCC-BEN

MCC

NEWI

Bangor

G-MING

Salford

Manchester Metropolitan

RNCM

UMIST

Liverpool LJMU

NNW

Central Lancaster

Daresbury

PO Lab

ITE Bangor

Keele

Bolton Inst.

Liverpool Hope

Staffordshire

Edge Hill

Sussex

Brighton

Edinburgh

UHI

Chichester

Yale

ColegLlysfasi

Deeside

WCoH

Swansea College

Bridgend

Gorseinon

Aberdare

Neath Pembroke

Llandrillo Menai

Aberystwyth

HEAnet Ulster

HarlechPowys

Meirion-DwyforCeredigion

MST Radar

Falmouth

Darlington

BGSExeter

St Mark & John

Bucks

UKERNA

C&G

Seeseparate diagram

See separate diagram

Bournemouth

Royal HollowaySurreyReading

Cranfield

UEA

Essex

U-Net

Leicester

UC Northampton

UCL

CWC

Mid KentCollege

Stephenson

ChichesterCollege

Soton City

BCFTCS

LCC

City CollegeManchester

Halton

NewcastleNewcastle

CarlisleCollege

Cleveland College

New College Durham

Southport

SandwellWorcester Coll. CWC


SuperJANET III Backbone


SuperJANET – ULCC 3 * 155 Mbit links + 2 * 34 Mbit

Plus 3 * 155 Mbit links4 * 155 Mbit transatlantic links

Oct 00

SuperJANET Connections in London

August 2000


UK to Europe TEN-155 October 2000

Peering with CANARIE made in LondonUses 10 (15) Mbit PVC on the Dante-US line for all Europe4 * 155 Mbit European links operational


Monitoring: ping plots UK - CANARIE Ping average round trip time & Lost packet % for icfamon.dl.ac.uk – cepheid.physics.utoronto.ca

15 Oct 00 100 byte packets 15 Oct 00 1000 byte packets


Monitoring: US Traffic - Trends UKERNA Traffic Kbit/s. Blue Traffic from US Maroon Traffic to US10 Nov 1999 1 Nov 2000

UKERNA – US Traffic Daily totals Mbytes

Weekend

Summervacation

Weekday

John Macallister PPNCG OxfordXmas 99


Monitoring: ping plots UK - US Ping average round trip time & Lost packet % for 100 byte packets icfamon.dl.ac.uk – ns2.slac.stanford.edu

13 Oct 99 15 Oct 00


Peering with ESnetUsed CAR and WRED to give priority to Esnet traffic into JANET.Rtt and packet loss for 100 byte packets.Tests DL to SLACStart 21 Feb 00

CAR+WREDenabled

Config changes toCAR and WRED

Further configchanges

High Priority

Normal Priority

Robin Tasker PPNCG DL


Peering with ESnet and Abilene rtt and packet loss for 100 byte packetsPlot start 16 May 00

ESnetHigh PriorityDL - SLAC

AbileneHigh PriorityDL - Stanford

Further configchanges

Change in bandwidthrestrictions from Abileneinto JANET


The Grid

Government has discussed 165M pounds to develop Computing Grid Infrastructure.Across Research Councils Initiative – not just Particle Physics Setting up management structure to co-ordinate PPARC activities.Have run Globus workshop so US developers could inform the community.Several sites now connected using Globus toolkit.Web site www.datagrid.org.uk


The Grid

Strong involvement in the EU DataGrid middleware proposal.

–Proposal accepted – Budget of 9.8M Euros

HEP experiments will need world Grid access.Grid Networking interests include:

–High throughput, reliable, very large data transfers–QoS issues – diffserv, RSVP and packet marking–Monitoring, Metrics, and prediction accuracy–Modelling

Physics Applications


The Grid: PAW with Globus

Core ServicesMetacomputing

Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf

High-level Services and Tools

DUROC globusrunMPI Nimrod/GMPI-IO CC++

GlobusView Testbed Status

GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDPLocal Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP

Toolkit


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDPLocal Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP

Toolkit


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDPLocal Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP

Toolkit

Linux GASS Server

Sun GASS Server

SGI GASS Server

CampusLAN

Zebra

HBOOK


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDPLocal Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP

Toolkit

PAW

Next Steps:•Working on “root”•Brokers•Data Location•Security + Global farms

BaBar Tau Analysis

Andrew McNab Manchester

Demonstrated at theEU IST 2000 NiceData from Italy & UK


ATM MAN 7 routers ~0.8ms / router Router+link 0.4 s/byte DL-Liverpool ~10 ms rtt - agreement

The Grid:Measurements on SuperJANET

64 bytes pc-SJ-DL req-resp

0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)


0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)


0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)1500 bytes pc-SJ-DL req-resp

0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)


0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

Sigma ~1.2 ms

y = 2.2589x + 5643.6

y = 2.1519x + 5533.9

y = 0.107x + 109.71

0

2000

4000

6000

8000

10000

12000

0 500 1000 1500 2000 2500 3000

Message length bytes

La

ten

cy u

s

0

50

100

150

200

250

300

350

400

450

500

La

ten

cy f

or

PC

=P

C u

s

PC-SJIII-DL PCdt PC-SJIII-DL PCPC=PC

Latency Mancester - DL

Richard Hughes-Jones PPNCG Manchester


The Grid: Latency on SuperJANET

64 bytes pc-SJIII-sun req-resp

0

100

200

300

400

500

600

10000 15000 20000 25000 30000 35000 40000 45000

Latency us

N(t

)


0

100

200

300

400

500

600

10000 15000 20000 25000 30000 35000 40000 45000

Latency us

N(t

)


0

100

200

300

400

500

600

10000 15000 20000 25000 30000 35000 40000 45000

Latency usN

(t)


0100200300400500600700

10000 15000 20000 25000 30000 35000 40000 45000

Latency us

N(t

)


0

100

200

300

400

500

600

10000 15000 20000 25000 30000 35000 40000 45000

Latency us

N(t

)

64 bytes pc-MC req-resp

0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

2500

N(t

) g

w-m

c


0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

N(t

) g

w-m

c


0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

2500

N(t

) g

w-m

c


0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

2500

3000

N(t

) g

w-m

c1024 bytes pc-MC req-resp

0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

2500N

(t)

gw

-mc

Pink: LAN Blue: MAN-SJ CORE Pink: Man-RAL MB Blue: Man-RAL ~5 s nice exponential ~ 2 ms 4.5 -> 6.7 ms


The Grid: Throughput on SuperJANET

Manc – UCL 10 Mbit Manc – Glasgow

0

2

4

6

8

10

12

0 200 400 600 800 1000 1200 1400

Transmit Time/packet us

Re

cv W

ire

ra

te M

bit

s/s

100 bytes 200 bytes400 bytes 800 bytes600 bytes 1000 bytes1200 bytes 1400 bytes

0

5

10

15

20

25

30

35

40

45

50

0 200 400 600 800 1000 1200 1400


Re

cv W

ire

Ra

te M

bit

/s

100 bytes 200 bytes400 bytes 600 bytes800 bytes 1000 bytes1200 bytes 1400 bytes

Manc – RAL 24 Mbit link Manc – RAL 2 Mbit Managed BW

0

5

10

15

20

25

30

35

0 500 1000 1500 2000 2500


Rec

v W

ire

Rat

e in

clud

ing

ATM

Mbi

t/s

100 bytes 200 bytes

400 bytes 600 bytes

800 bytes 1000 bytes


0

0.5

1

1.5

2

2.5

0 500 1000 1500 2000 2500 3000Transmit Time/packet us

Re

cv W

ire

Rat

e in

clu

din

g A

TM

Mb

it/s

100 bytes 200 bytes

400 bytes 600 bytes




The Grid: Multi-stream ftp Oxford to SLACSingle stream 8 ftp streams ~ 0.8 single stream

John Macallister PPNCG Oxford


The Grid: Multi-stream ftp SLAC to Oxford


SuperJANET4

SuperJANET4 Topology based on Worldcom Trident Network

–Tariff: distance independent–SLA: Fix problems not refunds

Backbone initially SDH based

2.5 Gbit (STM-16)Gigabit capacity with growth built in 10 Gbit in 2001,

20 Gbit in 2002Link existing Regional Networks Migration to DWDM during 2001Technology review in 2003


SuperJANET4: Backbone and Access links

Worldcom supply the transmission:–SDH–Circuit protection – fallback in 50 ms

UKERNA overlay the IP:–Packet over SDH–Core PoP IP routers at Worldcom–Backbone Access Router at MANs–Routers managed by UKERNA NOSC

Access Link Fibers:–Installed by Worldcom–Large MAN 2.5 Gbit -> 10-20 Gbit–Medium MAN 622 Mbit -> 2.5 Gbit

4 node DWDM development net


SuperJANET4: Worldcom High Level Design

DX DX

DX

DX

DX

DX

DX

DX

DX

DX

DX

DX

DX

DX

DX

DX

Manchester

London

EdinburghGlasgow

Warrington

Newcastle

Leeds

Leicester

PortsmouthExeter

Bristol

Reading

Birmingham

Cardiff

Nottingham

Greenwich

Cambridge

Belfast

STM16

STM 4

STM leased

Dark Fibre

STM 64 WC

Metro DWDM


SuperJANET4: Routing

CISCO 12016 routers from LogicalCore PoP

–Responsible for switching packets–No “edge” functions – congestion avoidance–Symmetric L3 routing – eBGP peering to BAR–Packet over SDH

Backbone

C-PoP

SDH ADM

Regional MAN

BAR

AccessLinks

eBGPPeering

Backbone Access Router–Provide filtering – drop if src. not from MAN–Traffic management – classify packets–Interface to MANs with Gigabit Ethernet

Multicast–Integral part of SuperJANET4 – not tunneled –Run sparse-mode PIM


SuperJANET4: MAN Development

Support for Router and Link upgradesIncrease:

–Capacity 1 Gbit now 10-20 Gbit in 2 years–Resilience – 99.95 % uptime–Reach – additional sites & schools

NetNorthWest as an example

NNW

SuperJANET4 BAR

Manchester

Liverpool

Preston

SDH

Gigabit Ethernet

SDH Trunk

SDH trunks 622 Mbit, will be increased with traffic load


SuperJANET4: Network Services

Services co-located at CPoPs or BARs–Provides High BW access

Centrally managed by UKERNA–Remote console access–Remote power management–Network and access control provided

Provider responsible for service not networkExamples:

–DNS servers–www caches–usnet servers–Video servers - content from academic / external–Videoconference H.323 MCUs

Backbone

Regional MAN

BAR

C-PoP

AccessLinks

Server

SDH ADM

Server

Server


SuperJANET4: Project UpdateRollout Status:

–Backbone in place at Worldcom C-PoPs – SDH resilience shown.–Routers at Logical UK, acceptance in progress, delivery 15 Nov 00–Manchester-ULCC traffic live on 5 Dec 00–All sites Using SuperJANET4 by 31 Mar 01

Network Users–HE sites – e.g. Universities already on SuperJANET III–FE sites – e.g. Colleges almost all connected at 2 Mbit–Schools – Further study in progress

Challenges–Quality of Service

•Priority traffic – packet marking & WRED

•Jitter sensitive traffic H.323 – Low latency queuing MDRR

–MPLS•VPNs


Cisco Router Acceptance Testing


Conclusion & Status

Network Links to US and Europe are OK (at the moment !)Peering with Abilene, ESnet and CANARIE OK, interesting effects with CAR – WRED !Considerable UK interest in the Internet2 tests Grid: active in several areas including networksSuperJANET4 rollout is on target

R. Hughes-Jones Advanced Networks Workshop Montreal November 2000 Networking for the Particle Physics Community - a UK Perspective uIntroduction to the.

Documents

ukerna slide

janet slide

backbone slide

us maroon traffic

esnet traffic

uk bishop grosseteste

esnet high priority

mbit links u4