TAB, 03. March 2006 Bruno Hoeft German LHC – WAN present state – future concept Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O.

TAB, 03. March 2006Bruno Hoeft

German LHC – WANpresent state – future concept

Forschungszentrum Karlsruhe GmbHInstitute for Scientific Computing

P.O. Box 3640D-76021 Karlsruhe, Germany

http://www.gridka.de

Bruno Hoeft


GridKaFZK-LAN

WAN (March 03rd. 06)

DWDM10GE

FZKR-Inet-IWR

X-WiNDWDMAR

HEP

10GE

T0 DWDM lightpath

T0‘ DWDM lightpathBackup (end 2006)T1 DWDM lightpathBandwidth ntd

T2 DWDM lightpathBandwidth ntd

10GE physical link

1GE physical link


X-WiNAR

GridKa - backboneFZK-LAN

HEP

DWDM

10GE10GE

10GE

T0 DWDM lightpath

T0 DWDM lightpathBackup (end 2006)T1 DWDM lightpathBandwidth ntd


10GE physical link

1GE physical link

T0CERN T1- SARA (CBF – DFN / Surfnet)

WAN (March/April 06)

Power Supply 1 Power Supply 2

9 8 7 6 5 4 3 2 1

FANSTATUS

WS-C6509-NEB

WS-SUP720

SUPERVISOR 720 WITH INTEGRATED SW

ITCH FABRIC

EJECT

DISK 0

EJECT

DISK 1

CONSOLEPORT 2

PORT 1

STATUS PHONE

WS-X6548-GE-TX

4748

373835

3625

262324

131411

121

2

4 8 P O R T

12

11

9

10

7

8

5

6

3

4

1

2

24

23

21

22

19

20

17

18

15

16

13

14

36

35

33

34

31

32

29

30

27

28

25

26

48

47

45

46

43

44

41

42

39

40

37

38

10/100/1000 BASE-TETHERNET

SWITCHING MODULE

WS-SUP720


ITCH FABRIC

EJECT

DISK 0

EJECT

DISK 1

CONSOLEPORT 2

PORT 1

WS-X6748-SFP

48 PORT GIGABIT ETHERNET SFP

STATUS

12

34

56

78

910

1112

1314

1516

1718

1920

2122

2324

2526

2728

2930

3132

3334

3536

3738

3940

4142

4344

4546

4748

WS-X6704-10GE

4 PORT 10 GIGABIT ETHERNET

LINK 1

LINK 2

LINK3

LINK 4

PORT1TX RX

PORT2TX RX

PORT3TX RX

PORT4TX RX

FANSTATUS

1

2

3

4

5

6

7

8

9


Catalyst 6500SERIES

WS-SUP720

SUPERVISOR 720 WITH INTEGRATED SWITCH FABRIC

EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

-48 V TO -60 V80 A

Cisco Systems, Inc.Switch must be in off "O" position to Install/Remove power supply.

Fastener must be fully enaged prior to operating power supply.

INPUTOK

FANOK

OUTPUTFAIL

-48 V TO -60 V80 A



INPUTOK

FANOK

OUTPUTFAIL

R-Inet-GISCisco C6509

FZKR-Inet-IWR


9 8 7 6 5 4 3 2 1

FANSTATUS

WS-C6509-NEB

WS-SUP720


ITCH FABRIC EJECT

DISK 0

EJECT

DISK 1

CONSOLEPORT 2

PORT 1

STATUS PHONE

WS-X6548-GE-TX

47

48

3738

35

36

25

26

2324

1314

11

12

1

2

4 8 P O R T

12

11

9

10

7

8

5

6

3

4

1

2

24

23

21

22

19

20

17

18

15

16

13

14

36

35

33

34

31

32

29

30

27

28

25

26

48

47

45

46

43

44

41

42

39

40

37

38


SWITCHING MODULE

WS-SUP720


ITCH FABRIC EJECT

DISK 0

EJECT

DISK 1

CONSOLEPORT 2

PORT 1

WS-X6748-SFP


STATUS

12

34

56

78

910

1112

1314

1516

1718

1920

2122

2324

2526

2728

2930

3132

3334

3536

3738

3940

4142

4344

4546

4748

WS-X6704-10GE


LINK 1

LINK 2

LINK3

LINK 4

POR

T1

TX RX

POR

T2

TX RX

POR

T3

TX RX

POR

T4

TX RX

- CNAF (CBF – DFN / Switch / Garr)(Q3/4)

Atlas T1-T1 2010

Hi-Lumi 2*500 MB/s


https://uimon.cern.ch/twiki/bin/view/LHCOPN/LHCopnTables

New location:https://twiki.cern.ch/twiki/bin/view/LHCOPN/LHCopnTables


X-WiNAR

GridKa - backboneFZK-LAN

HEP

DWDM

10GE10GE

10GE

T0 DWDM lightpath

T0 DWDM lightpathBackup (end 2006)T1 DWDM lightpathBandwidth ntd


10GE physical link

1GE physical link

T0CERN T1-SARA (CBF – DFN / Surfnet)-CNAF (CBF – DFN / Switch / Garr)(Q3/4)

WAN (2006/7)


9 8 7 6 5 4 3 2 1

FANSTATUS

WS-C6509-NEB

WS-SUP720


ITCH FABRIC

EJECT

DISK 0

EJECT

DISK 1

CONSOLEPORT 2

PORT 1

STATUS PHONE

WS-X6548-GE-TX

4748

373835

3625

262324

131411

121

2

4 8 P O R T

12

11

9

10

7

8

5

6

3

4

1

2

24

23

21

22

19

20

17

18

15

16

13

14

36

35

33

34

31

32

29

30

27

28

25

26

48

47

45

46

43

44

41

42

39

40

37

38


SWITCHING MODULE

WS-SUP720


ITCH FABRIC

EJECT

DISK 0

EJECT

DISK 1

CONSOLEPORT 2

PORT 1

WS-X6748-SFP


STATUS

12

34

56

78

910

1112

1314

1516

1718

1920

2122

2324

2526

2728

2930

3132

3334

3536

3738

3940

4142

4344

4546

4748

WS-X6704-10GE


LINK 1

LINK 2

LINK3

LINK 4

PORT1TX RX

PORT2TX RX

PORT3TX RX

PORT4TX RX

FANSTATUS

1

2

3

4

5

6

7

8

9


Catalyst 6500SERIES

WS-SUP720


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

WS-SUP720


EJECT

DISK 0

EJECT

DISK 1

CONSOLE PORT 2

PORT 1

-48 V TO -60 V80 A



INPUTOK

FANOK

OUTPUTFAIL

-48 V TO -60 V80 A



INPUTOK

FANOK

OUTPUTFAIL

R-Inet-GISCisco C6509

FZKR-Inet-IWR


9 8 7 6 5 4 3 2 1

FANSTATUS

WS-C6509-NEB

WS-SUP720


ITCH FABRIC EJECT

DISK 0

EJECT

DISK 1

CONSOLEPORT 2

PORT 1

STATUS PHONE

WS-X6548-GE-TX

47

48

3738

35

36

25

26

2324

1314

11

12

1

2

4 8 P O R T

12

11

9

10

7

8

5

6

3

4

1

2

24

23

21

22

19

20

17

18

15

16

13

14

36

35

33

34

31

32

29

30

27

28

25

26

48

47

45

46

43

44

41

42

39

40

37

38


SWITCHING MODULE

WS-SUP720


ITCH FABRIC EJECT

DISK 0

EJECT

DISK 1

CONSOLEPORT 2

PORT 1

WS-X6748-SFP


STATUS

12

34

56

78

910

1112

1314

1516

1718

1920

2122

2324

2526

2728

2930

3132

3334

3536

3738

3940

4142

4344

4546

4748

WS-X6704-10GE


LINK 1

LINK 2

LINK3

LINK 4

POR

T1

TX RX

POR

T2

TX RX

POR

T3

TX RX

POR

T4

TX RX

T0CERN T0‘

LHC-OPN 2007:T0-T1: Atlas 100 MB/s + CERN - GridKa Alice 25 MB/s +

LHCb 30 MB/s +CMS 280 MB/s 10 Gb capacity


Experiments

•Atlas2007 2008 2009 2010 2010+

•T0 T1 : 100 MB/s ???•T1 T0 : 2,51 MB/s ???•T1 T1: 50MB/s 250-300 MB/s 300-350 MB/s 500 MB/s Hi-Lumi 2x500 MB/s•T1 t2 : 30MB/s 60 MB/s 70 MB/s ~100 MB/s (3 tier-2 centres)•T2 t1 : ignore

•Alice – no roadmap•T0 T1 300 Mb/s•T1 T2 20 MB/s (for 3.5 tier-2s)•T1 T2 3 MB/s

•CMS -- no roadmap•T0 t1: 280 MB/s•T1 t2: 60 MB/s – 600 Mbit•T2 t1: 12 MB/s – 120 Mbit

•LHCb• data taking

•T0 T1: 6,7 MB/s 9 MB/s•T1 T1: 11 MB/s 22 MB/s

• reprocessing •T1 T1: 22 MB/s 60 MB/s•T0 T1: 21 MB/s 29 MB/s

• 1 month stripping•T1 T1: 76MByte/sec•T0 T1: 15MByte

T0 T1 in 2007 ?? 240MByte ??


PCSS Poznan- Alice (5-7MB/s) Warsaw University

-CMS (60 MB/s)-Alice (5-7MB/s)-LHCb ??

Cyfronet Cracow-Atlas (10/20/25/33 MB/s)-Alice (5-7MB/s)-LHCb ??

FZU Prague-Atlas (10/20/25/33 MB/s)-Alice (5-7MB/s)

LMU + MPI Munich-Atlas (10/20/25/33 MB/s)UNI Freiburg

-Atlas (10/20/25/33 MB/s)

GSI Darmstadt-Alice (5-7MB/s)

DESY Hamburg-CMS (60 MB/s)-Atlas (10/20/25/33 MB/s)

UNI Wuppertal-Atlas (10/20/25/33 MB/s)

RWTH Aachen-CMS (60 MB/s)

GridKa

Russia ??


German LHC net

• p2p star net– T2 Centres exchanging lhc data

– all Tier-2 sites linked to GridKa

– Exception: t2 – t2 link• Desy and RWTH Aachen (redundance and strong connection

between federated CMS Tier-2 site)

– Routing between German T2 centres at GridKa• one Router excl. for WAN LHC connectivities

– Interlinks to Tier-2 to Tier-3 centres (100Mbit/s cms + atlas) general research network special volumen agreement for ger. LHC / interface rates


LHC Network






-Atlas (10/20/25/33 MB/s)





GridKa

Russia ??CERN

SARA/Nikhef

CNAF/Bologna


LHC Network






-Atlas (10/20/25/33 MB/s)





GridKa

Russia ??CERN

SARA/Nikhef

CNAF/Bologna

German – LHC Network 2007:T1-T2: German GSI ~10 MB/s +

UNI Freiburg 10 MB/s +Munich 10 MB/s +RWTH Aachen 60 MB/s +UNI Wuppertal 10 MB/s +Desy 70 MB/s ??

European Belgium ???Switzerland ???FZU Prague 20 MB/sPoland 100 MB/s

German – LHC Network 2008:T1-T2: German GSI ~10 MB/s +

UNI Freiburg 20 MB/s +Munich 20 MB/s +RWTH Aachen 60 MB/s +UNI Wuppertal 10 MB/s +Desy 80 MB/s ??

European Belgian ???Switzerland ???FZU Prague 30 MB/sPoland 110 MB/s


TAB, 03. March 2006 Bruno Hoeft German LHC – WAN present state – future concept Forschungszentrum Karlsruhe GmbH Institute for Scientific Computing P.O.

Documents

mbs atlas

mbs alice

mbs9 mbs t1 t1

mbs slide

mbs cms

mbs cms

mbs lhcb

mbs lhcb