ESLEA and HEPs Work on UKLight Network. ESLEA Exploitation of Switched Lightpaths in E- sciences Applications Exploitation of Switched Lightpaths in E-

ESLEA and HEP’s Work ESLEA and HEP’s Work on UKLight Networkon UKLight Network

ESLEAESLEA

Exploitation of Switched Lightpaths in E-Exploitation of Switched Lightpaths in E-sciences Applicationssciences Applications

Multi-disciplinedMulti-disciplined Protocol DevelopmentProtocol Development Exploitation by HEP ( ATLAS and CDF), Radio-Exploitation by HEP ( ATLAS and CDF), Radio-

astronomers, E-Health, HPC astronomers, E-Health, HPC Using dedicated point-to-point light path Using dedicated point-to-point light path

channels on channels on researchresearch UKLight Network for UKLight Network for R&DR&D purposes purposes

Bulk Data Transfers / Circuit Reservation and Bulk Data Transfers / Circuit Reservation and deployment /Transport Protocols / Real Time deployment /Transport Protocols / Real Time VisualizationVisualization

HEP ConnectionsHEP Connections

RAL-CERNRAL-CERN UCL-FermilabUCL-Fermilab Lancaster-EdinburghLancaster-Edinburgh RAL-LancasterRAL-Lancaster SARA-Lancaster SARA-Lancaster Lancaster-ManchesterLancaster-Manchester

Lancaster <-> Edinburgh Lancaster <-> Edinburgh ObjectivesObjectives

Investigate the use of an alternate (in this Investigate the use of an alternate (in this case UDT) protocol to maximise the case UDT) protocol to maximise the potential of an optical circuitpotential of an optical circuit

Utilise this protocol in such a was as to be Utilise this protocol in such a was as to be of practicable use to users of the grid.of practicable use to users of the grid.

What is UDT ?What is UDT ? UDT: UDT: UUDP-based DP-based DData ata TTransfer Protocolransfer Protocol Application level, end-to-end, unicast, Application level, end-to-end, unicast,

reliable, connection-oriented, data transport reliable, connection-oriented, data transport protocol.protocol.

Approximately 90% utilisationApproximately 90% utilisation

of available bandwidthof available bandwidth

ServersServers

Hardware :Hardware : Dual Xeon 3.2GHz dual coreDual Xeon 3.2GHz dual core 2 GB RAM2 GB RAM Dual PCI-X busDual PCI-X bus 2 x Gigabit Ethernet2 x Gigabit Ethernet SATA Raid controllerSATA Raid controller 6 x SATA disks6 x SATA disks 1 x SATA system disk1 x SATA system disk

OSOS Scientific Linux 3.0.5 with 2.4.21Scientific Linux 3.0.5 with 2.4.21

Disk Testing ResultsDisk Testing Results

Ext3 Write Ext3 Write Speed / CPU Speed / CPU LoadLoad

Ext3 Read Ext3 Read Speed / CPU Speed / CPU LoadLoad

XFS Write XFS Write Speed / CPU Speed / CPU LoadLoad

XFS Read XFS Read Speed / CPU Speed / CPU LoadLoad

System DiskSystem Disk 320 Mb/s320 Mb/s

(20%)(20%)

336 Mb/s336 Mb/s

(10%)(10%)-- --

Raid 0 (write Raid 0 (write caching caching disabled)disabled)

1544 Mb/s1544 Mb/s

(96%)(96%)

1304 Mb/s1304 Mb/s

(33%)(33%)

2160 Mb/s2160 Mb/s

(63%)(63%)

1304 Mb/s1304 Mb/s

(34%)(34%)

Raid 0 (write Raid 0 (write caching caching enabled)enabled)

1143 Mb/s1143 Mb/s

(70%)(70%)

1221 Mb/s1221 Mb/s

(30%)(30%)

1184 Mb/s1184 Mb/s

(33%)(33%)

1124 Mb/s1124 Mb/s

(28%)(28%)

Raid 5 (write Raid 5 (write caching caching disabled)disabled)

90 Mb/s90 Mb/s

(6%)(6%)

363 Mb/s363 Mb/s

(10%)(10%)

97 Mb/s97 Mb/s

(3%)(3%)

348 Mb/s348 Mb/s

(10%)(10%)

Raid 5 (write Raid 5 (write caching caching enabled)enabled)

905 Mb/s 905 Mb/s

(60%)(60%)

1222 Mb/s1222 Mb/s

(30%)(30%)

1001 Mb/s1001 Mb/s

(30%)(30%)

1216 Mb/s1216 Mb/s

(30%)(30%)

Network TestingNetwork Testing Tests were performed with default kernel Tests were performed with default kernel

and application settings and then again and application settings and then again after applying changes to maximise after applying changes to maximise network speedsnetwork speeds

BDP for this link should be :BDP for this link should be : BDP = Bandwidth (MB/s) * RTT (seconds)BDP = Bandwidth (MB/s) * RTT (seconds) BDP = (1 * 1024 / 8) * (0.3 / 1000)BDP = (1 * 1024 / 8) * (0.3 / 1000) BDP = 0.0384 MB (39.32KB)BDP = 0.0384 MB (39.32KB)

Basic Network Test resultsBasic Network Test results

ProtocolProtocol ““Out of the box” Out of the box” throughput throughput

Maximum ThroughputMaximum Throughput

TCPTCP 650 Mb/s650 Mb/s 940 Mb/s940 Mb/s

UDPUDP 957 Mb/s957 Mb/s 957 Mb/s957 Mb/s

UDTUDT 950 Mb/s950 Mb/s 950 Mb/s950 Mb/s

File Transfer Test ResultsFile Transfer Test Results

Transfer Transfer ProgramProgram

““Out of the box” tuningOut of the box” tuning Maximal ThroughputMaximal Throughput

ftpftp 565 Mb/s565 Mb/s 820 Mb/s820 Mb/s

GridFTPGridFTP 570 Mb/s570 Mb/s 850 Mb/s850 Mb/s

UDT test UDT test applicationapplication 460 Mb/s460 Mb/s 935 Mb/s935 Mb/s

What next ?What next ?

The Basic Network tests and the File The Basic Network tests and the File transfer tests need to be re-performed transfer tests need to be re-performed once the UKLight link between Lancaster once the UKLight link between Lancaster and Edinburgh is fully functionaland Edinburgh is fully functional

Integration of UDT into a functional Integration of UDT into a functional GridFTP server and clientGridFTP server and client

Deployment of modified software into test Deployment of modified software into test LCG sites.LCG sites.

Lancaster<->RAL LinkLancaster<->RAL Link

T1-T2 transfer testingT1-T2 transfer testing Avoid production network induced Avoid production network induced

bottlenecksbottlenecks Firewall @ RALFirewall @ RAL Internal LAN trafficInternal LAN traffic

Tested using :Tested using : Command line srmcp in shell scriptCommand line srmcp in shell script FTS controlled transfersFTS controlled transfers

AchievedAchieved Peak of 948MbpsPeak of 948Mbps Transferred:Transferred:

8TB in 24 hours - 800+ Mbps aggregate rate8TB in 24 hours - 800+ Mbps aggregate rate 36TB in 1 week - 500+ Mbps aggregate rate 36TB in 1 week - 500+ Mbps aggregate rate

Over 800Mbps when running, but 0Mbps in downtimes a Over 800Mbps when running, but 0Mbps in downtimes a problemproblem

Parallel file transfers increase rateParallel file transfers increase rate Better utilisation of bandwidthBetter utilisation of bandwidth Staggered initialisation of transfers reduces overhead from Staggered initialisation of transfers reduces overhead from

initialisation/cessation of individual transfers. Rate increase initialisation/cessation of individual transfers. Rate increase from 150Mbps to 900Mbpsfrom 150Mbps to 900Mbps

2% (18Mbps) reverse traffic flow for 900Mbps transfer2% (18Mbps) reverse traffic flow for 900Mbps transfer

FTS transfers not yet as successful as srmcp only transfersFTS transfers not yet as successful as srmcp only transfers Greater overheads?Greater overheads?

More optimisation needed More optimisation needed Single FTS file transfer gives 150Mbps Single FTS file transfer gives 150Mbps

Same as srmcp Same as srmcp Concurrent FTS file transfers scales at lower rate than srmcpConcurrent FTS file transfers scales at lower rate than srmcp

All single stream transfersAll single stream transfers FTS tests currently used single source fileFTS tests currently used single source file

Srmcp used with multiple source filesSrmcp used with multiple source files Rate varies dependent on directionRate varies dependent on direction

Possibly explained by difference in dCache setupPossibly explained by difference in dCache setup V0 DependencyV0 Dependency kernel settingskernel settings Disk I/O limitationsDisk I/O limitations SRM pool load balancingSRM pool load balancing To be investigatedTo be investigated

File size affects rate of transfer File size affects rate of transfer Single stream rate varies 150 to 180 Mbps with increase from 1 to 10 GB file sizeSingle stream rate varies 150 to 180 Mbps with increase from 1 to 10 GB file size

Lancaster<->SARA LinkLancaster<->SARA Link

Link not yet activeLink not yet active Tests similar to Lancaster-RAL and Lancaster-Edinburgh Tests similar to Lancaster-RAL and Lancaster-Edinburgh

Tests Tests Bulk File TransfersBulk File Transfers UDT Protocol TestingUDT Protocol Testing

Study of effect of International/Extended link lengthStudy of effect of International/Extended link length SARA storage capacity underused, RAL capacity SARA storage capacity underused, RAL capacity

currently too small for UK simulation storagecurrently too small for UK simulation storage Also, SARA to test ATLAS Tier1 fallback scenario (FTS Also, SARA to test ATLAS Tier1 fallback scenario (FTS

catalogues etc.)catalogues etc.) Are we capable of connecting to an alternate Tier1?Are we capable of connecting to an alternate Tier1?

Lancaster<->Manchester LinkLancaster<->Manchester Link

Intra-Tier2 site TestingIntra-Tier2 site Testing ““Homogeneous Distributed Tier2”Homogeneous Distributed Tier2” dCache Head node at Lancaster, pool nodes dCache Head node at Lancaster, pool nodes

at both Lancaster and Manchesterat both Lancaster and Manchester Test Transfers to/from RALTest Transfers to/from RAL Test of Job submission to close CE/WN’s Test of Job submission to close CE/WN’s Possible testing of xrootd within dCachePossible testing of xrootd within dCache

www.eslea.uklight.ac.ukwww.eslea.uklight.ac.uk Connecting to UKLightConnecting to UKLight DocumentsDocuments