J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011) 12/04/2011 1 Improving CMS data transfers among its distributed Computing Facilities J. Flix Port d'Informació Científica (PIC), Barcelona, Spain N. Magini CERN IT-ES-VOS, Geneva, Switzerland A. Sartirana École Polytechnique, Palaiseau, France [On behalf of CMS experiment] EGI User Forum – 11-14 April 2011, Vilnius, Lithuania
30
Embed
Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 1
Improving CMS data transfers among its
distributed Computing Facilities
J. FlixPort d'Informació Científica (PIC), Barcelona, Spain
N. MaginiCERN IT-ES-VOS, Geneva, Switzerland
A. SartiranaÉcole Polytechnique, Palaiseau, France
[On behalf of CMS experiment]
EGI User Forum – 11-14 April 2011, Vilnius, Lithuania
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 2
Outline
‣ CMS data transfer workflows and Management System
‣ Measuring the performance of CMS data transfers
‣ Using data transfer statistics to improve transfer operations
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 3
CMS Computing Model
CAF
900 MB/s agg.
50-500MB/s10-20MB/s
CMSdetector
WLCG Computing Grid InfrastructureTIER-0
CERN
TIER-1 TIER-1
TIER-2
7 Tier-1s(“online” to the DAQ)
High availability centres
Custodial mass storage of share of data
Data reconstruction and reprocessing
Data skimming & selection
Distribute analysis data ���� Tier-2s
~50 Tier-2sin ~20 countries
End-user physics analyses
Detector Studies
Monte Carlo Simulation ���� Tier-1
Distribute analysis data ���� Tier-2s
Tier-0(the accelerator centre)
Data acquisition & initial processing
Long-term mass data storage
CMS CERN Analysis Facility (latency critical data processing, high priority analysis)
Distribution of data ���� Tier-1 centres
TIER-2 TIER-2
10-50MB/s agg.
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 4
‣ Coupled & highly-specialized components which fulfill a specific “simple” task in a reliable way‣ Central agents: routing, task assignment, …‣ Run at CERN‣ Site-specific agents: download, export, mass storage staging and migration …‣ Run at every CMS site on a gLite VOBOX (to simplify proxy management) or UI
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 5
‣ Agents inter-communicatewith a central blackboard
‣ transfer Management DB (TMDB) @ Oracle RAC backend‣ Block replica location & file mapping‣ Block subscription and allocation‣ File metadata information (filesize, cksum, etc)‣ Transfer state (at node; in transfer; wanted; available)
‣ Transfer status monitored from PhEDEx webpage‣ DataService to retrieve information from the database through simple http access‣ 1 T0, 7 T1’s, 50 T2’s + T3’s in current PhEDEx transfer topology (~100 Sites)
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 6
CMS Transfer Workflow
PhEDExCMS transfer management system
WLCG
MiddlewareFTS
SRM
gridFTP
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 7
CMS Transfer Workflow
Transfer request is placed through
PhEDEx web interface
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 8
CMS Transfer Workflow
PhEDEx central agents create and distribute transfer tasks to site agents
It takes into account the
aggregated throughput
and errors on the links
in the routing algorithm
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 9
CMS Transfer Workflow
PhEDEx Download agent submits
transfer batch job to FTS server
FTS chosen according
to CMS policy
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 10
CMS Transfer Workflow
FTS contacts source and destination SRMs to get transfer URLs
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 11
CMS Transfer Workflow
FTS executes the transfer as third-party with gridFTP
FTP extension
GSI security
Can run parallel streams
Flexible: other configurations can be
also used
a) srmCopy started by FTS server
b) srmCopy started directly by
PhEDEx Download agent with
SRM client (not encouraged)
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 12
gLite File Transfer Service (FTS)
‣ Provides scheduling of multiple asynchronous file transfers on CHANNELS‣ Single direction transfer queue between two endpoints- Not tied to a physical network path
‣ Each endpoint (source and destination) can be:
‣ A single site, e.g.‣ CERN-RAL
‣ IN2P3-BELGIUMULB
‣ A group of sites (“cloud”), e.g.‣ RALLCG2-CLOUDCMSITALY
‣ CLOUDCMSFRANCE-RALLCG2
‣ All sites (“star”), e.g.‣ CNAF-STAR
‣ STAR-FNAL
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 13
Typical FTS server deployment (CMS)
Tier-0FTS
Tier-1
Tier-1
.
.
.
At Tier-0Dedicated channel to each of the Tier-1s
Tier-1FTS
Tier-1
Tier-1
.
.
.
At each Tier-1Dedicated channel from each of the other Tier-1s
Tier-1FTS
Tier-2assoc.
Tier-2assoc.
.
.
.Tier-2others
At each Tier-1Dedicated channels to and from each of the associated Tier-2s
CLOUD and/or STAR channels to/from other Tier-2sSTAR-T2 channels for each associated Tier-2
Tier-1...
Tier-2others
.
.
.
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 14
FTS channels :: configuration
‣ FTS channel configuration defines:‣ Transfer limits
‣ Maximum number of concurrent active transfers‣ Protect network, storage
‣ Shared among VOs according to policy
‣ Transfer priorities
‣ Between users in the same VO on a channel
‣ Transfer parameters
‣ Number of parallel TCP streams, buffer size
‣ Timeouts (SRM Gets/Puts; No_Tx_activity; …)
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 15
FTS channels :: throughput
‣ Overall throughput for a link in a channel
Link throughput = rate/stream * streams/file * active transfers/link
‣ In a dedicated channel:
‣ Expect ~constant rate/stream up to saturation (in absence of source/destination contention)
‣ Fixed number of available active transfer slots/link
‣ In cloud or star channel:
‣ Rate/stream can be significantly different for links in same channel
‣ Available active transfer slots/link depends on overall channel occupancy‣ Slow links keep transfer slots busy for longer (no ‘fair-share’ concept in STAR/CLOUD channels)
� Low performing Tier-2/Tier-3 sites can block those channels
+overheads
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 16
Evolution of transfer workflows (CMS)
Scale and complexity of CMS data transfers has been steadily increasing thanks to focused effort on improving transfer quality and throughput (200 TBs/day)
T0 →→→→ T1
T1 ↔↔↔↔ T1
T1 ↔↔↔↔ all T2s
T2 ↔↔↔↔ all T2s
* As more and more data transfer links are commissioned, the sites start competing for the same slots in the FTS channels [CMS has ~2300 data transfer links commissioned]
* Making optimal use of bandwidth requires identifying and isolating the problematic links
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 17
FTSMonitor
‣ FTS server database contains detailed transfer information
‣ Wealth of knowledge that can be used to spot issues
‣ Information is exposed through FTS Monitor (CCIN2P3)
-Transfer Sum
mary -
https://forge.in2p3.fr/projects/ftsmonitor/files
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 18
FTSMonitor
‣ FTS server database contains detailed transfer information
‣ Wealth of knowledge that can be used to spot issues
‣ Information is exposed through FTS Monitor (CCIN2P3)
-Channel Config details -
https://forge.in2p3.fr/projects/ftsmonitor/files
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 19
FTSMonitor
‣ FTS server database contains detailed transfer information
‣ Wealth of knowledge that can be used to spot issues
‣ Information is exposed through FTS Monitor (CCIN2P3)
-Ind. Transfer details -
https://forge.in2p3.fr/projects/ftsmonitor/files
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 20
FTSMonitor
- Ind. Transfer details -
- Transfer Summary -
- Channel Config details -
We can trace down transfer problems/issues and details of those failed attemps
We can trace down if channel is properly configured for CMSAdditionally, if timeouts need to be adjusted
The most important informationwe can gather from individual successful file transfersPhEDEx aggregated plots (result of heterogeneus configs) is not enough to evaluate link throughputs
https://forge.in2p3.fr/projects/ftsmonitor/files
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 21
FTS monitor parser
‣ In CMSwe developed a tool to extract data from FTSMonitors worldwide‣ Full statistics about successful transfers are extracted daily, and summary reports
are produced (the tool treat thrughputs and overheads separately)
‣ Several views available
‣ Monthly averages of transfer rate per file and per stream on all channels, for each
point-to-point (we continuously transfer test files in most of the ~2300 links in CMS)
‣ Historical evolution of rate per stream and for each point-to-point transfer link
‣ Comparions of SRM preparation/release times for all of the transfers, at origins
and destination (available since the most recent FTM version)
‣ Identification of congested channels (in preparation)
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 22
e.g. Channel STAR-CIEMAT @ PIC FTS
~x45
Large fluctuationsMaybe due to theDestination site
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 23
e.g. Transfers from T1_IT_CNAF ���� T1_ES_PIC
Of course, point-to-point
throughputs on 1-month period
are not constant
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 24
e.g. Tier-2 exports/imports from/to PIC Tier-1
~x40
~x100
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 25
e.g. Tier-1 ↔ ↔ ↔ ↔ Tier-1 transfer rates
~200!!~x200!!
All these transfers go
Through LHCOPN
All these transfers go
Through LHCOPN
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 26
Improving Channel Configuration
‣ Using data extracted from FTS Monitoring to improve transfer operations
‣ Example: PIC����T2 exports
‣ Massive PIC�T2s transfers in early October ’10 following processing campaign
‣ Clogged by slow links on PIC-STAR FTS channel
‣ Links with low rate-per-stream identified
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 27
Improving Channel Configuration
‣ Using data extracted from FTS Monitoring to improve transfer operations
‣ Example: PIC����T2 exports
‣ Created “cloud” FTS channels for “fast” and “slow” links
‣ Improved FTS channel occupancy
‣ Increased number of transfer attempts
‣ Improved overall export throughput
Acknowledgment to M. Caubet
(PIC FTS admin)
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 28
Identifying infrastructure issues
‣ The wealth of data available allows to spot potential issues in the site or network infrastructure
‣ Example: PIC import/export asymmetry
‣ Rate-per-stream lower for exports than for imports on most links‣ Doesn’t seem to depend on distance‣ Potential site issue?‣ One possible explanation could be a known
limitation of the Solaris kernel used on the disk servers at PIC which penalizes the outgoing transfers when the other endpoint runs a Scientific Linux 5 kernel.
‣ Dedicated testing to reveal more…
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 29
Expanding the scope
‣ Gathering data for most of FTS servers included. Working on the missing atm (2 out of 8)‣ Lot of potential to improve data transfers. Now focused on T0����T1s and T1s ↔↔↔↔T1s
‣ Identify “reference” statistics and publish corresponding plots to monitor regularly ‣ For central shifters & site administrators‣ For data transfer experts – Task Force to improve data transfers in CMS
‣ Spot problems in sites and network and assist site administrators with troubleshooting‣ Include more statistics and FTS servers‣ Distributions by file size‣ Transfer preparation/release times‣ Channel occupancy‣ …
‣ PhEDEx to provide “Latency Analysis” – For T1s ↔T1s, use LHCOPN dashboard info as well
‣ The tools can be easily used by other VOs
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 30
Summary & Outlook
‣ PhEDEx ensures reliable data transfers with FTS
‣ The scale and complexity of CMS transfers has constantly increased over the years
‣ The FTS Monitor offers detailed information on transfers
‣ Extracting and analyzing transfer statistics (CMS work started by CHEP’10) provides useful insight to improve transfer operations
‣ Either re-configuring FTS channels(getting rid of them, see: gLite File Transfer Service in the EMI Data Area, MOLNAR, Zsolt, in this Conf.)
‣ Identifying performance problems @ the network level or significant OverHeads(T1↔↔↔↔T1 LHC transfers differ x200! and this is FTS-independent...)
‣ Discussion with FTS developers about transfer monitoring in future FTS versions
Acknowledgments
Lionel Schwarz from CC-IN2P3 for prompt support in fixing issues and including new features in the FTS Monitor. Local contacts and FTS server administrators at all Tier-1 s for deployment of local FTS Monitors & feedback on operating the service.