Descriptive Data Analysis of File Transfer Data Sudarshan Srinivasan Victor Hazlewood Gregory D. Peterson
Dec 28, 2015
Descriptive Data Analysis of File
Transfer Data
Sudarshan Srinivasan
Victor Hazlewood
Gregory D. Peterson
2
Objective
· Understanding the GridFTP log transfer data we have at NICS.
· Analyze the data and identify areas of potential improvement.
· Perform predictive analysis to improve efficiency.
· Apply knowledge to XSEDE service providers.
3
NICS GridFTP Infrastructure
4
GridFTP Logging
· Gridftp data transfer protocol version 5.2.2.
· Two types of logging: "usage" logging and "log_transfer" logging (enabled in 5.2.2).
· Prior to 5.2.2 endpoint IP address data was filled with 0.0.0.0.
· Thanks to the Globus folks for fixing this bug!
5
Transfer Logs
· NICS uses a PostgreSQL database for storing transfer log data.
· Two new tables: n_gridftp_usage and n_gridftp_usage_detail.
· n_gridftp_usage: quick lookup of aggregate monthly GridFTP usage information.
· n_gridftp_usage_detail: Detailed records of each data transfer.
· Log data includes: starttime, endtime, nbytes, user, filename, source and destination end points.
Log Data Collection
· Data from each GridFTP server is copied to log files to a central NFS location.
· Each month we run a processing script on the log files that checks for errors in the log entry.
· Following this, we run a script to load the log files into database table.
· We chose transfer log data for the year 2013 for this analysis.
DATE=20130401132041.657463 HOST=datamover1.nics.utk.edu PROG=globus-gridftp-server NL_EVNT=FTP_INFO START=2013041132041.534646 USER=username NBYTES=1048576 VOLUME=/ STREAMS=1 STRIPS=1 DEST=[192.249.6.164] TYPE=RETR CODE=226
7
Log Data Analysis· Two variables were identified: number of transfers
and total amount of data transferred.
· Data transfer rate based on starttime, endtime and nbytes.
· Monthly visual comparison of data coming into and going out of NICS from everywhere.
· Intra XSEDE site number of transfers and data transferred coming into and going out of NICS.
· Bucketing of transfer data based on transfer size (ts).
· R statistical computing language was used to plot all histograms and graphs.
8
Basic Statistics for the year 2013
Type Quantity
Total Transfers 67,160,380
Average transfers per month 5,596,698
File transfers ts > 64 GB 813 (0.001%)
File transfers 1 MB < ts < 64GB 19,374,549 (28.85%)
File transfers ts < 1 MB 47,785,018 (71.15%)
9
Number of transfers and amount transferred for the year 2013
Number of transfers (in millions)Total = 83.54 millions
Total amount transferred (in TB)Total = 1235.7millions
MonthTota
l am
ou
nt tr
ansf
err
ed
(in
TB
)N
um
be
r of
tran
sfe
rs(i
n m
illio
ns) Mean
10
Percentage of transfers vs Transfer size for the year 2013
Total transfers: 67160380
Transfers size (ts)
Pe
rce
nta
ge
of t
ran
sfe
rs
11
Transfer speed for top 500 transfers with transfer size > 1GB
Month
gbp
s
12
Monthly comparison between number of transfers coming into and going out
of NICS for year 2013
Month
Tota
l nu
mb
er o
f tra
nsf
ers
(in
mill
ion
s)
13
Monthly comparison between total amount of data coming into and going
out of NICS for year 2013
Month
Tota
l am
ou
nt o
f dat
a m
ove
d(i
n T
B)
Transfer data buckets for November 2013
14
All transfers for November 2013Total transfers: 2181157
Transfer size (ts)
Pe
rce
nta
ge
of t
ran
sfe
rs
All transfers for November 2013, ts < 1MBTotal transfers: 749747
Pe
rce
nta
ge
of t
ran
sfe
rs
Transfer size (ts)
All transfers for November 2013, 1MB < ts < 64GBTotal transfers: 1431385
Pe
rce
nta
ge
of t
ran
sfe
rs
Transfer size (ts)
All transfers for November 2013, ts > 64GBTotal transfers: 25
Pe
rce
nta
ge
of t
ran
sfe
rs
Transfer size (ts)
15
Intra XSEDE Sites and Abbreviation
Site Name Abbreviation
Texas Advanced Computer Center TACC
Pittsburgh Supercomputing Center PSC
San Diego Supercomputer Center SDSC
National Institute for Computational Sciences/ Georgia Institute of
Technology
NICS/GaTech
Indiana University IU
Open Science Grid OSG
National Center for Atmospheric Research
NCAR
16
Intra XSEDE site data coming into NICSN
um
be
r of
tran
sfe
rs(i
n th
ousa
nd
s)To
tal a
mo
unt
tran
sfe
rre
d(i
n T
B)
Month
TACCPSCSDSCNICS/GaTech
IUOSGNCAR
17
Intra XSEDE site data going out of NICS
Month
Nu
mb
er
of tr
ansf
ers
(in
thou
san
ds)
TACCPSCSDSCNICS/GaTech
IUOSGNCAR
Tota
l am
ou
nt tr
ansf
err
ed
(in
TB
)
18
Intra XSEDE site data coming into and going out of NICS together
TACCPSCSDSCNICS/GaTech
IUOSGNCAR
Nu
mb
er
of tr
ansf
ers
(in
thou
san
ds)
Tota
l am
ou
nt tr
ansf
err
ed
(in
TB
)
Month
19
Future Work· Currently in progress:
– Moving from using PostgreSQL database to loading data completely in memory in a separate machine.
– Using Apache Spark for fast large-scale data processing.– Combining SQL, streaming, and complex analytics.– Using advanced data mining and machine learning
algorithms provided in libraries in Python.
· Next Step:– Analyze by combing job data, filesystem data, and archive
data for analysis.– Visualize data flow within XSEDE network on a
geographical map.
Thank You!
Questions?