Data Transfers in the ALCF

Data Transfers in the ALCFRobert ScottTechnical Support Analyst

Argonne Leadership Computing Facility

Transferring Data To/From ALCF

sftp and scp (for “small” transfers)– If you must use scp, eureka, and in the future tukey are better systems

to scp to/from. All paths will be the same as they are on our other systems.

– Eureka is also a better host for compressing and uncompressing large file archives

GridFTP (for large transfers)– Other site must accept our CA– ssh / cryptocard access available– For more information on Grid FTP please go here:

http://www.alcf.anl.gov/resource-guides/using-gridftp-intrepid Globus Online (for large transfers)

Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy

2

Getting Started with Globus Online

You’ll need a Globus Online account– Go here to signup: https://www.globusonline.org/– Globus Connect: to transfer files to and from your local

machine (laptop)– SSH key or crypto card to access an ALCF endpoint– Go here to initiate your transfer:

https://www.globusonline.org/xfer/StartTransfer

Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy

3

What is Globus Online?

Reliable File Transfer– Easy: Simple signup, login and use – and minimized intervention– Fast: GO can move large filesets in hours– Secure: No Need to worry about security configs or certs, and one-

time passwords work– Research-focused: Globus Online is the only file transfer service with

the scientific researcher in mind No IT required

– No client software installation– Works with existing GridFTP servers– Globus Connect solves “last mile problem” for local machines– Consolidated support and troubleshooting

Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy

4

Transferring Data To/From ALCF using GO Globus Online (for large transfers)

– Globus Online addresses the challenges faced by researchers in moving, sharing, and archiving large volumes of data among distributed sites.

– ALCF BG/P endpoints: alcf#dtn_intrepid, alcf#dtn_surveyor, alcf#dtn

– Check if your laboratory, university or research center has already an endpoint.

– Globus Connect to transfer files to and from your local machine.– http://www.alcf.anl.gov/resource-guides/data-transfer

Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy

5

Globus Online Demo

Let’s start the Demo….

Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy

6

Table of BG/Q File Systems on ALCF Resources

Argonne Leadership Computing Facility – www.alcf.anl.gov

Name System Type Path Production Backed-up Users

vesta-home Vesta GPFS /home Yes No General Use

vesta-scratch(Intensive job output, large files)

Vesta GPFS /projects Yes No I/O

mira-home(general use)

Mira GPFS /home Yes No —

mira-fs0(Intensive job output, large files)

Mira GPFS /projects Yes No I/O

NOTE: all “filesystems” are also available via the /gpfs/$filesystem_name path on these systems.

Thank You!

Questions?

Argonne Leadership Computing Facility - supported by the Office of Science of the U.S. Department of Energy

8