Introduction to the Linux Command Line for High-Performance Computing Dr. Charles J Antonelli LSAIT ARS June, 2014.

Introduction to theLinux Command Line for

High-Performance ComputingDr. Charles J Antonelli

LSAIT ARSJune, 2014

cja 2014 2

RoadmapLinux overview

The command shell

Navigating the filesystem

Basic commands & wildcarding

Shell redirection & pipelining

Editing text files

Permissions

Processes

06/14

cja 2014 3

PollHow many here to learn the command line

For HPC, specifically Flux, work?

For other computing cluster work?

For desktops/other Linux/Unix/Mac OS X?

Other?

06/14

cja 2014 4

Course TextWilliam E Shotts, Jr.,“The Linux Command Line: A Complete Introduction,”No Starch Press, January 2012.

Download Creative Commons Licensed version athttp://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdf.

06/14

cja 2014 5

Linux Overview

06/14

cja 2014 6

A compute node

12 Intel cores

48 GB RAM

Local disk

Network

06/14

cja 2014 7 06/14http://linuxforums.org.uk/index.php?action=media;sa=item;in=262

cja 2014 8

The command shell

06/14

cja 2014 9

The command lineA basic way of interacting with a Linux system

Execute commands

Create files and directories

Edit file content

Access the web

Copy files to and from other hosts

Run HPC jobs

… do things you can’t do from the conventional point-and-click Graphical User Interface (GUI)

06/14

cja 2014 10

Why command line?1. Linux was designed for the command line

2. You can create new Linux commands using the command line, without programming

3. Many systems provide only the command line, or poorly support a GUI interface• Such as most HPC systems

4. Many things can be accomplished only through the command line• Much systems administration & troubleshooting

5. You want to be cool

06/14

cja 2014 11

The command shellThe command shell is an application that reads command lines from the keyboard and passes them to the Linux operating system to be executed

When you login to a remote Linux system using a tool like ssh, you will automatically be connected to a shell

On your desktop, laptop, or tablet, you may have to find and execute a terminal emulator application to bring up a shell in a window

06/14

cja 2014 12

The command shell

06/14http://www.livefirelabs.com/208_global/images/unix-operating-system.gif

cja 2014 13

Types of shells

Standard Linux shell (used in this course)

bash “Bourne-again” shell

Other shellstcsh “Enhanced C shell”, better version of cshcsh “C shell”, common on older Unix systemssh Bourne shell, original AT&T Unix shell

What shell am I using?echo $SHELL

06/14

cja 2014 14

Connecting via ssh

Terminal emulatorsLinux and Mac OS X

Start Terminal

Use ssh command

Windows

SSH Secure Shell (deprecated)

U-M PuTTY/WinSCP (U-M Blue Disc)https://www.itcs.umich.edu/bluedisc/

PuTTYhttp://www.chiark.greenend.org.uk/~sgtatham/putty/

06/14

cja 2014 15

Lab 0Task: Start a local shell on Mac OS X

1. Login to Mac OS X with your uniqname and Kerberos password

2. If there is a Terminal icon in the Dock , double-click itOtherwise:a. Bring up a Finder window:

Click on the desktop, type Command-N

b. Start the Terminal Application:In the Finder window, click on Applications on the left, scroll down on the right until you find the Utilities folder, double-click on the Utilities folder, scroll down on the right until you find the Terminal application, double-click it

3. This creates a Terminal window with a shell running inside it

4. From a Terminal window, Command-N will start a new Terminal

06/14

cja 2014 16

Lab 0

06/14

cja 2014 17

The shell promptThe “~$ “ is the shell prompt

This means the shell is waiting for you to type something

Format can vary, usually ends with “$” , “%” or “#”

If $ or %, you have a normal shellThis shell has your privileges

If #, you have a so-called “root shell”This shell has administrator privileges

You can do a great deal of irreversible damage

06/14

cja 2014 18

Typing into the shellBasic input line editing commands

Backspace erases previous character

Left and right arrow move insertion point on the line

Control-U erases the line so you can start over

Enter executes the line you typed

Control-C interrupts whatever command you started and returns you to the shell prompt (usually)

Up and down arrow will access your command history

Type “exit” without the quotes to exit the shell

Click the red circle at upper left of the Terminal window to close it

06/14

cja 2014 19

Lab 1Task: Enter some basic commands

~$ date

~$ id

~$ ps

~$ df -kh

~$ who

~$ top

06/14

cja 2014 20

Interlude: Flux

06/14

cja 2014 21

FluxFlux is a university-wide shared computational discovery / high-performance computing service.

Interdisciplinary Provided by Advanced Research Computing at U-M (ARC)

Operated by CAEN HPC

Hardware procurement, software licensing, billing support by U-M ITS

Used across campus

Collaborative since 2010Advanced Research Computing at U-M (ARC)

College of Engineering’s IT Group (CAEN)

Information and Technology Services

Medical School

College of Literature, Science, and the Arts

School of Information

06/14

http://arc.research.umich.edu/resources-services/flux/

cja 2014 22

The Flux clusterLogin nodes Compute nodes

Storage…

Data transfernode

06/14

cja 2014 23

A Flux node

12 Intel cores

48 GB RAM

Local disk

Network

06/14

cja 2014 24

Logging in to Fluxssh flux-login.engin.umich.edu

MToken (or Software Token) required

You will be randomly connected a Flux login nodeCurrently flux-login1 or flux-login2

Firewalls restrict access to flux-login.To connect successfully, either

Physically connect your ssh client platform to the U-M campus wired or MWireless network, or

Use VPN software on your client platform, or

Use ssh to login to an ITS login node (login.itd.umich.edu), and ssh to flux-login from there

06/14

cja 2014 25

A shell on a login node

Now you have a shell on the login node

Your shell prompt has changed to something like[flux-login1 ~]$

Try those simple commands here!

Flux is for HPC workThe login nodes are reserved for copying data, editing and compiling programs and scripts, and short test runs

For HPC computation, you should run on the compute nodesA batch job will always run on the compute nodes

An interactive batch job will get you a shell on a compute node

06/14

cja 2014 26

A shell on a compute node

Getting a shell on a compute nodeEnter this command on the login node:[flux-login1 ~]$ ~cja/hpc100/getshell

After a short delay, you’ll get an interactive shell:qsub: waiting for job 11147004.nyx.engin.umich.edu to startqsub: job 11147004.nyx.engin.umich.edu ready[nyx5509 ~]$

You can type commands into this shell for all of our in-class experiments

Type “exit” to exit this shell

After two hours, your interactive shell will be killed

06/14

cja 2014 27

Navigating the filesystem

06/14

cja 2014 28

Linux Filesystem Concepts

Files are stored in a directory

Directories may contain other directories as well as files

A hierarchy of these directories is called a directory tree

A directory tree (a connected graph with no cycles) has a single topmost root directory

A directory tree rooted at the system root directory “/” is called a filesystem

06/14

cja 2014 29 06/14http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html

cja 2014 30

Linux Filesystem Concepts

A file is accessed using its path name

Absolute path name/dir1/dir2/…/dirn/filename

/usr/X11R6/bin

Relative path namecurrent-working-directory/filename

bin

Every shell maintains a notion of a current working directoryInitialized at login to your home directory

Changed via cd command

Two special directories. refers to the current directory

.. refers to the current directory’s parent directory

06/14

cja 2014 31

Basic commands

06/14

cja 2014 32

PrerequisitesSome fundamental commands:~$ file file # what kind of file is file?

~$ cat file # display contents of text file

~$ less file # paginate text file

~$ man command # get info about command

Exercise: figure out how to make the date command display the date in Coordinated Universal Time (UTC)

06/14

cja 2014 33

Lab 2Task: navigate the file system

Commands:

~$ cd # make your home directory the current working directory

~$ pwd # print working directory

~$ mkdir foo # create directory foo

~$ cd foo # cd to the foo directory

~$ mkdir bar # create directory bar

~$ cd .. # cd back to the foo directory

~$ tree # display foo’s directory tree

06/14

cja 2014 34

Navigating the filesystem

Some fundamental commands:~$ pwd # print working directory

~$ cd dir # make dir the current working directory

~$ cd # cd to your home dir

~$ cd ~cja # cd to cja’s home dir

~$ mkdir dir # create directory dir

~$ rmdir dir # remove (empty) directory dir

~$ tree # display dir tree

06/14

cja 2014 35

Listing info on filesls – list information about files~$ ls # list contents of cur dir

~$ ls dir # list contents of dir

~$ ls –l # list details of files in cur dir

~$ ls –t # list newest files first

~$ ls –R dir # list all files in tree dir

~$ ls –lt dir # options can be combined

06/14

cja 2014 36

Working with filesThese commands manipulate files~$ mv foo bar # rename file foo to bar

~$ cp foo bar # copy file foo to bar

~$ cp –r dir1 dir2 # copy dir tree dir1 to dir2

~$ cp f1 f2 dir # copy f1 and f2 to directory dir

~$ mkdir dir # create empty directory dir

~$ rmdir dir # remove empty directory dir

~$ rm file # remove file file

~$ rm –r dir # remove directory tree dir

06/14

cja 2014 37

Lab 3Exercise:

Create a directory named tutorial in your home directory. In that directory, create a directory named sample and a directory named test . Create a file named msg in directory test that contains a copy of the file /etc/motd .

06/14

cja 2014 38

Compressing and archiving

These commands compress and archive files~$ gzip foo # compress foo to foo.gz

~$ gunzip foo # uncompress foo.gz to foo

~$ tar cf foo.tar bar# archive subtree bar in file foo.tar

~$ tar xf foo.tar # restore archive from file foo.tar

~$ tar zcf foo.tgz bar # archive and compress

~$ tar jcf foo.tjz bar # archive and compress better

06/14

cja 2014 39

WildcardsAll Linux commands take wildcarded arguments

Wildcards:? Matches a single character* Matches zero or more characters[chars] Matches any of the chars[c1-c2] Matches chars ‘c1’ through ‘c2’[^chars] Matches any but the chars

~$ ls foo.? # match files named foo.x, where x is any character

~$ echo *.[cs] # echo files that end in .c or .s

~$ mv [o-z]* save # move files starting with o through z to directory save

~$ echo [^A-Z]? # ???

06/14

cja 2014 40

Shell redirection & pipelining

06/14

cja 2014 41

Shell redirectionA Linux command can have its inputs and outputs redirected~$ ls >foo # put list of files in current

directory into file foo

~$ ls >>bar # add list of files in current directory to end of file foo

~$ sort <bar # sort lines from file bar

~$ sort <<EOF # sort lines entered at keyboardwhiskeybravotangoEOF

~$ wc -l </etc/motd >~/mycounts# count number of lines from file /etc/motd and put result in file mycounts in my home directory

06/14

cja 2014 42

More Linux commands

More useful Linux tool commands~$ grep string # show lines of input containing

string

~$ tail # show last few lines of input

~$ head # show first few lines of input

~$ sort # sort the input

06/14

cja 2014 43

Shell pipeliningA Linux command can have its output connected to the input of another Linux command~$ ls | wc –l # count files in current

directory

~$ last | grep reboot # when did we reboot?

Exercise:

How many people are running MATLAB on your compute node?

06/14

cja 2014 44

Editing text files

06/14

cja 2014 45

Editing text files

Simple editornano or pico

"What you see is what you get” editor

Simple to learn if you want to get started quickly

Supported editorsvi or vim

emacsPowerful but more complex

If you have time and inclination to become proficient, spend time here

06/14

cja 2014 46

Text files

Watch out for source code or data files written on Windows systems

Use these tools to analyze and convert source files to Linux formatfile

dos2unix

06/14

cja 2014 47

Permissions

06/14

cja 2014 48

File Permissions

Three permission bits, aka mode bitsFiles: Read, Write, Execute

Directories: List, Modify, Search

Three user classesUser (File Owner), File Group, Other

man chmod

06/14

cja 2014 49

File Permissions, examples

-rwxr-xr-x cja lsaitfile read, write, and execute rights for the owner, read and execute for others

-rwxr-x--x cja lsaitsame permissions as above, except only the owner can read the file, while all can execute

chmod u=rwx,g=rx,o=x file

drwxr-x--x cja lsaitlist, modify, and search for the owner, list and search for group, and execute only for others

06/14

cja 2014 50

Processes

06/14

cja 2014 51

Processes

On Linux, every program runs in a process

You can examine these processes

man ps

ps

ps ax

top

06/14

cja 2014 52

Processes

You can signal a running processTo stop it, or "kill" it

man kill

06/14

cja 2014 53

Any Questions?Charles J. AntonelliLSAIT Advocacy and Research [email protected]://www.umich.edu/~cja734 763 0607

Please evaluate our course:http://umichlsa.qualtrics.com/SE/?SID=SV_3pzjrMdjGXtUDAh

06/14

cja 2014 54

References1. http://en.wikipedia.org/wiki/History_of_Linux2. http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html3. William E Shotts, Jr., “The Linux Command Line: A Complete Introduction,” No

Starch Press, January 2012. Download Creative Commons Licensed version athttp://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdf.

06/14

cja 2014 55

ExtraTask: compile and execute simple programs on the Flux compute node

Copy sample code to your login directory:cdcp ~brockp/cac-intro-code.tar.gz .tar -xvzf cac-intro-code.tar.gzcd ./cac-intro-code

Examine, compile & execute helloworld.f90:ifort -O3 -ipo -no-prec-div -xHost -o f90hello helloworld.f90./f90hello

Examine, compile & execute helloworld.c:icc -O3 -ipo -no-prec-div -xHost -o chello helloworld.c./chello

Examine, compile & execute MPI parallel code:mpicc -O3 -ipo -no-prec-div -xHost -o c_ex01 c_ex01.cmpirun -np 2 ./c_ex01

06/14