Unix: Beyond the Basics
1
Login
• Requesting a tak account
http://iona.wi.mit.edu/bio/software/unix/bioinfoaccount.php
• Windows
PuTTY
Need to setup X-windows for graphical display
• Macs
Access the Terminal: Go Utilities Terminal
2
Login using Secure Shell
ssh –Y user@tak
3
Command Prompt user@tak ~$
PuTTY on Windows
Terminal on Macs
Hot Topics website: http://jura.wi.mit.edu/bio/education/hot_topics/
• Please login to our tak server, and create a directory for the
exercises in your lab share, and use it as your working directory $ mkdir unix2
$ cd unix2
• Download all files into your working directory
• You should have the files below in your working directory: – foo.txt, sample1.txt, exercise.txt , join2filesByFirstColumn.pl , datasets folder
– you can check with ls command
4
Unix Commands Review (Unix Essentials)
$ sort –k2,3 foo.tab
$ cut –f1,5 foo.tab
$ cut –f1-5 foo.tab
$ wc –l foo.txt
How many lines are in this file?
-f: select only these fields -f1,5: select 1st and 5th fields -f1-5: select 1st, 2nd, 3rd, 4th, and 5th fields
5
start end
-n or -g: sorting numbers -n is recommended than –g, except for scientific notation or a leading '+' -r: reverse order
What you will learn…
• sed • awk • groupBy (bedtools) • loops • join files together • scripting
6
sed: stream editor for filtering and transforming text
• Print lines 10 - 15:
• Delete 5 header lines at the beginning of a file:
• Remove all version numbers (ex: '.1') from the end of a list of sequence accessions:
$ sed '1,5d' file > fileNoHeader
$ sed -n '10,15p' bigFile > selectedLines
$ sed 's/\.[0-9]\+//g' accsWithVersion > accsOnly
7
s: substitute g: global modifier d: delete line p: print the current pattern space -n: only print those lines matching our pattern
awk • Name comes from the original authors:
Alfred V. Aho, Peter J. Weinberger, Brian W. Kernighan
• A simple programing language
• Good for short programs, including parsing files
8
awk
9
• Print the 2nd and 1st fields of the file: $ awk ' { print $2"\t"$1 } ' foo.tab • Convert sequences from tab delimited format to fasta format: $ awk ' { print ">"$1"\n"$2 } ' foo.tab > foo.fa $ head -1 foo.tab Seq1 ACTGCATCAC $ head -2 foo.fa >Seq1 ACGCATCAC
#: comment, ignored by awk By default, awk splits each line by spaces
Character Description
\n newline
\r carriage return
\t horizontal tab
awk: field separator
• Issues with default separator: white space
– one field is gene description with multiple words
– consecutive empty cells
• To use tab as the separator:
10
$ awk –F "\t" '{ print NF }' foo.txt or $ awk 'BEGIN {FS="\t"} { print NF }' foo.txt BEGIN: action before read input NF: number of fields in the current record FS: input field separator OFS: output field separator END: action after read input
awk: arithmetic operations
11
Add average values of 4th and 5th fields to the file: $ awk '{ print $0"\t"($4+$5)/2 }' foo.tab $0: all fields
Operator Description
+ Addition
- Subtraction
* Multiplication
/ Division
% Modulo
^ Exponentiation
** Exponentiation
awk: making comparisons
12
Print out records if values in 4th or 5th field are above 4: $ awk '{ if( $4>4 || $5>4 ) print $0 } ' foo.tab
Sequence Description
> Greater than
< Less than
<= Less than or equal to
>= Greater than or equal to
== Equal to
!= Not equal to
~ Matches
!~ Does not match
|| Logical OR
&& Logical AND
awk • Conditional statements:
• Looping:
if (condition1) action1 else if (condition2) action2 else action3
for ( i = 1; i <= NF; i++ ) print $i
13
Display expression levels for the gene NANOG: $ awk '{ if(/NANOG/) print $0 }' foo.txt or $ awk '/NANOG/ { print $0 } ' foo.txt or $ awk '/NANOG/' foo.txt Add line number to the above output: $ awk '/NANOG/ { print NR”\t”$0 }' foo.txt NR: number of the current record
Calculate the average expression (4th, 5th and 6th fields in this case) for each transcript $ awk '{ total= $4 + $5 + $6; avg=total/3; print $0"\t"avg}' foo.txt or $ awk '{ total=0; for (i=4; i<=6; i++) total=total+$i; avg=total/3; print $0"\t”avg }' foo.txt
sub function for substitution: sub( regexp, replacement, target)
Summarize by Columns: groupBy (from bedtools)
-g grpCols column(s) for grouping
-c -opCols column(s) to be summarized
-o Operation(s) applied to opCol: sum, count, min, max, mean, median, stdev, collapse distinct freqdesc (i.e., print desc. list of values:freq) freqasc (i.e., print asc. list of values:freq)
Print the maximum value (3rd column) for each gene (2nd field) in the file $ groupBy -g 2 -c 3 -o max foo.tab
14
Join files together • Unix join
• BaRC scripts: /nfs/BaRC_Public/BaRC_code/Perl/
sorting not required $ join2filesByFirstColumn.pl file1 file2 $ submatrix_selector.pl matrixFile rowIDfile or 0 [columnIDfile]
$ join -1 2 -2 3 FILE1 FILE2 Join files on the 2nd field of FILE1 with the 3rd field of FILE2, only showing the common lines. FILE1 and FILE2 must be sorted on the join fields before running join
15
Shell Flavors
• Syntax (for scripting) depends on which shell echo $SHELL
• Several shells available, bash is the default on tak and commonly used.
• Some of the shells (incomplete listing):
16
Shell Name
sh Bourne
bash Bourne-Again
ksh Korn shell
csh C shell
Shell Script
• Automation: avoid having to retype the same commands many times
• Ease of use and more efficient
• Outline of a script:
#!/bin/bash shebang: interprets how to run the script commands… set of commands used in the script
#comments write comments using “#”
• Commonly used extension for script is .sh (eg. foo.sh), file must have executable permission
17
Bash Shell: for loop
• Process multiple files with one command
• Reduce computational time with many cluster nodes
for myfile in `ls dataDir` do bsub wc –l $myfile done When referring to a variable, $ is needed before the variable name ($myfile), but $ is not needed when defining it (myfile). Note: this syntax is different from awk
18
Shell Script Example #!/bin/sh
# 1. Take two arguments: the first one is a directory with all the datasets, the second one is an output directory
# 2. For each data, calculate average gene expression, and save the results in a file in the output directory
inDir=$1 # 1st argument
outDir=$2 # 2nd argument; outDir must already exist
# Define variables: no spaces on either side of the equal sign
for i in `/bin/ls $inDir ` # refer to variable with $
do
# output file name
name="${i}_gene.txt" # {}: $i_gene not valid; prevent misinterpretation of variable as #characters
# calculate average gene expression
# NM_001039201 Hdhd2 5.0306 5.3309 5.4998
sort -k2,2 ${inDir}/${i} | groupBy -g 2 -c 3,4,5 -o mean,mean,mean >| ${outDir}/${name}
done
19
# You can use graphical editors such as nedit, gedit, xemacs to create shell scripts
Further Reading
• BaRC one liners: – http://iona.wi.mit.edu/bio/bioinfo/scripts/#unix
• Unix Info for Bioinfo: – http://iona.wi.mit.edu/bio/barc_site_map.php
• Online books via MIT: – http://proquest.safaribooksonline.com/
• MIT: Turning Biologists into Bioinformaticists – http://rous.mit.edu/index.php/Teaching
• Bash Guide for Beginners: – http://tldp.org/LDP/Bash-Beginners-Guide/html/
20