Introduction to Unix – CS 21

Introduction to Unix – CS 21

Lecture 12

Lecture Overview A few more bash programming

tricks The here document Trapping signals in bash

cut and tr sed awk

The Here Document Redirects stdin to a specific set of

text located inside the same file <<COMMAND << MARKER DataMARKER

Example of Here

Why Is This Useful? Allows you to keep all relevant

information in one file Example: Database you want to

search Don’t need to clutter up your

directory with unnecessary temporary files

Trapping Signals Catching a signal is also called

trapping a signal You can tell bash programs what to

do when they receive different signals

Analogy: When a postcard arrives, what do I do?

The ‘trap’ Command Usage: trap ‘COMMAND’ Signals Example:

trap ‘cat errorMsg’ 4 6 In order to prevent you from

running a program forever, signal number 9 cannot be trapped

Example Of trap

Two Helper Filters cut

Break individual lines apart tr

Change characters into different characters

The ‘cut’ Command More precise control over

individual lines Will cut out certain words from

each individual line so they can be processed

Usage: cut [FLAGS] FILE

Flags -d

Delimiter -f

Field number Example

cut –d: -f3 myFile

Example Of Cut

The ‘tr’ Command Translate Change on a one to one basis

characters from one thing to another

Usage: tr ‘Set1’ ‘Set2’ Example: tr ‘abc’ ‘ABC’ < myFile

Example Of tr

Two More Powerful Tools sed

Stream Editor awk

Alfred Aho, Peter Weinberger, and Brian Kernighan

The ‘sed’ Command/Language Filter

Like grep, sort, or uniq, it takes input and performs some operation on it to filter the output

Usage: sed ‘Address Command’ Address specifies where to make changes Command specifies what change to make Example:

sed ‘4d’ textFile

Address Specification Addresses could be line numbers

or regular expressions No address – each line One address – only that line Two comma separated addresses – All

lines in between ! – All other lines

Commands Available To sed a\

Append text c\

Replace text i\

Insert text before d

Delete lines s

Make substitutions

Examples

More Examples

Substitution Example Same syntax as vi

When Would You Want To Use sed? sed works on streams, so it is

perfect to be placed in the middle of a pipe in order to change the output from one format to another

Example: If a program always prints out 4 lines

of junk for every good line, sed can be used to weed out the junk

Example

awk Answers the question:

What do I do if I want to search for a pattern and actually use it?

Combination of grep and commands Searches for some pattern or condition and

performs some command on it Complete programming language

Looks a lot like C syntactically Variables are declared bash style

Pattern And Command awk in its most basic form simply

executes a command on all lines that match (or adhere to) a certain pattern

Usage: awk ‘Pattern { Command }’ FILE Just like sed, if there is no pattern,

then every line will be matched

Example

Different Ways To Run Awk awk ‘Pattern { Command }’ awk –f awkFile inputFile

Since awk itself can be a complex language, you can store all the commands in a file and run it with the –f flag

Important awk Concepts Record

Every line of an input file is a record The current record can be referenced with $0

Field Every word in a record is called a field Each field is numbered and can be

referred to $1 is the first record, $2 is the second, etc.

Special Predefined awk Variables RS

The character that acts as a record separator

Default is the end of a line FS

The character that acts as a field separator

Default is whitespace (space, tab, etc) Can be redefined with the –F flag

Example

Other awk Variables NF = number of fields in the

current record NR = Total number of records seen

so far OFS = Output field separator ORS = Output record separator

BEGIN And END Blocks Two special patterns that can be

matched BEGIN

Commands are executed before any records are looked at

END Commands are executed after all

records are processed

Example

Awk Patterns /regular expression/ -> same as egrep Relational expression

>, <, >=, <=, == Pattern && pattern Pattern || pattern Pattern1 ? Pattern2 : pattern3

If Pattern1 is True, then Pattern2, else pattern 3 (pattern) ! Pattern Pattern1, pattern2

Example Patterns

Awk Actions Enclosed in { } () Grouping $ Field reference ++ -- Increment, decrement ^ Exponentiation + - ! Plus, minus, not * / % Multiplication, division, and

modulus

Control Flow Statements Inside of commands, you can have

control flow if while for

If Syntaxif (condition){ Statements}else{ Statements}

While Syntaxwhile (Condition){ Statements}

For Syntaxfor (Declaration ; Condition ; Increment ){ Statements}

for ( j=0; j < 5; j++){ print “hello world”}

When Would You Want To Use awk? Whenever you want to search for

some pattern and perform some action

Example: I want to go through and calculate the average score on the Midterm

Example

Another Example Adding 12 points to everyone’s

midterm score

Putting Them Together – awk and sed

awk Versus bash $ arguments Always enclose everything to awk in

single quotes so they don’t get interpreted

$1 to awk means something completely different than $1 to bash $1 in awk means first field $1 in bash means first command line

argument

Next Time Looking at some of the string and

mathematical awk functionality find Putting everything together

The complete bash programming necessities

Quiz 2 – Next Tuesday