Xuan Guo Chapter 3: Utilities for Power Users Graham Glass and King Ables, UNIX for Programmers and Users, Third Edition, Pearson Prentice Hall, 2003.

Xuan Guo CSc 3320 1

Chapter 3: Utilities for Power Users

Graham Glass and King Ables, UNIX for Programmers and Users,

Third Edition, Pearson Prentice Hall, 2003.

Xuan Guo CSc 3320 2

Regular Expression

Suppose we have a 10,000 lines text file, and we want to search words from the text file.

Query 1: Words in forms of “aa _ _ cc”Query 2: Words “atlanta” or “Atlanta” Query 3: Words consisting of more than three “yyy”

Xuan Guo CSc 3320 3

Regular Expression

Query1

Query 2

Query 3

Regular Expression

Engine

Regular Expression

Regular Expression

Regular Expression

Application

Xuan Guo CSc 3320 4

Regular Expression

1. Vi2. Sed, Awk, Grep3. Java, C#

http://www.zytrax.com/tech/web/regex.htm

Xuan Guo CSc 3320 5

Regular Expression

• Query 1: Words in forms of “aa _ _ cc”

• aa..cc

• Query 2: Words “atlanta” or “Atlanta” • [aA]tlanta (atlanta|Atlanta)

• Query 3: Words consisting of more than three “yyy”

• (y){3,}

Xuan Guo CSc 3320 6

More Example

[ab][a-z][A-Z][0-9]\d[^0-9][a-z 0-9](ae|bd)

a?a+a*(ab){3,5}(ab){3,}(ab){3}

Xuan Guo CSc 3320 7

Other Issues

1. Anchors ^, $2. Metacharacters[, ], {, }, \, ^, $, ?, *, +, ., (, )

Xuan Guo CSc 3320 8

Exercise

1 Which of the following matches regexp a(ab)*a 1) abababa

2) aaba

3) aabbaa

4) aba

5) aabababa

Xuan Guo CSc 3320 9

Exercise

2 Which of the following matches regexp ab+c?

1) abc2) ac3) abbb4) bbc

Xuan Guo CSc 3320 10

Exercise

3 Which of the following matches regexp a.[bc]+ 1) abc2) abbbbbbbb3) azc4) abcbcbcbc5) ac6) asccbbbbcbcccc


Exercise

4 Which of the following matches regexp (abc|xyz) 1) abc2) xyz3) abc|xyz


Exercise

5 Which of the following matches regexp [a-z]+[\.\?!] 1) battle!2) Hot3) green4) swamping.5) jump up.6) undulate?7) is.?


Exercise

6 Which of the following matches regexp [a-zA-Z]*[^,]= 1) Butt=2) BotHEr,=3) Ample4) FIdDlE7h=5) Brittle =6) Other.=


Exercise

7 Which of the following matches regexp [a-z][\.\?!]\s+[A-Z]

(\s matches any space character)1) A. B2) c! d3) e f4) g. H5) i? J6) k L


Exercise

8 Which of the following matches regexp

(very )+(fat )?(tall|ugly) man 1) very fat man2) fat tall man3) very very fat ugly man4) very very very tall man


Exercise

9 Which of the following matches regexp

<[^>]+> 1) <an xml tag>2) <opentag> <closetag>3) </closetag>4) <>5) <with attribute=”77”>


Answer

(1) 2, 5 (2) 1 (3) 1, 2, 3, 4, 6 (4) 1, 2 (5) 1, 4, 6 (6) 1, 5, 6 (7) 4, 5 (8) 3, 4 (9) 1, 3, 5


Basic Regular Expression & Extended Regular Expression

Meta-characters in Basic Regular Expression^ $ . *  [ ] \{ \} \

vi, grep, sed accept basic regular expression.

Meta-characters in Extended Regular Expression

| ^ $ . * + ? ( ) [ ] { } \

egrep, grep –E, sed –E accept extended regular expression


Grep(Global or Get Regular Expression and Print)

• Filtering patterns: egrep, fgrep, grep– grep -hilnvw pattern {fileName}*– displays lines from files that match the pattern– pattern : regular expression

• -h : do not list file names if many files are specified• -i : ignore case• -l : displays list of files containing pattern• -n : display line numbers• -v : displays lines that do not match the pattern• -w : matches only whole words only


Grep variations

– fgrep : pattern must be fixed string– egrep : pattern can be extended regular expression

• -x option in fgrep: displays only lines that are exactly equal to string

– extended regular expressions:• + matches one or more of the single preceding

character• ? matches zero or one of the single preceding

character• | either or (ex. a* | b*)• () *, +, ? operate on entire subexpression not just on

preceding character; ex. (ab | ba)*


Differences

• grep Search a Pattern from current directory.

egrep (grep -E in linux) is extended grep where additional regular expression metacharacters have been added like +, ?, | and ().

fgrep (grep -F in linux) is fixed or fast grep and behaves as grep but does not recognize any regular expression metacharacters as being special.


48 Dec 3BC1997 LPSX 68.00 LVX2A 138 //line 1483 Sept 5AP1996 USP 65.00 LVX2C 189 //line 247 Oct 3ZL1998 LPSX 43.00 KVM9D 512 //line 3219 dec 2CC1999 CAD 23.00 PLV2C 68 //line 4484 nov 7PL1996 CAD 49.00 PLV2C 234 //line 5487 may 5PA1998 USP 37.00 KVM9D 644 //line 6471 May 7Zh1999 UDP 37.00 KV30D 643 // line 7

grep ”38$" exam1.dat

grep "^[^48]" exam1.dat

grep "[Mm]ay" exam1.dat

grep "K...D" exam1.dat

grep "[A-Z][A-Z][A-Z][9]D" exam1.dat

grep "9\{2,3\}" exam1.dat


Examples

grep “38$" exam1.datgrep "^[^48]" exam1.datgrep "[Mm]ay" exam1.datgrep "K...D" exam1.datgrep "[A-Z][A-Z][A-Z][9]D" exam1.datgrep "9\{2,3\}" exam1.dat


CSV file

• A CSV file consists of any number of record, separated by line breaks of some kind;

• each record consists of fields, separated by some other character or string, most commonly a literal comma or tab.


CSV files

Invent.dat

1. Pen 5 20.002. Pencil 10 2.003. Rubber 3 3.504. Cock 2 45.50


Pattern Scanning and Processing

• awk: utility that scans one or more files and performs an action on all lines that match a particular condition

• The conditions and actions are specified in an awk program.

• awk reads a line– breaks it into fields separated by tabs/spaces – or other separators specified by -F option


awk Command

• awk program has one or more commands:• awk [condition] [ \{ action \} ]• where condition is one of the following:

– special tokens BEGIN or END– an expression involving logical operators, relational

operators, and/or regular expressions


awk Command

• awk [condition] [ \{ action \} ]• action is one of the following kinds of C-like

statements– if-else; while; for; break; continue– assignment statement: var=expression– print; printf;– next (skip remaining patterns on current line)– exit (skips the rest of the current line)– list of statements


awk Command

• accessing individual fields: – $1, ..., $n refer to fields 1 thru n– $0 refers to entire line

• built-in variable NF means number of fields• % awk -F: '{ print NF, $1 }' /etc/passwd• prints the number of fields and the first field in

the /etc/passwd file• -F: means to use : as the field separator


awk Command

• BEGIN condition triggered before first line read • END condition triggered after last line read• FILENAME: built-in variable for name of file

being processed• We will use this data in following examples:


awk Example

Serial NO Product Quantity Unit Price

1 Pen 5 20.00

2 Rubber 10 2.00

3 Pencil 3 3.50

4 Cock 2 45.50

$1 $2 $3 $4

“invent.dat”


awk Example

1.Print the name of each product

awk ‘{print $2}’ invent.dat

Pen

Pencil

Rubber

Cock


awk Example

2.Print the name of each product and its unit price

awk ‘{print $2”>>”$4}’ invent.dat

Pen>>20.00

Pencil>>2.00

Rubber>>3.50

Cock>>45.50


awk Example

3.Print each line

awk ‘{print $0}’ invent.dat

1. Pen 5 20.00

2. Pencil 10 2.00

3. Rubber 3 3.50

4. Cock 2 45.50


awk Example

4. Print the name and unit price of the products whose quantity are greater than 5

awk ‘ $3>=5 {print $2 “>>” $4}’ invent.dat

Pen>>20.00

Pencil>>2.00


awk Example

5. Print the name and unit price of the products which contain the word “Pen”

awk ‘ /Pen/ {print $2 “>>” $4}’ invent.dat

Pen>>20.00

Pencil>>2.00


awk predefined variables

Variable Example

FILENAME name of file being processed

Invent.dat

RS New line

FS whitespace

NF number of fields 4

NR current line #


awk Example

awk '{print FILENAME;print NR}' invent.dat

invent.dat

1

invent.dat

2

invent.dat

3

invent.dat

4


awk Example

Compute the overall value of these product

1. Pen 5 20.002. Pencil 10 2.003. Rubber 3 3.504. Cock 2 45.50


awk Example

BEGIN {

print "---------------------------"

print "BEGIN section is only printed once.“

print "==========================="

}


awk Example

{

total = $3 * $4

recno = $1

item = $2

gtotal += total

printf "%d %s Rs.%f\n", recno, item, total

}


awk Example

END {

print "---------------------------"

printf "Total Rs. %f\n" ,gtotal

print "END section is only printed once."

print "==========================="

}


awk Example

example2

awk –f example2 invent.data


---------------------------

BEGIN section is only printed once.

===========================

1 Pen Rs.100.000000

2 Pencil Rs.20.000000

3 Rubber Rs.10.500000

4 Cock Rs.91.000000

---------------------------

Total Rs. 221.500000

END section is only printed once.

===========================


awk actions

Built-in functions: exp(), log(), sqrt(), substr() etc.

If condition, for loop, while loop


awk another example

% cat /etc/passwdnobody:*:-2:-2:Unprivileged User:/:/usr/bin/falseroot:*:0:0:System Administrator:/var/root:/bin/sh...lp:*:26:26:Printing Services:/var/spool/cups:/usr/bin/false


awk Example

% cat p2.awk BEGIN { print "Start of file: "} { print $1 " " $6 " " $7 } END { print "End of file", FILENAME }

% awk -F: -f p2.awk /etc/passwdStart of file: nobody / /usr/bin/falseroot /var/root /bin/sh...lp /var/spool/cups /usr/bin/falseEnd of file /etc/passwd


awk Operators

• built-in variable NR contains current line #• remember, “-F:” uses colon as separator

% cat p3.awk NR > 1 && NR < 4 { print NR, $1, $6, NF }

% awk -F: -f p3.awk /etc/passwd 2 root /var/root /bin/sh 73 daemon /var/root /usr/bin/false 7


awk Variables

% cat p4.awk BEGIN {print "Scanning file"}{ printf "line %d: %s\n", NR, $0 lineCount++; wordCount += NF; }END { printf "lines = %d, words = %d\n", lineCount, wordCount }

% awk -f p4.awk /etc/passwdScanning fileline 1: nobody:*:-2:-2:Unprivileged User:/:/usr/bin/falseline 2: root:*:0:0:System Administrator:/var/root:/bin/sh...line 37: lp:*:26:26:Printing Services:/var/spool/cups:/usr/bin/falselines = 37, words = 141


awk Control Structures

% cat p5.awk

{ for (i = NF; i >= 1; i--) printf "%s ", $i; printf "\n";}

% awk -f p5.awk /etc/passwdUser:/:/usr/bin/false nobody:*:-2:-2:Unprivileged Administrator:/var/root:/bin/sh root:*:0:0:System ... Services:/var/spool/cups:/usr/bin/false lp:*:26:26:Printing


awk Condition Ranges

• Condition ranges: – two expressions separated by comma

• awk performs action on every line – from the first line that matches first expression – until line that matches second condition

% awk -F: ' /nobody/,/root/ {print $0}' /etc/passwdnobody:*:-2:-2:Unprivileged User:/:/usr/bin/falseroot:*:0:0:System Administrator:/var/root:/bin/sh


awk Built-in Functions

• Built-in functions: – exp()– log()– sqrt()– substr() etc.

% awk -F: '{print substr($1,1,2)}' /etc/passwdnoro...lp


Stream Editor (sed)

• sed – scans one or more text files– performs an edit on all lines that match a condition– actions and conditions may be stored in a file– may be specified at command line in single quotes– commands begin with an address or an

addressRange or a Regular expression– does not modify the input file– writes modified file to standard output


Sed syntax

• sed -option 'general expression' [data-file]Replace words action:s/old pattern/new pattern/

Delete lines action:/pattern/d


Sed syntax

• sed -option 'general expression' [data-file]Search action:-n /pattern/p


Paris PS1 Charles Chin 01/20/86 30Ind PS2 Susan Green 04/05/86 32SUST PS2 Lewis SUST 08/11/85 23JUST IS1 Xiao Ming 11/30/84 9HEBUT IS1 John Main 12/03/84 8SUST PS2 Da Ming 06/01/86 35Paris IS3 Peter Webor 07/05/82 32Paris PS2 Ann Sreph 09/28/85 10Paris IS3 Margot Strong 02/29/82 9


Examples

Search lines that starts with HEBUTsed -n ’/^HEBUT/p' studentssed ’/^HEBUT/p' students // NOT GOOD

HEBUT IS1 John Main 12/03/84 8


Examples

Replace string “SUST” with “SDUST”

sed 's/SUST/SDUST/' students


Paris PS1 Charles Chin 01/20/86 30Ind PS2 Susan Green 04/05/86 32SDUST PS2 Lewis SUST 08/11/85 23JUST IS1 Xiao Ming 11/30/84 9HEBUT IS1 John Main 12/03/84 8SDUST PS2 Da Ming 06/01/86 35Paris IS3 Peter Webor 07/05/82 32Paris PS2 Ann Sreph 09/28/85 10Paris IS3 Margot Strong 02/29/82 9


Examples

Replace string “SUST” with “SDUST”

sed 's/SUST/SDUST/g' students


Examples

Delete lines that contain “../../86”

sed ‘/..\/..\/86/d’ students

• % sed 's/^/ /' file > file.new– indents each line in the file by 2 spaces

• % sed 's/^ *//' file > file.new– removes all leading spaces from each line of the file

• % sed '/a/d' file > file.new– deletes all lines containing 'a'


Ranges by patterns

You can specify two regular expressions as the range.

Assuming a "#" starts a comment, you can search for a keyword, remove all comments until you see the second keyword.

In this case the two keywords are "start" and "stop:" sed '/start/,/stop/ s/#.*//'

The first pattern turns on a flag that tells sed to perform the substitute command on every line. The second pattern turns off the flag.

If the "start" and "stop" pattern occurs twice, the substitution is done both times.

If the "stop" pattern is missing, the flag is never turned off, and the substitution will be performed on every line until the end of the file.


Question

Does sed utility change students?How can we save the output?

Xuan Guo Chapter 3: Utilities for Power Users Graham Glass and King Ables, UNIX for Programmers and Users, Third Edition, Pearson Prentice Hall, 2003.

Documents