Unix Comp-145 L ECTURE 7: UNIX U TILITIES AND F ILTERS B ASED ON : S. D AS, “Y OUR U NIX : T HE ULTIMATE G UIDE ”, 2 ND E DITION, M C G RAW H ILL, 2006.

Unix Comp-145

LECTURE 7: UNIX UTILITIES AND FILTERS

BASED ON: S. DAS, “YOUR UNIX: THE ULTIMATE GUIDE”, 2ND EDITION, MCGRAW HILL, 2006

BROOKDALE COMMUNITY COLLEGE 111/4/2009 rwj

BROOKDALE COMMUNITY COLLEGE 2

UNIX OPERATING SYSTEM

• PIPE FACILITY TO CREATE NEW FUNCTIONS• ACCESS & MANIPULATE FILES• EDITING AND DISPLAY CMDS: CUT/PASTE,

HEAD, TAIL• ADDITIONAL TOOLS

11/4/2009 rwj


Pipe Facility to Create New Functions

• PIPE FACILITY COVERTS THE OUTPUT OF ONE FUNCTION OR COMMAND INTO INPUT FOR ANOTHER FUNCTION

• EXAMPLECommand1 [args] | command2 [args] | command3

11/4/2009 rwj

COMMANDS THAT ACCESS & MANIPULATE FILES

$ find$ pg$ more$ less$ wc$ diff$ sort$ uniq


$ head$ tail$ cut$ paste$ tr$ grep

FILE LEVEL CONTENT LEVEL


• RECURSIVELY EXAMINE A DIRECTORY TREE TO IDENTIFY FILES MATCHING SOME CRITERIA AND THEN INVOKE SOME ACTION.

• SYNTAXfind [pathlist] [selectionCriteria] [action]

• EXAMPLE$ find / -name a.out -print

$ find . -name “*.c” –print OR find . -name ‘*.c’ –print./q6.c./testit/q7.c$

• USEFUL OPTIONS:-name xx xx=name of file looking for.-print Prints selected file on standard output.-inum # Select files matching # where # is a file’s Inode number.

find COMMAND


pg COMMAND

• A TERMINAL PAGING PROGRAM ON UNIX AND UNIX-LIKE SYSTEMS FOR PERUSING THROUGH TEXT FILES

• CAN BE USED TO PAGE THROUGH THE OUTPUT OF A COMMAND VIA A PIPE

• ITS INTERFACE IS SIMILAR TO vi, BUT COMMANDS ARE DIFFERENT

• more, & less ARE MORE COMMONLY USED PAGING PROGRAMS

$ pg fileNameNOTES:1. pg is NOT available on this UNIX platform in this lab, instead use more fileName


more COMMAND


• USED TO DISPLAY (NOT MODIFY) THE CONTENT OF A FILE 1 SCREEN AT A TIME

• VERY BASIC PAGER, ORIGINALLY ALLOWED ONLY FORWARD NAVIGATION THROUGH A FILE, SOME NEWER IMPLEMENTATIONS SUPPORT LIMITED BACKWARD MOVEMENT$ more [options] [fileName]command | more [/c] [/p] [/s] [/tn] [+n]

more [[/c] [/p] [/s] [/tn] [+n]] < [Path]fileName

more [/c] [/p] [/s] [/tn] [+n] [Path]fileNames11/4/2009 rwj


more COMMAND

• Useful Options-num THIS OPTION SPECIFIES AN INTEGER WHICH IS THE SCREEN SIZE (IN LINES).

-d PROMPTS THE USER WITH THE MESSAGE "[PRESS SPACE TO CONTINUE, 'Q' TO QUIT.]" AND WILL DISPLAY "[PRESS 'H' FOR INSTRUCTIONS.]" INSTEAD OF RINGING THE BELL WHEN AN ILLEGAL KEY IS PRESSED.

-l PREVENT A PAUSE AFTER ANY LINE THAT CONTAINS A FORM FEED, USUALLY TREATS ^L (FORM FEED) AS A SPECIAL CHARACTER.

-f CAUSES MORE TO COUNT LOGICAL, RATHER THAN SCREEN LINES (I.E., LONG LINES ARE NOT FOLDED).

-p DO NOT SCROLL. INSTEAD, CLEAR THE WHOLE SCREEN AND THEN DISPLAY THE TEXT.


more COMMAND

• Useful Options (cont’d)

-c DO NOT SCROLL. INSTEAD, PAINT EACH SCREEN FROM THE TOP, CLEARING THE REMAINDER OF EACH LINE AS IT IS DISPLAYED.

-s SQUEEZE MULTIPLE BLANK LINES INTO ONE.

-u SUPPRESS UNDERLINING.

+/ THE +/ OPTION SPECIFIES A STRING THAT WILL BE SEARCHED FOR BEFORE EACH FILE IS DISPLAYED. (EX.: more +/PREAMBLE AMENDMENT1.TXT)

+num START AT LINE NUMBER num

11/4/2009 rwj


less COMMAND


• USED TO DISPLAY (NOT MODIFY) THE CONTENT OF A FILE 1 SCREEN AT A TIME

• SUPPORTS AN EXTENDED CAPABILITY OVER more BY OF ALLOWING BOTH FORWARD AND BACKWARD NAVIGATION THROUGH A FILE

$ less [options] [fileName]

11/4/2009 rwj


less COMMAND

• Useful Options-g HIGHLIGHTS JUST THE CURRENT MATCH OF ANY SEARCHED STRING. -I CASE-INSENSITIVE SEARCHES. -M SHOWS MORE DETAILED PROMPT, INCLUDING FILE POSITION. -N SHOWS LINE NUMBERS (USEFUL FOR SOURCE CODE VIEWING). -S DISABLES LINE WRAP ("CHOP LONG LINES"). LONG LINES CAN BE

SEEN BY SIDE SCROLLING. -? SHOWS HELP. ng JUMP TO LINE NUMBER n. DEFAULT IS THE START OF THE FILEnG JUMP TO LINE NUMBER n. DEFAULT IS THE END OF THE FILE


wc COMMAND

• SHORT FOR WORD COUNT, DISPLAYS 3 SEPARATE COUNTS ABOUT A FILE’S CONTENT IN THIS ORDER:

[# of lines] [# of words] [# of characters] filename

• COMMAND SYNTAX AND RESULTS$ wc [-c | -m | -C ] [-l] [-w] [ fileName ... ] $ command | wc

7 15 67 [fileName] – NO FILE NAME IF INPUT WAS A COMMAND

$

11/4/2009 rwj


wc COMMAND (CONT’D)

• OPTIONS-c COUNT BYTES. -m COUNT CHARACTERS. -C SAME AS -m. -l COUNT LINES. -w COUNT WORDS DELIMITED BY WHITE SPACE CHARACTERS OR

NEW LINE CHARACTERS. DELIMITING CHARACTERS ARE EXTENDED UNIX CODE (EUC) CHARACTERS FROM ANY CODE SET DEFINED BY iswspace()

fileName NAME OF FILE TO WORD COUNT.


diff COMMAND

• COMPARES 2 FILES OR 2 DIRECTORIES AND PRINTS THE LINES THAT ARE DIFFERENT, WHICH MAY BE THE ENTIRE FILES. PRODUCES "DELTA" -- LINES THAT ARE CHANGED OR ABSENT IN

EITHER OF FILES• CAN IDENTIFY WHICH LINES IN ONE FILE THAT

HAVE TO BE CHANGED TO MAKE IT IDENTICAL TO THE OTHER.diff [-b] [-i] [-t] [-w] [-c] [-C] [-e] [-f] [-h] [-

n] [-D string] [-l] [-r] [-s] [-S name] [fileone filetwo ] OR [directoryone directorytwo]


diff COMMAND (CONT’D)

• OPTIONS-e option PRODUCES A SCRIPT OF ONLY a (APPEND), c (CHANGE), AND d

(DELETE) COMMANDS FOR THE EDITOR ed, WHICH CAN BE USED TO CREATE A MODIFIED FILE2 FROM FILE1.

-i IGNOR CASE OF LETTER-t EXPAND TABS INTO SPACES ON OUTPUT-w IGNORE ALL WHITE SPACES (BLANKS AND TABS)-l DISPLAY LONG OUTPUT LISTING-r RECURSIVELY DESCEND THROUGH SUBDIRECTORIES-s DISPLAY FILES THAT ARE THE SAME-S name BEGIN THE DIRECTORY COMPARISON WITH FILE NAME


diff COMMAND (CONT’D)

• OPTIONS (CONT’D)-c[n] DISPLAYS A COMPARISON WITH N LINES OF CONTEXT. THE DEFAULT

VALUE OF N IS 3. THE OUTPUT BEGINS WITH THE IDENTIFICATION AND CREATION DATES OF EACH FILE. EACH CHANGE ENCOUNTERED IS SEPARATED WITH A DOZEN

ASTERISKS (*). LINES REMOVED FROM FILE1 ARE PRECEDED BY A HYPHEN (-). LINES REMOVED FROM FILE2 ARE PRECEDED BY A PLUS (+). LINES CHANGED FROM FILE1 TO FILE2 AND VICE VERSA ARE

PRECEDED WITH AN EXCLAMATION MARK (!).

.


cmp COMMAND

• COMPARES TWO FILES AND TELLS YOU WHICH LINE NUMBERS ARE DIFFERENT.

• SYNTAXcmp [-c] [-i N] [-l] [-s] [-v] firstFile secondFile

11/4/2009 rwj

c Output differing bytes as charactersi Ignore differences in the first N bytes of inputl Write the byte number (decimal) and the differing bytes

(octal) for each differences Write nothing for differing files; return exit statuses onlyv Output version info

Compare file1 to file2: cmp file1.txt file2.txt file.txt file2.txt differ: char 1011, line 112


sort COMMAND

• Sorts the lines in a text file using primary and/or secondary “keys” (columns of data).

• COLUMN DATA DELIMITED BY WHITE SPACE• SYNTAX

sort [-b] [-d] [-f] [-i] [-k #][-m] [-M] [-n] [-r]

[-u] [-tdelimiter][+fields] fileName [-o outfile]

command | sort

• EXAMPLE:o SORT A FILE USING THE 5TH COLUMN AND OUTPUT THE RESULT

INTO A NEW FILE.sort –k 5 –t: fileName >> sortedFile


sort COMMAND (CONT’D)

• White Space is displayed to represent one or more unprintable characters.

o EXAMPLES INCLUDE: CTL-H(S) SPACE CHARACTER(S) TAB CHARACTER(S)


sort COMMAND (CONT’D)

• Useful options:-f Case insensitive sort.-k # Sorts based on #th row.

-f #1,#2 Starts sort based on #1st row (field) and ends on #2nd row (field).

-m list Merges sorted file into list.

-n Sorts numerically.

-r Reverses sort order (z to a, instead of a to z).

-u Removes repeated lines (aka, leaves only unique lines in file.

-tdelimiter Changes the default value of delimiter to identify fields (columns) in output file.

-o outfile Places output in file name.


• DISPLAYS THE FIRST TEN LINES OF A FILE, UNLESS OTHERWISE STATED

• SYNTAXhead [-#] [-n #] fileName

• EXAMPLE$ head -25 fileName

11/4/2009 rwj

-# The number of lines to display. -n# The number of lines to display.

head COMMAND


• DISPLAYS A FILE’S THE LAST 10 LINES, OR THE # PROVIDED

• USEFUL FOR MONITORING LOGFILE GROWTH• SYNTAX

tail [-#] [-n #] [-f] [-r] [-l] [-b] [-c] fileName

• EXAMPLE tail -25 fileName

-# The number of lines to display. -n# The number of lines to display.-f Permits opening of files currently in use, e.g., logfiles-r Copies lines from the specified starting point in the file in reverse

order. The default for r is to print the entire file in reverse order.

tail COMMAND


• CUTS A COLUMN OF DATA (I.E., FIELDS) OUT OF EACH LINE IN A DELIMITED TEXT FILE.

• SYNTAXcut [[-b] [-c] [-f] list] [-n] [-d delim] [-s] [fileName]command | cut

• EXAMPLE cut –f3 fileName

-b The list following -b specifies byte positions (for instance, -b1-72 would pass the first 72 bytes of each line).

-c Specifies columns to be extract.-f Fields (columns) assumed to be separated in the file by a delimiter

character-d <ch> The character following -d is the field delimiter (-f option only).

Default is tab. Space or other characters with special meaning to the shell must be quoted.

cut COMMAND


• MERGE CORRESPONDING OR SUBSEQUENT LINES OF FILES IN COLUMN’D FORMAT INTO A DELIMITED TEXT FILE.

• PERMITS VIEWING CONTENTS SIDE BY SIDE AS WELL AS REORGANIZING A DELIMITED FILE.

• SYNTAXpaste [-s] [-d list] [fileName]paste fileName1 fileName2 | lpr

• EXAMPLE$ paste fileName1 fileName2 fileName3 > fileName4

paste COMMAND


• OPTIONS-s Concatenate all of the lines of each separate input file in

command line order. The NEWLINE character of every line except the last line in each input file will be replaced with the TAB character, unless otherwise specified by the -d option.

-d <ch> When the option is not specified:• The NEWLINE characters in the file specified by the last file will not be

modified. • Default delimiter is TAB.• If a backslash character appears in list, it has the following semantics:

\n Newline character. \t Tab character. \\ Backslash character. \0 Empty string (not a null character). If \0 is immediately followed

by the character “x” or “X”, the results are unspecified.

NOTE: Any character follow the backslash, the results are unspecified.

paste COMMAND (CONT’D)


uniq COMMAND

• LISTS UNIQUE LINES IN THE INPUT.• SYNTAX

uniq [-c | -d | -u ] [ -f fields ] [ -s char ] [-n] [+m] [inputFile [ outputFile ] ]

• EXAMPLEuniq –u inputFile OR uniq –c inputFilecut –d: -f3 inputFile | sort | uniq –c > outputFile

• USEFUL OPTIONS:-c Count of the frequency of occurrence of all lines along with the line.-u Select only the unique lines found in the input. Suppress the lines that

are repeated.-d Selects only the repeated lines found in input, suppresses unique lines.

NOTE: If 2 input files are included in the command, uniq overwrites the content of the 2nd file.


tr COMMAND

• MANIPULATES INDIVIDUAL CHARACTERS IN A FILE OR STREAM.

• SYNTAXtr [-c] [-d] [-s] [string1] [string2] [< inputFile] [> outputFile]

• EXAMPLEtr ‘:/’ ‘~-’ < inputFile | head –n 3

Changes all ‘:’ to ‘~’ and all ‘/’ to ‘–’ in input1 and displays top 3 lines on std output.

head –n 3 inputFile | tr ‘[a-z]’ ‘[A-Z]Reads and transfers top 3 lines in inputFile to tr which changes all lower case characters to

upper case and displays output on std output.

tr –d ‘:/’ < inputFile | head –n 3Strips all “:” and “/” characters from input1, passes results to HEAD which puts top 3 lines

to std output

tr -cd '\11\12\40-\176' < input1 > output1 Strips all non printable characters from input1 and puts results into output1


tr COMMAND (CONT’D)

• USEFUL OPTIONS:-d ‘st’ Deletes characters found in ‘st’ contained in inputFile.

-s ‘st’ Squeeze characters found in ‘st’ contained in inputFile. Way to eliminate multiple spaces

-c Complement characters.

NOTE: If 2 input files are included in the command, uniq overwrites the content of the 2nd file.


grep COMMAND

• FINDS TEXT WITHIN A FILE OR OUTPUT OF A COMMAND.

• SYNTAXgrep [options] PATTERN [FileName...]

grep [options] [-e PATTERN | -f FileName] [FileName...]

• PATTERNS FOR SEARCHING. Matches any single character. * Wild character. Example C* would find CC or CAT if in the input... {} Matches any character contained within the bracket. ^ Metacharacter that represents anchors the search to the beginning of

the line, e.g., ^T searches for any line starting with a T. $ Metacharacter that represents the end of the line. If use ‘\.$’ then it

would pull up any lines that ended with . (period) \ Means to take the next character as a literal, e.g., search for the string

“D\ D” in a file using ‘D\\ D’.


grep COMMAND (CONT’D)

• TO SEARCH FOR META-CHARACTERS E.G., & ! . * ? AND \, PRECEDE IT WITH A BACKSLASH (\).

• EXAMPLES:grep \\$ file1Search for a $ (dollar sign) in the file named file1

grep -E "abc|xyz" file1Display all lines in the file1 file that match either the string abc or xyz.

grep -v "^#" file1Display all lines that do not match a pattern

grep "^[a-zA-Z]" file1Display every line in file1 whose first character is a letter

ls -l *.ps | grep SepDisplay files that end in “.ps” and created in the month of September

Unix Comp-145 L ECTURE 7: UNIX U TILITIES AND F ILTERS B ASED ON : S. D AS, “Y OUR U NIX : T HE ULTIMATE G UIDE ”, 2 ND E DITION, M C G RAW H ILL, 2006.

Documents

c print

c ommand slide

text files c

c ut p aste

f c auses

c p s tn n pathfilenames

evel c ontent

c ommands t hat