Basic Text Processing, Redirection and Pipes
Dec 29, 2015
222
Lecture Overview
Basic text processing commands head, tail, wc
Redirection and pipes
Getting to know vi
Basic vi commands
Advanced vi commands
333
Text Processing Commands
UNIX is strongly text oriented
It contains many utilities to search, modify or view text files
This provides a simple mechanism for storing data and for passing it between applications and utilities
444
Viewing Portions of a File –head and tail
The head command shows only the first few lines of a file
With no options, head shows first 10 lines Command line options for head:
-N – specify number of lines to show (N is a number, not the letter N)
-cN – show first N bytes, not lines
head [options] [files]
555
Viewing Portions of a File –head and tail
The tail command is similar to head, but shows the last lines of the file
In addition, tail provides these options: +N – show lines starting with line N -f – output appended data as the file grows. With
this option, the number of lines printed keeps growing as the file grows
666
head and tail – Examples
Given a file called 'my_phones.txt':ADAMS, Andrew 7583BARRETT, Bruce 6466BAYES, Ryan 6585BECK, Bill 6346BENNETT, Peter 7456GRAHAM, Linda 6141HARMER, Peter 7484MAKORTOFF, Peter 7328MEASDAY, David 6494NAKAMURA, Satoshi 6453REEVE, Shirley 7391ROSNER, David 6830
777
head and tail – Examples
head -3 my_phones.txt
ADAMS, Andrew 7583BARRETT, Bruce 6466BAYES, Ryan 6585
tail -3 my_phones.txt
NAKAMURA, Satoshi 6453REEVE, Shirley 7391ROSNER, David 6830
head –c30 my_phones.txt
ADAMS, Andrew 7583BARRETT, Br
888
head and tail – Examples
tail +9 my_phones.txt
MEASDAY, David 6494NAKAMURA, Satoshi 6453REEVE, Shirley 7391ROSNER, David 6830
tail –c50 my_phones.txt
toshi 6453REEVE, Shirley 7391ROSNER, David 6830
999
Counting Words – wc
The wc command counts the number of bytes, words and lines in a file
Command line options for wc: -c – print the number of bytes -w – print the number of words -l – print the number of lines
With no options, all three are printed
wc [options] [files]
101010
Lecture Overview
Basic text processing commands head, tail, wc
Redirection and pipes
Getting to know vi
Basic vi commands
Advanced vi commands
111111
Input and Output Redirection
Normally, when you run a command, it gets input from the keyboard (stdin),and sends output to the screen (stdout) Errors are sent to a third stream (stderr), which
by default also points to the screen
Input and output (as well as errors) can be read from or written to a file, by using the redirection meta-characters: '<' and '>'
121212
Output Redirection
We use the '>' operator to redirect the output of a command
For example, to create a file containing a listing of '.c' files in the current directory:ls *.c > c_files.txtcat !$
boxes.cparse_tools.cshape.ctools.c
131313
Input Redirection
We use the '<' operator to redirect the input of a command
For example, to see only the last two lines in the file 'c_files.txt':tail -2 < c_files.txt
shape.ctools.c
141414
Input / Output Redirection
Input and output redirection can be combined in a single command
For example:cat -n < my_phones.txt > numbered.txthead -4 !$
head -4 numbered.txt 1 ADAMS, Andrew 7583 2 BARRETT, Bruce 6466 3 BAYES, Ryan 6585 4 BECK, Bill 6346
151515
Appending Output
When redirecting output to a file, the specified file is either created (if it does not exist), or overwritten (if it exists)
The '>>' operator tells the shell to append the output to the file, without overwriting it
To redirect stderr in addition to stdout, we use the operator '>&'
161616
Appending Output – Example
echo first line > lines.txtecho second line >> lines.txtecho third line >> lines.txtcat lines.txt
first linesecond linethird line
echo fourth line > lines.txtcat lines.txt
fourth line
171717
Forced Redirection
Output redirection fails if we try to: Overwrite an existing file Append to a non-existent file
This behavior can be overridden by using the '!' modifier
When '!' is added to a redirection operator, the redirection is forced to succeed
181818
Forced Redirection – Examples
The following will create the file 'lines.txt' if it does not exist,or overwrite it if it does:
This will also create 'lines.txt' if it does not exist, but append to it if it does
echo first line >! lines.txt
echo first line >>! lines.txt
191919
Pipes
The following commands count how many '.c' files are in the current directory
This is not very efficient, and also leaves behind a temporary file as a side effect
What we really want is to link the output of ls directly to the input of wc
ls *.c > c_files.txtwc -l < c_files.txt
202020
Pipes
A pipe takes the output of one command, and passes it as input to another
Program 1(ls *.c)
Program 1(ls *.c)
Program 2(wc –l)
Program 2(wc –l)
stdout(or temp file)
stdin (or temp file)
Pipe
212121
Pipes
The symbol for a pipe is a vertical bar – '|'
The format of a command line which uses a pipe is:
Now the two commands from the previous example can be simply combined:
command1 [args] | command2 [args]
ls *.c | wc -l
222222
Pipes
The use of pipes is actually a simple and easy form of inter-process communication
Pipes can also be chained, so complex utilities can be created by combining simple building blocks
When data passes through several pipes, these are sometimes referred to as filters
232323
Pipes – Examples
Print lines 6 and 7 of 'my_phones.txt':
Another way to get the same results:
To find the most recently changed file:
head -7 my_phones.txt | tail -2
GRAHAM, Linda 6141HARMER, Peter 7484
tail +6 my_phones.txt | head -2
ls -t | head -1
242424
Lecture Overview
Basic text processing commands head, tail, wc
Redirection and pipes
Getting to know vi
Basic vi commands
Advanced vi commands
252525
Editing Files With vi
vi is the standard text editor in UNIX
Unlike most editors or word processors that you have encountered, it is modal – the same input produces different results in different modes Insert mode is for inserting text Command mode is for everything else
262626
Editing Files With vi
vi has hundreds of commands, whose names require memorization
It does not have menus or a GUI
All of these characteristics make vi: Very uncomfortable for beginners, but – Very powerful for experts
272727
Starting With vi
To edit a file, type:
You are now in command mode, so you cannot insert any text!
To enter insert mode, type: 'i' Now you can type in text as with any other
editor or word processor
vi [options] [file]
282828
Basic vi Commands
To switch back to command mode, press Escape (this has no effect if you are already in command mode)
In command mode, you can move around using the arrow keys, or these keys: 'h', 'j', 'k', 'l'
To delete a character, type 'x'
292929
Terminating a vi Session
To exit vi, you need to switch to a third mode, called last line mode, by typing ':'
Then, type one of the following: w – to save your changes q – to quit an unmodified file wq – to save your changes and quit q! – to quit without saving your changes
303030
Learning vi
The best way to learn vi is: Start using it with a few basic commands Use available help sources to learn more features as
you need them
Getting help: man vi Typing ':help' within vi UNIX books, the internet, etc.
313131
Lecture Overview
Basic text processing commands head, tail, wc
Redirection and pipes
Getting to know vi
Basic vi commands
Advanced vi commands
323232
vi Command Types
And now for a more detailed look
vi commands can be divided into groups: Commands for entering insert mode Commands for moving the cursor Commands for deleting or changing text Commands for searching or replacing text Miscellaneous commands
333333
Ways to Enter Insert Mode
Command Description
i Insert before cursor
I Insert at start of line
a Append after cursor
A Append to end of line
o Open a line below current line
O Open a line above current line
343434
Moving the Cursor
Command Description
h, j, k, l Left, down, up, right
Arrow keys Left, down, up, right
w Forward to start of next word
b Backward to start of previous word
Ctrl-F Forward one screen
Ctrl-B Backward one screen
353535
Moving the Cursor
Command Description
Ctrl-U Forward half a screen
Ctrl-D Backward half a screen
Ctrl-E Shift text down one line in window (cursor stays on same line)
Ctrl-Y Shift text up one line in window (cursor stays on same line)
% Move to the "mate" of this parenthesis or bracket
363636
Moving the Cursor
Command Description
$ Move to end of current line
0 Move to start of current line
^ Move to first non-whitespace character in current line
nG, :n Go to line n
gg Go to start of file
G Go to end of file
373737
Deleting Text
Command Description
x Delete character under cursor
X Delete character before cursor
dd Delete entire line
dw Delete current word
D Delete until end of line
383838
Changing Text
Command Description
cc Change entire line
cw Change current word
C Change until end of line
r Replace character under cursor
R Start overwriting until ESC
s Change one character and insert
393939
Copy and Paste
Command Description
yy Yank (copy) current line
yw Yank current word
p Put after (or below) cursor
P (capital) Put before (or above) cursor
]p Put, with indent adjusted
404040
Lecture Overview
Basic text processing commands head, tail, wc
Redirection and pipes
Getting to know vi
Basic vi commands
Advanced vi commands
414141
Defining Range Using Motion
With some commands, it is possible to add a motion modifier which defines the range that they effect
The general format is:
where motion is any of the available motion commands (e.g. d$)
command[motion]
424242
Defining Range Using Motion
In the command dw, d is the command (delete), and w is the motion (word) dd (and similarly, cc) is a special case – operates
on whole lines
Examples:
c$ Change until end of line
dG Delete until end of file
y% Yank block (from current to pair bracket)
434343
Repeating Commands
Most vi commands can be repeated, using the following format:
repeat defines how many times the given command should be executed
repeat and motion can be combined in a single command (e.g. 3dh)
[repeat]command
444444
Combining Repetition and Motion – Examples
Note: In 4yw, 4 is the repetition;in y4w, it is part of the motion
5dd Delete 5 lines, starting from current
3x Delete 3 characters, starting from current
2w Move forward 2 words
3dw Delete 3 words, starting from current
4yw Yank 4 words, starting from current
y4w Same as above
454545
Searching for Patterns
pattern can also be a regular expression
Command Description
/pattern Search forward
?pattern Search backward
n Repeat previous search
N Repeat previous search in reverse direction
464646
Replacing Text
The following command can be used to search and replace a pattern
This searches from the current position The g flag defines whether to replace only the
first occurrence, or all matches pattern can also be a regular expression
:s/search_string/replace_string/[g]
474747
Additional Commands
Command Description
. Repeat previous command
u Undo last command
U Undo all changes on current line
Ctrl-R Redo last undo
~ Switch case and advance cursor
J Merge current and next lines
484848
Additional Commands
Command Description
>> Indent line one tab-stop
<< Un-indent line one tab-stop
'' Go back to position before last move
ma Mark current position as a
'a Jump to position marked as a
494949
Commands in Last Line Mode
Command Description
:r file Insert file in cursor position
:w file Write current file with new name
:f Display details of current file
:!command Run a shell command
:r !command Run command, and insert result
:set var Set a vi environment variable
505050
Last Line Mode – Examples
:!pwd Show current directory
:!ls List contents of current directory
:r !date Insert current date in cursor position
:set nu Display line numbers
:set nonu Do not display line numbers
:set ai Auto-indent new lines according to indentation of previous ones
:set noai Do not auto-indent new lines
515151
The vi Startup File – .exrc
vi provides various customization options
Whenever vi is started, it first reads a file named .exrc from your home directory
This file may contain any last line mode command, such as: set – to set vi environment variables map – to define new behavior for keys