Unix Seminar #1

Unix Seminar #1

T.J. Borrelli

Lecturer for CS and NSSA

February 6th, 2009

Processes.1
Types of processes:ForegroundShell waits for task to complete (before next prompt)Task is connected to terminalShell is tasks parentBackgroundShell does not wait for task to completeTask is connected to terminalShell is tasks parentDaemonShell does not wait for task to completeTask is not connected to terminalShell is not tasks parent (parent = PID 1)
Daemon (pronounced dee-mon) from Celtic mythology for a good or neutral spirit or inspiration.

Try and Learn
Start the following in the backgroundsleep 30 &When your prompt comes back, type
ps

Job Control.1
Shows you the processes that are running in the background, or suspended in the fore/backgroundIdeal for the multitaskingCan have several tasks running at onceCan switch among them at willStop one taskRestart anotherCan view stopped tasksCan kill tasks

Job Control.2

Job Control.3
Useful commands:^Z (ctrl-z)Stops a foreground jobJob is frozen in place until controlledbgPlaces the most recently stopped job in the backgroundAs though you had typed & at end of the lineCan use bg %N to move job N to background if there is more than one stopped job
(NOTE: the number after the % is not a process ID, it is a job number)jobsList existing fore/back-ground jobsEach appears with a job numberJob number IS NOT THE SAME AS process id

Job Control.4
More job control commandsfgBring the most recently stopped job OR background process to the foregroundCan use fg %N to bring job N to the foreground if there is more than one stopped job
(NOTE: the number after the % is not a process ID, it is a job numberkill %2Kill job 2 any legitimate job number may be usedCan also
kill 12954Kills by process IDkill -9 12954 kills with extreme prejudice^C (ctrl-c)Control-C kills the foreground job

Simple Shell Script.1
and Try and Learn
Using vi create a file named firstscript

Once created save the fileAt the prompt, type
bash firstscript
to run your script
#!/bin/bash
# First shell script
echo "List my home directory
ls -l

Simple Shell Script.2
What is a script?Text fileFirst line must be: #!/bin/bashContains one Unix command per lineLines that begin with # are comments (except that first one)When run, the commands are run in the order they appear in the script file

Script Execution.1
Run the script with the shellbash scriptnameRun the script as an executable commandType man chmod to learn about chmodTry:
chmod u+x scriptname
./scriptnameNOTE: no need to invoke the shell explicitlyThe shell to use is found in the #!/bin/bash line of the script

Script Execution.2
Scripts are run in subshellsCreate a script named myps

Type:
ps; bash mypsWhat is the value of PID and what processes are running?
#!/bin/bash
echo "My PS"
echo "PID=$$"
ps lx

Script Execution.3
When running a script, the shell starts a
shell

shell

Fork
(copy)

Input from terminal

Input from script

copy of itself called a subshell
The subshell reads commands directly from the script file

Environment.1
Scripts execute in an environmentEnvironment = state informationIncludes:Open filesCurrent directoryUser/groupSubshells inherit their environment from the shell that starts them

Environment.2
Create two scripts called firstScript and secondScript
#!/bin/bash

# This is called secondScript

echo secondScript

echo Working Directory

pwd

echo " "

echo User/group

id
Make both of these executable and run firstScriptNote: your directory and id are the same inside secondScript as in firstScript
#!/bin/bash

# This is called firstScript

echo firstScript

echo Working Directory

pwd

echo " "

echo User/group

id

./secondScript

Environment.3
NOW: rerun firstScript as:
./firstScript >outputThe >output sends the output to the file outputThis is called redirection as it redirects the output to the fileNOTE: the output of the secondScript is redirected to the file as well that of firstScript

Simple Shell Variables.1
AssignmentVAR_NAME="a string"NOTE: name is all upper case not a requirement, a convention to make variables stand out in scriptsNO spaces around the = requiredThe string is inside quotes requiredNO semicolon at end of line not allowedPrintingecho "$VAR_NAME"NOTE: the name is preceded by $ requiredThe $VAR_NAME is in quotes required sometimes

Simple Shell Variables.2
variables are not in the environmenti.e., they are not inherited by subshellsVariables can be placed in the environment by exporting themexport VAR_NAMEExports without setting the valueexport ANOTHER_VAR="a string"Exports and sets the value at onceModify firstScript and secondScript to try this out

The Command Line.1
The shell processes commands line-by-lineEach line is readThen parsed: separated into individual wordsThen executed: the first word is the command to execute and the remaining words (if any) are arguments to the commandThe parsing step can be altered by the use of special characters called meta characters

Special characters used to represent some other meanings

&Processes a job in background

sh MyJobInBackground &

;allows for multiple commands on line

echo n Enter name: ; read Name

> Redirects output

ls l > MyFiles

>> Same except appends to end of file

cd MyOtherDir ; ls l >> MyFiles

< Redirects input

read Line < MyFiles ; echo $Line

( )Starts subshell in which to run command(s)

A=5 ; echo $A ; (A=3;echo $A) ; echo $A

Shell Metacharacters (wildcards)

Shell Metacharacters (Cont)
$Substitutes variables?Matches for a single character*Matches for zero or more characters[abc] Matches for one character form a set of characters[!abc] Matches for one character not from a set of characters

Filename Generation.1
The shell will automatically generate a list of file names for you based on the use of wild card charactersCharacters:* any string of characters of any length? any single character[aeiou] any single character from the set inside the bracketsCan use with echo to find out what files match a patternecho a*files starting with aecho *[0-9]*files containing a digitecho [A-Ka-k]??Filenames 3 characters long that start
with an upper- or lower-case a,b,c,d,,k

EX: A=ls ; echo $AA=`ls`; echo $A
A=`ls`

for F in $A

do

if [ -f $F ] ; then

echo -n $F

fi

done

`back ticks`

Script.1
Write the following shell script

#!/bin/bash

# Name this script quotes

VAR="Hi There"

CMD=date

echo Trying with single quotes

echo '$VAR `$CMD`'

echo " "

echo Trying with double quotes

echo "$VAR `$CMD`"

I/O Redirection.1
As discussed earlier, there are three files automatically opened for each process:

I/O redirection is about hooking these files up to other filesWhen a file descriptor is assigned to something other than a terminal, it is called I/O redirection
Name File descriptor Purpose

---- --------------- -----------------------

stdin 0 keyboard

stdout 1 screen (normal output)

stderr 2 screen (error messages)

File descriptor.1
A small unsigned integerUsed by kernel to reference open file and I/O streamThe first three, 0, 1, and 2 are assigned to your terminal.A new open file will get a number of ?
All I/O, including files, pipes, and sockets, is handled by the kernel via a mechanism called the file descriptor.

First three file descriptors.2
STDIN = 0/dev/stdinSTDOUT=1 /dev/stdoutSTDERR=2/dev/stderr

We have already seen two types of redirection
echo hi there >output
Output (stdout) goes to disk file instead of screen

cat file | tr 'a-z' 'A-Z'
Output goes to stdin
of tr instead of screen
I/O Redirection.2

hi there

cat

tr

X

X

X

echo

0

1

2

0

1

2

0

1

2

I/O Redirection.3
Appending to a file
cat file1 >>file2
Redirects output to file2, just as with >HOWEVER appends to file2 instead of overwritingTry this:
echo "Today's time and date" >newfile

date >>newfile

cat newfile

I/O Redirection.4
Redirecting stderr
cat * >allfiles
If some of the files are not readable, will cause errors to go to screen.Solution, send stderr (file descriptor 2) to errorfile
cat * >allfiles 2>errorfile

cat

allfiles

errorfile

0

1

2

I/O Redirection.5

cat * >allfiles 2>errorfile

If we NEVER want to see the errors we can send them to a special file called /dev/null:

cat * > allfiles 2>/dev/null

/dev/null - is the bit bucket

Its like a black hole from which no data ever returns

cat

allfiles

0

1

2

Try and Learn
The find command will walk the file system looking for files the match your criteria and then act on them.
find / -name *.so print
Finds all files whose names end with .so starting at the root
(-name *.so)When it finds one, it prints its name (-print)Because you do not have permission to all directories, this will generate error messagesTry it as written above and seeExecute this command, discarding error messages

I/O Redirection.5
Sending stdout and stderr to same place
cat * >bigfile 2>&1

The 2>&1 says to send
file descriptor 2 to the same
place as 1.Order counts consider:
cat * 2>&1 >bigfile
Why? At the time we reach
2>&1, 1 is still
connected to the screen
cat

allfiles

cat

allfiles

0

1

2

0

1

2

find
Rerun the find command redirecting all of its output to a fileRedirect both normal output and error outputNow, try this:
find / -name '*.so' -print >file1 2>&1 >file2
what happens above? Can you explain?
NOTE:

>file1 redirects stdout to file1

2>&1 redirects stderr to file1 as well

>file2 redirects stdout a SECOND TIME to file2

So normal output goes to file2, errors go to file1

I/O Redirection.6
We can also redirect stdintr a-z A-Z

I/O Redirection.7
The exec command
exec file
exec 3&5# stdout goes to outfile
exec 5>&-# close outfile

The exec command can be used for a number of purposes but one is global file redirection within a script

Issuing this command will send

all stdout (fd 1) to file temp

Sends stdout (fd1) BACK to

screen /dev/tty

Exec > temp

Ls l /proc/self/fd

Exec > /dev/tty

More temp

lrwx------ 1 ypan faculty 64 Mar 26 16:27 0 -> /dev/pts/3

l-wx------ 1 ypan faculty 64 Mar 26 16:27 1 -> /raid/home/faculty/ypan/foobar/temp

lrwx------ 1 ypan faculty 64 Mar 26 16:27 2 -> /dev/pts/3

Pipelines.1
Each command has three I/O files automatically associated with it. Each has a name and a number (called the file descriptor)

The output of one command can be connected to the input of the next using a pipeline
cat file | tr 'a-z' 'A-Z'
Prints the file the output goes to tr filter which (in this case) translates all lower case characters to their upper case counterparts
Name File descriptor Purpose

---- --------------- -----------------------

stdin 0 keyboard

stdout 1 screen (normal output)

stderr 2 screen (error messages)

NOTICE HOW STDOUT OF ONE PROCESS IS CONNECTED

TO STDIN OF ANOTHER

103

Side note (time permitting): this command runs quickly how can you

check to see that the PID is actually different

Slow it doooown.

(who;sleep 2) | wc -l &

ps

# that example was kinda cheating

# to REALLY show that pipes behave this way try

# create a large file (~ 300,000 lines)

sort largefile | wc -l

ps

tborrelli@bart ~]$ ps f

PID TTY STAT TIME COMMAND

16498 pts/5 Ss 0:00 -bash

11509 pts/5 R 0:01 \_ sort out

11510 pts/5 S 0:00 \_ wc -l

11636 pts/5 R+ 0:00 \_ ps f

Pipelines.2
Pipelines can be very long
cat file | head -75 | tail -10Using man learn about head and tailExplain what the above pipeline does, assuming the file has 110 lines of text in itPipelines are neat because we dont have to create a bunch of temporary files!
$ cat file > file.1

$ head -75 file.1 > file.2

$ tail -10 file.2

$ rm file.1 file.2

Try and Learn
Type the command:Using the file file1 from the last exercise, try
cat file1
echo $?
You should see a 0 printedNow, change the permissions so that you do not have read permission for file1Repeat the above testWhat is the output this time

Positional Parameters.1
Shell scripts can receive parametersThe shell parses the command that starts the script into wordsEach word, except the script name, is passed to the script as a positional parameterYou can get a parameter if you know its position on the line$1 = 1st parameter, $2 = 2nd parameter, $9 = ninth parameter, ${10} = tenth parameter.$10 IS $1 with a 0 appended to it

Try and Learn
Write a script called swap that prints out its first two parameters in reverse ordere.g., swap a b prints swap Hello you guys prints The builtin variable $# tells you how many parameters there areChange swap so that it prints an error message and exits if there are not exactly 2 argumentsUse conditional execution and grouping as in:
[ $# -eq 1 ] || { echo Error message; exit 1; }

test command
Arguments indicate what test to performSort of like a Boolean expressiontest returns 0 if the test succeeds, 1 if it does notAlternative syntax to test [ ]Example:if test -d pathname
then
echo It IS a directory
fiif [ -d pathname ] # SPACES ARE VERY IMPORTANT HERE!
then
echo It IS a directory
fi

Type man test form more types of tests

If statements options
if [ $VAR -eq $VAR2 ]; then ;fi #equal toif [ $V -ne $V2 ]; then ;fi #not equal toif [ $V -gt $V2 ]; then ;fi # greater thanif [ $V -lt $V2 ]; then ;fi # less thanif [ $V -ge $V2 ]#greater than or equal toif [ $V -le $V2 ] # less than or equal to
There are others such as -f for file test -d directory -r readable,-w writable, -x executable

If you know how many parameters there are, you can get at all of them using the shift commandshiftDeletes $1 and shifts all the rest to
the left by 1 place:
$1 $2 $3 Repeated use gets to all parametersSpecial for loopfor VARNAME# executes 1 time/parameter
do
. . .# $VARNAME changes to next
# parameter on each iteration
done

In addition to $#, there are two other special variables for use with positional parameters$*list of all parameters$@list of all parametersThese appear to be the sameThey differ when quoted with double quotes

Try One More
Modify the for loop tofor NAME in "$*"
do
echo
doneCreate a script with the above in it and run itBe sure to include some quoted phrases as well as single words in your parametersNow change the for statement to
for NAME in "$@"
and try againHow do these to differ?What if you leave the quotes off?Which appears to operate the same as the for NAME (by itself)?

Here Documents.1
It reads in inline text for a program expecting inputData is read by using

Here Document.2
So Each line of the document is evaluated by the shellVariables are substituted (and other meta characters are evaluated)To avoid this, if you wish, single quote the markercommand

Commands
ps bg fg jobs kill and & sleepdatechmod cdlstouchmkdirrmdirmv
cp

rm

cat

cut paste

sort uniq