Complete Notes

UNIX OPERATING SYSTEM & SHELL PROGRAMMING


An Introduction of UNIX

What is UNIX ?

1. Unix is a computer operating system.2. An operating system is the program that controls all the other part of a

computer system , both the H/W and S/W. It allocates the computer’s resources and schedules tasks. It allows you to make use of the facilities provided by the system. Every computer requires an operating system.

3. UNIX is a multi-user , multi-tasking operating system. Multiple users may have multiple tasks running simultaneously. This is very different than PC operating systems.

4. UNIX is a machine independent operating system. Not specific to just one type of the computer H/W. Designed form the beginning to be independent of the computer hardware.

5. UNIX is a software development environment. Was born in and designed to function within this type of environment.

6. The UNIX trademark previously owned by AT & T and then deeded to UNIX Systems Laboratories (USL) , an AT & T subsidiary passed to Novell when it acquired USL. After a brief period of negotiations with rival UNIX vendors Sun Microsystems , Santa Cruz Operation , International Business Machines and Hewlett-Packard , Novell granted exclusive licensing rights to the UNIX trademark to X/Open Co. Ltd , an Open System industry standards branding agent based in the united Kingdom.

History of UNIX

In 1965 , Bell laboratories joined with MIT and General Electric to develop a new operating system called Multics (Multiplexed Information Computing Service) , which would provide multi-user , multi-processor and multi-level (hierarchical) file system.

In 1969 , AT&T developed the first UNIX file system with a few utilities and gave the name UNIX to it. In 1970 , January 1 time zero for UNIX started. In 1973 , UNIX was re-written in C by Dennis Ritchie . Being written in a high level language decreased the effort to port it to new machines.

In 1980 , BSD 4.1 (Berkelely Software Development ) was developed.

In 1983, SunOs , BSD 4.2 , System were released.

In 1983 , AT&T and Sun Microsystems jointly developed System V Release 4 (SVR 4).This was later developed in UNIX ware and Solaris 2.


Features of UNIX

1. The file and Process2. Multi-user3. Multi-tasking4. The Building Block Approach5. UNIX Tool-Kit6. Pattern Matching7. Programming Facility8. Documentation9. Communication System10. Security11. File Encryption/Decryption12. Portability

Benefits of UNIX

1. Stability2. Independence3. Remote Execution and administration4. Cost5. Ease of setup6. Fewer library conflicts7. No Viruses8. Long Filename9. Great Editor10. Productive environment for S/W development11. Distributed Processing and Multi-tasking12. Portability13. Reliable14. Efficiency & Flexibility15. Security

UNIX VERSIONS:

Xenix FreeBSD , NetBSD & OpenBSD Solaris AIX HP-UX Digital UNIX Tru64 UNIX IRIX SCO Open Server SCO UnixWare Linux

References :- Sumitabh Das Page No:- 18


Boot Block :- This block contain a small boot program and partition table. Super Block:-It contains globle information about the file system.It also

maintains a free list of inodes and data blacks that can be immediately allocated by the kernel when creating a file.

UNIX File System

Components of UNIX file System


This is mainly what it contains:- The size of the file system. The block size used by the file system. The number of free data blocks available and a partial list of

immediately allocable free data blocks. Number of free inodes available and a partial list of immediately

usable inodes. Last time of updating. The state of the file system (Whether “clean “ or “dirty”).

Inode Block :-This region contains the inode for every file of the file system. Data Block:-All data and programs created by users reside in this area.

INODE TABLEinode:- A structure maintained in a special area of the disk for storing the attribute of

a file. This table contains for every file its permission, ownership details, time stamps and number of links .However ,it doesn’t contain the filename.

inode number :- A number identifying an inode for a file which is unique in a single file system. It is displayed with the –i option of ls command.

The inode contains the following attributes of a file :-

File Type

File permission

Number of Links

The UID of the owner

The GID of the group owner

File size in bytes

Date and Time of lat change of the inode

An array of pointers that keep track of all disk blocks used by the file.


UNIX COMMANDSUnix is a operating system, used by most large, powerful

computers.

A commands is an instruction given th the shell ; kernel will obey that instruction. UNIX provides several commands for its users to easily work with it.

The general format of a command is :-

command -options command_arguments


DIRECTORY ORIENTED COMMANDS1.ls(list)-To shows the list of files and direrctories.

Syntax: ls [-option] [name of file name/ directory name]

Where

Option Description

-x Multicolumnar Output

-F Marks executables with *,directories with /, and symbolic links with @

-a Shows all filenames beginning with a dot including . And ..

-R Recursive list (Directory., Subdirectory., files)

-r Sorts filename in reverse order

-1 One filename in each line

-l long listing (Seven attributes of a file)

-d dirname Lists only dirname if dirname is a directory

-t Sorts filenames by last modification time

-lt Sorts listing by last modification time

-u Sorts filename by last access time

-lu Sorts by ASCII collating sequence but listing shows last access time

-lut As above but sorted by last access time

-i Displays inode number

Example$ls

xyz, pqr, abc , MCA4

$ls –l

Total 72

-rw-r—r– 1 kumar metal 19514 May 10 13:54 MCA4

-rw-xr-r- 1 kumar metal 4174 May 10 15:01 xyz

drwxr-xr-x 2 kumar meatl 84 Feb 12 12:30 pqr

$ls –li

Total 72

9045 -rw-r—r– 1 kumar metal 19514 May 10 13:54 MCA4

9056 -rw-xr-r- 1 kumar metal 4174 May 10 15:01 xyz

9045 drwxr-xr-x 2 kumar meatl 84 Feb 12 12:30 pqr


“[]” Represents a subset of related filenames. This can be used with range operator “-” to access a set of files . Multiple ranges must be separated by commas.

$ls pgm[1-5] This command will list only the files named , pgm1 , pgm2 , pgm3 , pgm4 , pgm5 if they exist in the current directory.

$ls pgm[1-5],[11-15]This command will list only the files named , pgm1 , pgm2 , pgm3 , pgm4 , pgm5, pgm11 , pgm12 , pgm13 , pgm14 , pgm15 if they exist in the current directory.

$ls a[bc]dThis command will list only the files named abd and acd

$ls a[!bc]dThis commands will list only the files named agd , ahd , akd , amd except abd and acd.

2.mkdir[Make Directory]This command is used to create new directory.General format is :-

mkdir [-p] <directory-name1> <directory-name2>The option –p is used to create consequences of directories using a single mkdir command.Example:-$mkdir ibrThis command will create ‘ibr’ a subdirectory of the current directory.

Wild Card Characters

‘*’ Represents any number of characters.

‘?’Represents a single character.

‘[RANGE]’Represents character list.

‘[Character_List]’ Matches any single character, Which is specified in Character List.

‘[!Character_list]’ Matches any single character , Which is not specified in Character List.

Example:-

$ls pgm*

This command will list out all the filenames of the current directory, which are starting with “pgm”.the sufffix to pgm may be any number of characters.

$ls *s

This command will display all the filenames of the current directory , which are ending with “s”.

$ls ?gms

This command will display four character filenames, which are ending with “gms” starting with any of the allowed character.

$ls gms?

This command will display four character filenames, which are starting with “gms” ending with any of the single character.


$mkdir x x/y This command will create x as a subdirectory of current working directory , y as subdirectory of x.

$mkdir ibr/ib/iThis command will make a directory i as a subdirectory of ibr/ib, but the directory structure –ibr/ib must exist.

$mkdir –p ibr/ib/iThen for the current directory , a subdirectory named ibr is created . Then for the directory ibr , a subdirectory named ib is created . After that the subdirectory i as created as a subdirectory of the directory ib.

3.rmdir[Remove Directory]This command is used to remove (delete) the specified directory. A directory should be empty before removing it.

Syntax:- rmdir [-p] <directory_name1> <directory_name2>

The option –p is used to remove consequence of directories using a single rmdir command.Example:-

$rmdir ibrThis command will remove the directory ibr , which is the subdirectory of the current directory.

$rmdir ibr/ib/iThis command will remove the directory i only.

$rmdir –p ibr/ib/iThis commad will remove the directory i , ib and ibr consequently.

4.cd[Change Directory]This command is used to change working directory to a specified directory.Syntax:- cd <existing_directory_name>Example:- cd /home/ibr

Then the directory /home/ibr becomes as the current working directory.

5.pwd[Present Working Directory / Print Working Directory]This command displays the full pathname for the current working directory.Example :-

$pwdOutput:- /home/bmi

Your present working directory is /home/bmi.

5. Find


find is one of the power tools of the UNIX system. It recursively examines a directory tree to look for files matching some criteria and then takes some action on the selected files. Syntax:-

find <path_list> <selection _criteria> action

This is how find operators: First , it recursively examines all files in the directories specified in path_list. It the matches each file for one or more selection_criteria. Finally , it takes some action on those selected files.

Example:-$find / -name a.out –print

Output:- /home/kumar/scripts/a.out/home/tiwari/scripts/reports/a.out/home/sharma/a.out

Example:-

$find /home/ibrahim –name “*.java” –print

$find /home/ibrahim –mtime 5 –print

$find /home/ibrahim –mtime +5 –print

$find /home/ibrahim –mtime -5 -print

6. du [Disk Usage]This command reports the disk spaces that are consumed by the files ina specified directory . Including all its subdirectories.

Syntax:- du [-option] [<directory_name /file_name>]Options Description

-a Displays counts for all files . Not just directories.-b Displays size in bytes.-c Displays output along with grand total of all arguments.-k Displays the size in Kilobytes.-m Displays the size in Megabytes.

Example:- $du o/p :- ?$du /home/bmi/text

Output:- 12 /home/bmi/text$du –a /home/bmi/text

Output:-4 /home/bmi/text/x.txt4 /home/bmi/text/y.txt

12 /home/bmi/text7.df (Disk Free)


This df command reports the available free space on the mounted file system or disks.Syntax:- df [-option]

Options Description -l Shows local file system only -k Displays the size in kilobytes -m Displays the size in Megabytes -i Reports free used percentage of used inode.

This command reports the free space in blocks. Generally one block in 512 bytes. The i-nodes entry indicates that up to the specified number of files can be created on the file.Example:- $dfFile System 1-Blocks Used Available Used Mounted

On

/dev/hda8 1612808 63632 1467248 5% /

/dev/hda8 23302 3489 18610 16% /boot

/dev/hda9 1035660 7100 975952 1% /home

/dev/hda10 1778840 742916 945560 44% /usr

/dev/hda11 1517920 1517920 1423784 2% /var

File Oriented Commands

1.cat :- Means Concatenate i.e. To add more than one string.

It is one of the most important command of UNIX Operating System. It is useful for creating a file and displaying a contents of file in a small terminal . cat command provides the various facility in UNIX Operating System. This various facility is also called various flavors of cat command.


2.cp [Copy]:- This command is used to copy the content of one file into another. If the

destination is an existing file , the file is overwritten . If the destination is an existing directory the file is copied into that directory.Syntax:-

cp [-option] <source_file> < destination_file>where options can be:-

i – prompt before overwritten destination files.p - Preserve all information ,including owner , group , permission and timestamps.R - Recursively copies files in all subdirectories.

Example:- $cp myfile myfile1

Output:-$cat myfileI am file one.

$cat myfile1I am file one.

3. rm [Remove]:-This am command is used to remove a file from the specified directory. To

remove a file , you must have write permission for the directory that contains the file , but you need not have permission on the file itself . If you do not have write permission on the file , the system will prompt before removing.Syntax:-

rm [-option] <filename>

Various Flavors of cat command


where options are as follows:-r -deletes all directories including the lower order directories.

Recursively deletes entire contents of the specified directory and the directory itself .

i - Prompts before deletingf - Removes write – protected files also, without prompting.

Example:-$rm myfile1This command deletes the file myfile form the current directory.

$rm –f /usr/ibrahim

This command deletes all the files and subdirectories of the specified directory /usr/ibrahim.

Note that the directory ‘ibrahim’ also will be deleted.

4.mv [Move]This command is used to remove the specified files / directories.

Syntax:-$mv <source> <destination>

Example:-mv file1.txt file2.txt

Output:-$cat file1.txtFile Not Found

$cat file2.txt

I am file1.

5. wc [Word Count]This command is used to display the number of lines,words and characters of

information stored on the specified file.Syntax:-

wc [-option] <File_Name>Where

l - Displays the number of lines in the filew – Displays the number of words in the filec – Display the number of characters in the file

Examples:- Command Output

$cat file1 I am file1$wc file1 1 3 10 file1$wc –l file1 1 file1


$wc –w file1 3 file1$wc –c file1 10 file1

6.ln(Link)This command is used to establish on additional filename to a specified file.It

doesn’t mean that creating more copies of the specified file.

Syntax:-ln <filename> <additional_filename>

Where <Filename> is the file name for which <additional_filename> is to located on any directory. Thus UNIX allows a file to have more than one name and yet maintain a single copy in the disk. But changes to one of these files are also reflected in the disk .But changes to one of these files are also reflected to the others. If you delete one filename using rm command , then the other link names will still exist.

Example:-$ls –l test1.txt-rw-rw-r-- 1 bmi bmi 75 jan 13 14:35 test1.txt

$ln test1.txt test2.txt$ls –l test*.txt-rw-rw-r-- 2 bmi bmi 75 jan 13 14:35 test1.txt-rw-rw-r-- 2 bmi bmi 75 jan 13 14:35 test2.txt

$rm test1.txt$ls –l test2.txt-rw-rw-r-- 1 bmi bmi 75 jan 13 14:35 test2.txt

7.fileThis command lists the general classification of a specified file. It lets you to

know if the contents of the specified file is ASCII text , C program text , directory and others.

Syntax:- file <filename>

Example:-$file test1.txt

Outputtest1.txt : ASCII text.

$file *Output

test1.test: ASCII textxyz: Directory

8.cmp [Compare]This command is used compare two files.


Syntax:- cmp <filename1> <filename2>

This command reports the first instance of differences between the specified files. That is the two files are compared byte by byte and the location of the first mismatch is echoed to the screen.

Example:-$cat file1.txtI am RamWhat is your name?

$cat file2.txtI am RamWhat are you doing?

$cmp file1.txt file2.txtOutput

file1.txt file2.txt differ : byte 10 char 1, line2

9. comm [Common]

This command uses two sorted files as argumnets and reports what is common.It compares each line of the first with its corresponding line in the second file. The output of this command is in three column as follows:

Column1 : Contains lines common for both filename 1 and filename2 Column2 : Contains lines unique to filename2.Column3 : Contains lines unique to filename1.

Syntax:- comm [-option] <filename1> <filename2>

Option:-1 - Suppresses listing of column12 - Suppresses listing of column23 - Suppresses listing

Example:-$cat file1.txt $cat file2.txtI am Ram I am RamWhat is your name? What are you doing?

$comm file1.txt file2.txtOutput:-

I am Ram What are you doing?

What is your name?

10. nl [ Non-Blank Line]This command numbers all non-blank line in the specified text file and display

the same on the screen. Syntax:-

nl <filename>Example:-


$cat file1.txtI am RamWhat is your name?

$nl file1.txt

Output:-1 I am Ram2 What is your name?

11. tac This command reverses a file contents , sothat the last line becomes the fisrt

line. Syntax:-

tac <filename>Example:-

$cat file1.txtI am RamWhat is your name?

$tac file1.txt

Output:-What is your name? I am Ram

12. tailThis command displays the end of the specified file.

Syntax:- tail -n <filename>

Example:-$cat student.txtMohan always keeps himself to himself.We were in a hurry. The road being zigzag ,We had to cut off a corner to reach in time.In his youth , he was practically rolling in money.

$tail -1 file1.txtOutput:-

In his youth , he was practically rolling in money.


We had to cut off a corner to reach in time.In his youth , he was practically rolling in money.

12. headThis command displays the top of the specified file.

Syntax:- head -n <filename>

Example:-


$cat student.txtMohan always keeps himself to himself.We were in a hurry. The road being zigzag ,We had to cut off a corner to reach in time.In his youth , he was practically rolling in money.

$head -1 file1.txtOutput:-

Mohan always keeps himself to himself.


Mohan always keeps himself to himself.We were in a hurry. The road being zigzag ,

File Access PermissionsUNIX treats everything as files.There are 3 types of files in UNIX as follows:-

1.Ordinary File2.Directory File3.Special File

There are 3 types of modes for accessing these files as follows:-

1. Read Mode(r)2.Write Mode(w)3.Execute Mode(x)

Unix separates its users into 3 groups for securty and convenience as follows :-

1. user (u)2. user group(g)3. others(o)

13. chmod [ Change Mode] This command is used to change the file permissions for an existing file. We can use any one of the symbolic and octal notations to change file permissions.

Syntax:- chmod <user_symbols set/deny_symbol access_symbol > <filename>

14 chgrp[Change Group]This command is used to change the group ownership of specified file.Syntax:-

chgrp <new_group name> <filename>

15.chown [Change Owner]This command is used to change the owner of a specified file.Syntax:-

chown <new_owner> <filename>

General Purpose Commands


1. Date :

Syntax : $date

2. WhoSince UNIX is a multi-user operating system , several users may work on this

system. This command is used to display the users who are logged on the system currently.

Syntax & Example :-$who

Output:-Aman tty1 jan 07 10:17

Shwetabh tty2 jan 07 10:20Neha tty4 jan 07 10:30

The first column of the output represents the user names. The second column represents the corresponding terminal names and the remaining columns represents the time at which the users are logged on.

3. Who am I This commands tells you who you are. (Working on the current terminal).

Syntax & Example :-$who am i

Output:-Aman tty1 jan 07 10:17

4.man (Manual)

This command displays the syntax and detailed usage of the UNIX command , which is supplied as argument.Syntax :-

$man <UNIX_COMMAND>Example:-

$man wcThis will display the help details for “wc” command.

Almost all of the command offer – help option that displays a short listing of all the options.

$wc --help

5. cal (calculator)This command will display the calender for the specified month and year.

Syntax :-$cal [month] <year]

where month can be ranged form 1 to 12Example:-

$cal 2008


This command will display calender for thr year 2008 (from 1 to 12 months).$cal 1 2008This command will display the calender for the month january of the year 2008.

5. cal (calculator)This command will display the calender for the specified month and year.

Syntax :-

$cal [month] <year] where month can be ranged form 1 to 12Example:-

$cal 2008This command will display calender for thr year 2008 (from 1 to 12 months).$cal 1 2008This command will display the calender for the month january of the year 2008.

6. lpr (Line Printer)This command is used to print one or more files on printer.

Syntax :-$lpr [-option] <filename1>………………..<filenamen>

where options can be :-r – Removes file(s) form directory after printing.m – Mail inform you when printing is over.

Example:-$lpr file1.txt

7. expr ()This command is used to perform arithmetics operations on integer.The arithmetics operator and their corresponding functions are given below.

Syntax :-+ Addition- Subtraction\* Multiplication/ Division [It gives my quotient of the division]% Remainder of division [ Modulas operator ]

Example:-$x = 5$y = 2

(i) expr $x + $y7

(ii) expr $x + $y3

(iii) expr $x \* $y10

(iv) expr $x / $y2 [Decimal portion is truncated]


(v) expr $x % $y1

8. bc [Basic Calculator]

This command is used to perform arithmetic operation on integer as well as on floats (Decimal Number].

Type the arithmetic expression in a line and press [ENTER] key. Then the answer will be displayed on the next line. After you have finished your work press [ctrl+d] keys to endup.

Example :- Add 10 and 20 $bc10 + 2030^d

$bc10 / 33 [Decimal portion is truncated]^d$bcscale = 18 / 32.6

scale = 28 / 32.66^d

$bc1. ibase = 2 ----set ibase to binary (2).101 -----Type the input binary number.

5 -----Result in Decimal

2. obase = 2 ----set obase to binary(2).5 ----Type the input decimal number101 ----Result in Binary form^d

Here similarly for Octal and hexadecimal numbers.


Pipes & Filters Commands

PIPE :-Pipe is a mechanism in which the output of one command can be redirected as

input to another command.Syntax:-

command | command2The output of command1 is sent to command2 as input.

Example:-$ls | more

The output of the command “ls” is sent to the “more” command as input. So the directory listing of the current directory is displayed page by page .

$cat file1.txt | sort file1.txt |wc file1.txt

REDIRECTION :-

UNIX treats the keyboard as the standard input (value 0) and terminal screen as standard output (value 1) as well as standard error (value 2). However , input can be taken from sources other than the keyboard and output can be passed to any source other than the terminal screen. such a process is called “redirection”.

1.Redirecting inputs2.Redirecting outputs3.Redirecting Error messages

1.Redirecting inputs :-The ‘<‘ symbol is used to redirect inputs.

Example:- $cat < file1.txtthen the file1.txt is taken as input for the command – cat

2.Redirecting outputs:-The “>” symbol is used to redirect outputs.

Example:- $ls > list.docThen the output of the command ‘ls’ is stored on the file “list.doc”. We can

also use ‘1>’ instead of ‘>’.

3.Redirecting Error Messages:-The ‘2>’ symbol is used to redirect error messages.

Example :- $cat list1.docIf there is no file named “list1.doc” in the current directory , then the error

message is sent to the standard error device. We can redirect this error messages using ‘2>’ symbol.

Example:- $cat list1.doc 2> error.txtThen the error messages , if generated , will be stored on the disk – error.txt.

FILTERS:-There are some UNIX commands that accept input from standard input or files

, perform some manipulation on it , and produces some output to the standard output. Since these commands perform some filtering operations on data , they are


appropriately called as “Filters”. These filters are used to display the contents of a file in stored order , extract the lines of a specified file that contains a specific pattern etc.

1. sort :-This command sorts the contents of a given file based on ASCII values of

characters.Syntax:-

sort [-option] <filename> Options-m <filelist> =Merge sorted files specified in <filelist>-o <filename> =stores output in the specified <filename>-r = Sorts the contents in reverse order.-u = Removes duplicate lines and display sorted content.-c= Checks if the file is sorted or not.

Example:-$cat file.txt $sort file.txt E A

S EA SW W

2. grep [Global Regular Expression Printing] :-This command is used to search for a specified pattern form a specified file

and display those lines containing the patter.Syntax:-

grep [-option] pattern <filename>Where options -b ignores spaces , tab.-i Ignore case -v Displays only the lines that do not match the specified pattern.-e Displays the total number of occurrences of the pattern in the file.-n Displays the resultant lines along with their line number.Example:-

$cat emp.ext1001 Ram Computer CS1002 Merry Electronics ET1003 John Computer CS

$grep “CS” emp.txto/p:- 1001 Ram Computer CS

1003 John Computer CS

Regular Expression Character Set*: Represents any number of characters?: Represents any single character.[r1-r2]: Range[âbcd] : Matches a single character which is not a,b,c or d.^[character]: Matches the lines that are beginning with the character specified in <Character>


[character]$ :Matches the lines that are ending with the character specified in <character>Example:-

$grep “Com*” emp.txto/p:- 1001 Ram Computer CS

1003 John Computer CSRelated commands with grep:- 1.egrep [ Extended grep]

2.fgrep [ Fixed grep]egrep :- This command offers additional features than grep. Multiple patterns can be searched by using pipe symbol. $grep “Ram | hari” emp.txt

O/P:- NO RESULTS

$ $egrep “Ram | hari” emp.txt :-1001 Ram Computer CS1003 John Computer CS

fgrep :- It is similar to grep command. But It does not accept regular expressions unlike grep.$ $fgrep “computer” emp.txt :-

1001 Ram Computer CS$fgrep “com*” emp.txtNO SUCH RESULT3.uniq {Unique}

This command is used to handle duplicate line in a file . If this command is used without any option , It displays the lines by eliminating duplicate lines.Syntax:-

uniq [-option] <filename>Where options are as follows:-

-u ==Displays only the no –repeated lines.-d ==Displays only the duplicate lines.-c ==Displays each line by eliminating duplicate lines and prefixing the

number of times it occurs.Example:-

$cat abc.txtabcxyzabcpqr

$uniq abc.txtabcxyzabcpqr

$uniq –u abc.txtO/P:-

xyz


pqr

$uniq –d abc.txtO/P:-

abc

$uniq –c abc.txtO/P:-

2 abc1 xyz1 pqr

cut :-This command is used to cut the columns/ fields of a specified file [Like the

head and tail commands cut the line-rows].Syntax:-

cut [-option] <filename>

Where options

c <Columns> -Cuts the column specified in <columns>. You must separate the column numbers by using commas.

f <Fields> - Cuts the fields specified in <fields> . You must separate field number by using commas.Example:-

$cat emp.txt1001 John Computer Cs1002 Merry Electronics ET1003 Ram Computer CS

$cut –c 2-5, 11-15 emp.txtOutput:-

001 Compu 002 Elec003 Comp

$cut -f 2,3 emp.txtOutput:-

John ComputerMerry ElectronicsRam Computer

$cut –d “|” –f 1,3- emp.txt1001 Computer CS1002 Electronics ET1003 Computer CS

$cut –f 1-3 emp.txtOutput:-


1001 John Computer1002 Merry Electronics1003 Ram Computer

$cut –c 2-5, 11-15 emp.txtOutput:-

001 Compu 002 Elec003 Comp

$cut -f 2,3 emp.txtOutput:-

John ComputerMerry ElectronicsRam Computer

$cut –d “|” –f 1,3- emp.txt1001 Computer CS1002 Electronics ET1003 Computer CS

$cut –f 1-3 emp.txtOutput:-

1001 John Computer1002 Merry Electronics1003 Ram Computer

tr [Translating Character]This tr command filter manipulates individual characters in a line. More specifically , it translates characters using one or two compact expressions:

tr options expression1 expression2 standard input

tr takes input only from the standard input , it doesn’t take a filename as argument. By default it translates each character in expression1 to its mapped counterpart in expression2. The first character in the first expression is replaced with the first character in the second expression and similarly for the other characters.

1. Replace a character:-

$cat emp.txtOutput:-

1001 | John | Computer | CS1002 | Merry | Electronics | ET1003 | Ram | Computer | CS

$tr ‘|’ ‘-’ emp.txt Output:-


1001 - John - Computer - CS1002 - Merry - Electronics - ET1003 - Ram - Computer - CS

2. Changing Case a Text

$cat emp.txt | tr ‘[a-z]’ ‘[A-Z]’

Output:-1001 | JOHN | COMPUTER | CS1002 | MERRY |ELECTRONICS | ET1003 | RAM | COMPUTER |CS

$cat emp.txt | tr ‘C,p’ ‘c,P’ Output:-

1001 | John | comPuter | cS1002 | Merry | Electronics | ET1003 | Ram | comPuter | CS

3. Deleting Characters (-d):-$cat emp.txt

Output:-1001| John | Computer | CS1002| Merry | Electronics | ET1003| Ram | Computer | CS

$tr –d ‘|’ emp.txtOutput:-

1001 John Computer CS1002 Merry Electronics ET1003 Ram Computer CS

sed [Stream Editor]:-This is a multipurpose tool which combines the work of several filters. It isderived from ed , the original editor. sed performs non-interactive operations

on a data stream – hence its name .

sed uses instructions to act on text. An instruction combines an address for selecting lines , with an action to be taken on them as shown by the syntax:-

sed [options]/[Edit Commands] ‘address action’ files

The address and action are enclosed within single quotes. Addressing in sed is done in two ways:-

By one or two line numbers. By specifying a /- enclosed pattern which occurs in a line.

Where edit commands can be:-

i – Inserts after line


a – Appends after linec – Changes linesd – Deletes linesp – Prints linesq – Quits

Example1:-$sed ‘2q’ emp.txtOutput:-

1001 John Computer CS1002 Merry Electronics ET

This command displays the 1st two lines of the emp.txt file.

Example2:- $sed ‘2d’ emp.txtOutput:-

1001 John Computer CS1003 Ram Computer CS

This command displays the contents of emp.txt by deleting the second line.

Example3:- $sed –n ‘2p’ emp.txtOutput:-

1002 Merry Electronics ETThis command displays the second line of the emp.txt.

Example4:- $sed –n ‘2,4p’ emp.txtThis command displays the lines 2 through 4 of emp.txt.

Example5:- $sed –n ‘$p’ e mp.txtThis command displays the last line of emp.txt.Example6:- $sed -1 ‘/cs/p’ emp.txtThis command displays the lines that are containing the pattern “cs”.

Example7: - $sed ‘/cs/d’ emp.txtThis command displays the lines that are not containing the specified pattern“cs”.

Example8:- $sed ‘i\ ---------------------------------’ emp.txtThis command inserts the specified dash line before each line of the fileemp.txt.

Example9:- $sed ‘a\ --------------------------------’ emp.txtThis command inserts the specified dash line after each line of the fileemp.txt.

Example10:- $sed ‘3i\ ------------------------------- ‘ emp.txtThis command inserts the specified dash line before the third line of the file emp.txt.

Example10:- $sed ‘3a\ ------------------------------- ‘ emp.txtThis command inserts the specified dash line after the third line of the file emp.txt.


Example 11:- $sed ‘$a\ -----------------------------------’ emp.txtThis command inserts the specified dash line after the last line of the file emp.txt

yacc - yet another compiler-compiler

The yacc command converts a context-free grammar into a set of tables for a … The lex(1) command is useful for creating lexical analyzers usable by yacc.

The yacc command converts a context-free grammar into a set of tables for a simple automaton that executes an LALR(1) parsing algorithm. The grammar may be ambiguous. Specified precedence rules are used to break ambiguities. The output file, y.tab.c, must be compiled by the C compiler to produce a function yyparse(). This program must be loaded with the lexical analyzer program, yylex(), as well as main() and yyerror(), an error handling routine. These rou- tines must be supplied by the user. The lex(1) command is useful for creating lexical analyzers usable by yacc.

vi (Visual) Editor This editor can be invoked by typing vi at the $ prompt. If you specify a filename as an argument to vi , then the vi will edit the specified files , if it exits.

vi [<filename>]

A status line at the bottom of the screen (25th line) shows the filenames, current line and character position in the edited file.

vi + <line_number> <filename>Edits the file specified in <filename> and places the cursor on the

<line_number>th line.

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbccccccccccccccccccccccccccccccccccccccccdddddddddddddddddddddddddddeeeeeeeeeeeeeeeeeeeeeeeee~~~~~~~:wq

vi Modes

Insert/Input Mode Command Mode Ex-Mode

http://www.scit.wlv.ac.uk/cgi-bin/mansec?1+lex

http://www.scit.wlv.ac.uk/cgi-bin/mansec?1+LALR


Insert Mode :-

1) The text should be entered in this mode and any key press in this mode is treated as text.

2) We can enter into this mode from command mode by pressing any of the keys:

i Ia Ao Or Rs S

Input Mode

Ex-Mode

Command Mode

Shell

vi foo :x, :q:wq & ZZ

[Enter]

:[Esc]

i,I,a,A,o,O,r,R,s, and S

THETHREEMODES


Command Mode:-

It is the default mode when we start up vi-editor.

All the commands on vi-editor (Cursor movement text manipulation etc. ) should be used in this mode.

We can enter into this mode from insert mode by pressing the [Esc] key and from Ex Mode by pressing [Enter Key].

Ex Command Mode:-

1. The Ex command mode (Saving files , find , replace etc ..) can be entered at the last line of the screen in this mode.

2)We can enter into this mode from command mode by pressing [:] key.

Insert Commands

i Inserts before cursor.I Inserts at the beginning of the current line.a Appends at the end of the current line.A Appends at the start of the current line.o Inserts a blank line below the current line.O Inserts a blank line above the current line.

Delete Commands

x -Deletes a character at the cursor position.

<n>x-Deletes specified number (n) of character.

X -Deletes a character before the cursor position.

<n>X -Deletes specified number (n) of characters before the cursor position.

dw -Deletes from current position to end of the current word.db -Deletes from cursor position to beginning of current word.dd -Deletes current line.<n>dd -Deletes specified number of lines (n) from the current line.

Replace Commands

r – Replaces single character at the cursor position.R –Replaces characters until[ESC] key is pressed from current cursor position.s – Replaces single character at the cursor position with any number of character.


S – Replaces entire line.

Search Commands

/string [Enter] – Searches the specified string forward in the file.?string [Enter]- Searches the specified string backward in the file.n - Finds the next string in the same direction.N - Finds the next string in the opposite direction.

Yanking (Copy & Paste) Commands

yy or (y) : Yanks the current line into the buffer (Copy).nyy or ny : Copies the ‘n’ lines from the current line to the buffer.p : Paste the yanked text below the current line.P : Paste the yanked text above the current line.

Ex Mode Commands

:w – Saves file without quitting:w <filename> - Saves the content into a file specified in <filename>:x or :wq – Saves file and quits from vi.:q! – Quits from vi without saving.

SET Commands

The two set command i.e. set nu and set ic.

The first is used to associate line numbers with the file . You are currently working with .

The second is used to ignore the case while searching a pattern in the file.

In addition to those two there are several other options that the user can set to customize the environment of vi.

To get a list of all the options available with set , use the command: (ESC): set all

Thought the list is long there are only a few which are most commonly used .

A list of these along with the affect of each is shown in figure:-

:set nu Set display of line numbers on.:set nonu Set display of line numbers off (Default).:set eb Beep the speaker when an error occurs (Default).:set noeb Do not Beep the speaker when an error occurs. :set ai Set auto indent on.:set noai Set auto indent off (default).


:set ic Ignores case while searching a pattern.:set noic Do not Ignores case while searching a pattern (default).

.exrc Profile and Miscellaneous Commands

Type :ToCtrl-gs how line number of current line.Ctrl-l redraw the entire display.:!sh fork a shell.:type Ctrl-d to get back to vi.. repeat last text change command at current

cursor position.:set showmodes how when you are in insert mode.:set ic ignore case when searching.:set noic turn ignore case off.:set nu turn on line numbering.:set nonu turn line numbering off.

NOTE:-Place the :set commands in an executable file in your home directory named ~/.exrc to automatically turn on these features for every vi edit session.


UNIT II


Shell

The shell is a command-line interface, which means it is soley text-based. The user can type commands to perform functions such as run programs, open and browse directories, and view processes that are currently running. Since the shell is only one layer above the operating system, you can perform operations that are not always possible using the graphical user interface (GUI). Some examples include moving files within the system folder and deleting files that are typically locked. The catch is, you need to know the correct syntax when typing the commands and you may still be prompted for a password in order to perform administrative functions.

Shells are most commonly associated with Unix, as many Unix users like to interact with the operating system using the text-based interface. Two common Unix shells are the Bourne shell and the C Shell, which is used by BSD. Most Unix systems have both of these shells available to the user. Windows users may be more familiar with DOS, the shell that has long been included with the Windows operating system. Most computer users have no need to use the shell interface, but it can be a fun way to perform functions on your computer, as well as impress your friends.

Types of Shell

Bourne shell (sh)

This is the original Unix shell written by Steve Bourne of Bell Labs. It is available on all UNIX systems.

This shell does not have the interactive facilites provided by modern shells such as the C shell and Korn shell. You are advised to to use another shell which has these features.

C shell (csh)

This shell was written at the University of California, Berkeley. It provides a C-like language with which to write shell scripts - hence its name.

TC shell (tcsh)

This shell is available in the public domain. It provides all the features of the C shell together with emacs style editing of the command line.

Korn shell (ksh)

This shell was written by David Korn of Bell labs. It is now provided as the standard shell on Unix systems.

It provides all the features of the C and TC shells together with a shell programming language similar to that of the original Bourne shell.

http://unixhelp.ed.ac.uk/scrpt/index.html

http://unixhelp.ed.ac.uk/emacs/index.html



It is the most efficient shell. Consider using this as your standard interactive shell.

Bourne Again Shell (bash)

This is a public domain shell written by the Free Software Foundation under their GNU initiative. Ultimately it is intended to be a full implementation of the IEEE Posix Shell and Tools specification. This shell is widely used within the academic commnity.

bash provides all the interactive features of the C shell (csh) and the Korn shell (ksh). Its programming language is compatible with the Bourne shell (sh).

If you use the Bourne shell (sh) for shell programming consider using bash as your complete shell environment.

Variables

A variable stores a string (try running these commands in a Bourne shell)

name="John Doe" echo $name

The quotes are required in the example above because the string contains a special character (the space)

A variable may store a number

num=137

The shell stores this as a string even though it appears to be a number

A few UNIX utilities will convert this string into a number to perform arithmetic

expr $num + 3

I/O Redirection

Topics covered: specifying the input or capturing the output of a command in a file

Utilities covered: wc, sort

The wc command counts the number of lines, words, and characters in a file

wc /etc/passwdwc -l /etc/passwd

You can save the output of wc (or any other command) with output redirection

wc /etc/passwd > wc.file



You can specify the input with input redirection

wc < /etc/passwd

Many UNIX commands allow you to specify the input file by name or by input redirection

sort /etc/passwdsort < /etc/passwd

You can also append lines to the end of an existing file with output redirection

wc -l /etc/passwd >> wc.file

Backquotes

Topics covered: capturing output of a command in a variable Utilities covered: date The backquote character looks like the single quote or apostrophe, but slants

the other way It is used to capture the output of a UNIX utility A command in backquotes is executed and then replaced by the output of the

command Execute these commands date save_date=`date` echo The date is $save_date Notice how echo prints the output of 'date', and gives the time when you

defined the save_date variable Store the following in a file named backquotes.sh and execute it (right click

and save in a file) #!/bin/sh # Illustrates using backquotes # Output of 'date' stored in a variable Today="`date`" echo Today is $Today Execute the script with sh backquotes.sh The example above shows you how you can write commands into a file and

execute the file with a Bourne shell Backquotes are very useful, but be aware that they slow down a script if you

use them hundreds of times You can save the output of any command with backquotes, but be aware that

the results will be reformated into one line. Try this: LS=`ls -l` echo $LS

Pipes


Topics covered: using UNIX pipes Utilities covered: sort, cat, head Pipes are used for post-processing data One UNIX command prints results to the standard output (usually the screen),

and another command reads that data and processes it sort /etc/passwd | head -5 Notice that this pipe can be simplified cat /etc/passwd | head -5 You could accomplish the same thing more efficiently with either of the two

commands: head -5 /etc/passwd head -5 < /etc/passwd For example, this command displays all the files in the current directory sorted

by file size ls -al | sort -n -r +4 The command ls -al writes the file size in the fifth column, which is why we

skip the first four columns using +4. The options -n and -r request a numeric sort (which is different than the

normal alphabetic sort) in reverse order

e file sizes in the current directory.

Storing Frequently Used Commands in Files: Shell Scripts

Shell Scripts Topics covered: storing commands in a file and executing the file Utilities covered: date, cal, last (shows who has logged in recently) Store the following in a file named simples and execute it #!/bin/sh # Show some useful info at the start of the day date echo Good morning $USER cal last | head -6 Shows current date, calendar, and a six of previous logins Notice that the commands themselves are not displayed, only the results To display the commands verbatim as they run, execute with sh -v simple.sh Another way to display the commands as they run is with -x sh -x simple.sh What is the difference between -v and -x? Notice that with -v you see '$USER'

but with -x you see your login name Run the command 'echo $USER' at your terminal prompt and see that the

variable $USER stores your login name With -v or -x (or both) you can easily relate any error message that may

appear to the command that generated it When an error occurs in a script, the script continues executing at the next

command


Verify this by changing 'cal' to 'caal' to cause an error, and then run the script again

Run the 'caal' script with 'sh -v simple.sh' and with 'sh -x simple.sh' and verify the error message comes from cal

Other standard variable names include: $HOME, $PATH, $PRINTER. Use echo to examine the values of these variables

Storing File Names in Variables Topics covered: variables store strings such as file names, more on creating

and using variables Utilities covered: echo, ls, wc A variable is a name that stores a string It's often convenient to store a filename in a variable Store the following in a file named variables.sh and execute it #!/bin/sh # An example with variables filename="/etc/passwd" echo "Check the permissions on $filename" ls -l $filename echo "Find out how many accounts there are on this system" wc -l $filename Now if we change the value of $filename, the change is automatically

propagated throughout the entire script

Scripting With sed Topics covered: global search and replace, input and output redirection Utilities covered: sed Here's how you can use sed to modify the contents of a variable: echo "Hello Jim" | sed -e 's/Hello/Bye/' Copy the file nlanr.txt to your home directory and notice how the word 'vBNS'

appears in it several times Change 'vBNS' to 'NETWORK' with sed -e 's/vBNS/NETWORK/g' < nlanr.txt You can save the modified text in a file with output redirection sed -e 's/vBNS/NETWORK/g' < nlanr.txt > nlanr.new Sed can be used for many complex editing tasks, we have only scratched the

surface here

More on Using UNIX Utilities

Performing Arithmetic Topics covered: integer arithmetic, preceding '*' with backslash to avoid file

name wildcard expansion Utilities covered: expr


Arithmetic is done with expr expr 5 + 7 expr 5 \* 7 Backslash required in front of '*' since it is a filename wildcard and would be

translated by the shell into a list of file names You can save arithmetic result in a variable Store the following in a file named arith.sh and execute it #!/bin/sh # Perform some arithmetic x=24 y=4 Result=èxpr $x \* $y` echo "$x times $y is $Result"

Translating Characters Topics covered: converting one character to another, translating and saving

string stored in a variable Utilities covered: tr Copy the file sdsc.txt to your home directory The utility tr translates characters tr 'a' 'Z' < sdsc.txt This example shows how to translate the contents of a variable and display the

result on the screen with tr Store the following in a file named tr1.sh and execute it #!/bin/sh # Translate the contents of a variable Cat_name="Piewacket" echo $Cat_name | tr 'a' 'i' This example shows how to change the contents of a variable Store the following in a file named tr2.sh and execute it #!/bin/sh # Illustrates how to change the contents of a variable with tr Cat_name="Piewacket" echo "Cat_name is $Cat_name" Cat_name=ècho $Cat_name | tr 'a' 'i'` echo "Cat_name has changed to $Cat_name" You can also specify ranges of characters. This example converts upper case to lower case tr 'A-Z' 'a-z' < file Now you can change the value of the variable and your script has access to the

new value

Performing Search and Replace in Several Files

Processing Multiple Files Topics covered: executing a sequence of commands on each of several files

with for loops Utilities covered: no new utilities Store the following in a file named loop1.sh and execute it #!/bin/sh # Execute ls and wc on each of several files


# File names listed explicitly for filename in simple.sh variables.sh loop1.sh do echo "Variable filename is set to $filename..." ls -l $filename wc -l $filename done This executes the three commands echo, ls and wc for each of the three file

names You should see three lines of output for each file name filename is a variable, set by "for" statement and referenced as $filename Now we know how to execute a series of commands on each of several files

Using File Name Wildcards in For Loops

Topics covered: looping over files specified with wildcards

Utilities covered: no new utilities

Store the following in a file named loop2.sh and execute it

#!/bin/sh# Execute ls and wc on each of several files# File names listed using file name wildcardsfor filename in *.shdo

echo "Variable filename is set to $filename..."ls -l $filenamewc -l $filename

done

You should see three lines of output for each file name ending in '.sh'

The file name wildcard pattern *.sh gets replaced by the list of filenames that exist in the current directory

For another example with filename wildcards try this command

echo *.sh

Search and Replace in Multiple Files

Topics covered: combining for loops with utilities for global search and replace in several files

Utilities covered: mv Sed performs global search and replace on a single file sed -e 's/application/APPLICATION/g' sdsc.txt > sdsc.txt.new The original file sdsc.txt is unchanged How can we arrange to have the original file over-written by the new version? Store the following in a file named s-and-r.sh and execute it #!/bin/sh


# Perform a global search and replace on each of several files # File names listed explicitly for text_file in sdsc.txt nlanr.txt do echo "Editing file $text_file" sed -e 's/application/APPLICATION/g' $text_file > temp mv -f temp $text_file done First, sed saves new version in file 'temp' Then, use mv to overwrite original file with new version

Using Command-line Arguments for Flexibility

What's Lacking in the Scripts Above?

Topics covered: looping over files specified with wildcards


File names are hard-coded inside the script

What if you want to run the script but with different file names?

To execute for loops on different files, the user has to know how to edit the script

Not simple enough for general use by the masses

Wouldn't it be useful if we could easily specify different file names for each execution of a script?

What are Command-line Arguments? Topics covered: specifying command-line arguments Utilities covered: no new utilities Command-line arguments follow the name of a command ls -l .cshrc /etc The command above has three command-line arguments -l (an option that requests long directory listing) .cshrc (a file name) /etc (a directory name) An example with file name wildcards: wc *.sh How many command-line arguments were given to wc? It depends on how

many files in the current directory match the pattern *.sh Use 'echo *.sh' to see them


Most UNIX commands take command-line arguments. Your scripts may also have arguments

Accessing Command-line Arguments

Topics covered: accessing command-line arguments


Store the following in a file named args1.sh

#!/bin/sh# Illustrates using command-line arguments# Execute with# sh args1.sh On the Waterfrontecho "First command-line argument is: $1"echo "Third argument is: $3"echo "Number of arguments is: $#"echo "The entire list of arguments is: $*"

Execute the script with

sh args1.sh -x On the Waterfront

Words after the script name are command-line arguments

Arguments are usually options like -l or file names

Looping Over the Command-line Arguments Topics covered: using command-line arguments in a for loop Utilities covered: no new utilities Store the following in a file named args2.sh and execute it #!/bin/sh # Loop over the command-line arguments # Execute with # sh args2.sh simple.sh variables.sh for filename in "$@" do echo "Examining file $filename" wc -l $filename done This script runs properly with any number of arguments, including zero The shorter form of the for statement shown below does exactly the same

thing for filename


do ... Don't use for filename in $* Fails if any arguments include spaces Also, don't forget the double quotes around $@

If Blocks Topics covered: testing conditions, executing commands conditionally Utilities covered: test (used by if to evaluate conditions) This will be covered on the whiteboard See Chapter 8 of the book

The read Command Topics covered: reading a line from the standard input Utilities covered: no new utilities stdin is the keyboard unless input redirection used Read one line from stdin, store line in a variable read variable_name Ask the user if he wants to exit the script Store the following in a file named read.sh and execute it #!/bin/sh # Shows how to read a line from stdin echo "Would you like to exit this script now?" read answer if [ "$answer" = y ] then echo "Exiting..." exit 0 fi

Command Exit Status Topics covered: checking whether a command succeeds or not Utilities covered: no new utilities Every command in UNIX should return an exit status Status is in range 0-255 Only 0 means success Other statuses indicate various types of failures Status does not print on screen, but is available thru variable $? Example shows how to examine exit status of a command Store the following in a file named exit-status.sh and execute it #!/bin/sh # Experiment with command exit status echo "The next command should fail and return a status greater than zero" ls /nosuchdirectory echo "Status is $? from command: ls /nosuchdirectory" echo "The next command should succeed and return a status equal to zero" ls /tmp echo "Status is $? from command: ls /tmp" Example shows if block using exit status to force exit on failure Store the following in a file named exit-status-test.sh and execute it #!/bin/sh


# Use an if block to determine if a command succeeded echo "This mkdir command fails unless you are root:" mkdir /no_way if [ "$?" -ne 0 ] then # Complain and quit echo "Could not create directory /no_way...quitting" exit 1 # Set script's exit status to 1 fi echo "Created directory /no_way" Exit status is $status in C shell

Regular Expressions Topics covered: search patterns for editors, grep, sed Utilities covered: no new utilities Zero or more characters: .* grep 'provided.*access' sdsc.txt sed -e 's/provided.*access/provided access/' sdsc.txt Search for text at beginning of line grep '^the' sdsc.txt Search for text at the end of line grep 'of$' sdsc.txt Asterisk means zero or more the the preceeding character a* zero or more a's aa* one or more a's aaa* two or more a's Delete all spaces at the ends of lines sed -e 's/ *$//' sdsc.txt > sdsc.txt.new Turn each line into a shell comment sed -e 's/^/# /' sdsc.txt

Greed and Eagerness Attributes of pattern matching Greed: a regular expression will match the largest possible string Execute this command and see how big a string gets replaced by an

underscore echo 'Big robot' | sed -e 's/i.*o/_/' Eagerness: a regular expression will find the first match if several are present

in the line Execute this command and see whether 'big' or 'bag' is matched by the regular

expression echo 'big bag' | sed -e 's/b.g/___/' Contrast with this command (notice the extra 'g') echo 'big bag' | sed -e 's/b.g/___/g' Explain what happens in the next example echo 'black dog' | sed -e 's/a*/_/' Hint: a* matches zero or more a's, and there are many places where zero a's

appear Try the example above with the extra 'g' echo 'black dog' | sed -e 's/a*/_/g'


Regular Expressions Versus Wildcards Topics covered: clarify double meaning of asterisk in patterns Utilities covered: no new utilities Asterisk used in regular expressions for editors, grep, sed Different meaning in file name wildcards on command line and in find

command and case statement (see below) regexp wildcard meaning .* * zero or more characters, any type . ? exactly one character, any type [aCg] [aCg] exactly one character, from list: aCg Regexps can be anchored to beginning/ending of line with ^ and $ Wildcards automatically anchored to both extremes Can use wildcards un-anchored with asterisks ls *bub*

Getting Clever With Regular Expressions Topics covered: manipulating text matched by a pattern Utilities covered: no new utilities Copy the file animals.txt to your home directory Try this sed command, which changes the first line of animals.txt sed -e "s/big $.*$ dog/small \1 cat/" animals.txt Bracketing part of a pattern with $ and $ labels that part as \1 Bracketing additional parts of a pattern creates labels \2, \3, ... This sed command reverses the order of two words describing the rabbit sed -e "s/Flopsy is a big $.*$ $.*$ rabbit/A big \2 \1 rabbit/" < animals.txt

The case Statement Topics covered: choosing which block of commands to execute based on

value of a string Utilities covered: no new utilities The next example shows how to use a case statement to handle several

contingencies The user is expected to type one of three words A different action is taken for each choice Store the following in a file named case1.sh and execute it #!/bin/sh # An example with the case statement # Reads a command from the user and processes it echo "Enter your command (who, list, or cal)" read command case "$command" in who) echo "Running who..." who ;; list) echo "Running ls..." ls ;; cal) echo "Running cal..." cal


;; *) echo "Bad command, your choices are: who, list, or cal" ;; esac exit 0 The last case above is the default, which corresponds to an unrecognized entry The next example uses the first command-line arg instead of asking the user to

type a command Store the following in a file named case2.sh and execute it #!/bin/sh # An example with the case statement # Reads a command from the user and processes it # Execute with one of # sh case2.sh who # sh case2.sh ls # sh case2.sh cal echo "Took command from the argument list: '$1'" case "$1" in who) echo "Running who..." who ;; list) echo "Running ls..." ls ;; cal) echo "Running cal..." cal ;; *) echo "Bad command, your choices are: who, list, or cal" ;; esac The patterns in the case statement may use file name wildcards

The while Statement Topics covered: executing a series of commands as long as some condition is

true Utilities covered: no new utilities The example below loops over two statements as long as the variable i is less

than or equal to ten Store the following in a file named while1.sh and execute it #!/bin/sh # Illustrates implementing a counter with a while loop # Notice how we increment the counter with expr in backquotes i="1" while [ $i -le 10 ] do echo "i is $i" i=èxpr $i + 1` done

Example With a while Loop Topics covered: Using a while loop to read and process a file


Utilities covered: no new utilities Copy the file while2.data to your home directory The example below uses a while loop to read an entire file The while loop exits when the read command returns false exit status (end of

file) Store the following in a file named while2.sh and execute it #!/bin/sh # Illustrates use of a while loop to read a file cat while2.data | \ while read line do echo "Found line: $line" done The entire while loop reads its stdin from the pipe Each read command reads another line from the file coming from cat The entire while loop runs in a subshell because of the pipe Variable values set inside while loop not available after while loop

Interpreting Options With getopts Command Topics covered: Understand how getopts command works Utilities covered: getopts getopts is a standard UNIX utility used for our class in scripts getopts1.sh and

getopts2.sh Its purpose is to help process command-line options (such as -h) inside a script It handles stacked options (such as -la) and options with arguments (such as -P

used as -Pprinter-name in lpr command) This example will help you understand how getopts interprets options Store the following in a file named getopts1.sh and execute it #!/bin/sh # Execute with # # sh getopts1.sh -h -Pxerox file1 file2 # # and notice how the information on all the options is displayed # # The string 'P:h' says that the option -P is a complex option # requiring an argument, and that h is a simple option not requiring # an argument. # # Experiment with getopts command while getopts 'P:h' OPT_LETTER do echo "getopts has set variable OPT_LETTER to '$OPT_LETTER'" echo " OPTARG is '$OPTARG'" done used_up=èxpr $OPTIND - 1` echo "Shifting away the first \$OPTIND-1 = $used_up command-line arguments" shift $used_up


echo "Remaining command-line arguments are '$*'" Look over the script getopts looks for command-line options For each option found, it sets three variables: OPT_LETTER, OPTARG,

OPTIND OPT_LETTER is the letter, such as 'h' for option -h OPTARG is the argument to the option, such as -Pjunky has argument 'junky' OPTIND is a counter that determines how many of the command-line

arguments were used up by getopts (see the shift command in the script) Execute it several times with sh getopts1.sh -h -Pjunky sh getopts1.sh -hPjunky sh getopts1.sh -h -Pjunky /etc /tmp Notice how it interprets -h and gives you 'h' in variable OPT_LETTER Now you can easily implement some operation when -h is used Notice how the second execution uses stacked options Notice how the third execution examines the rest of the command-line after

the options (these are usually file or directory names)

Example With getopts Topics covered: interpreting options in a script Utilities covered: getopts The second example shows how to use if blocks to take action for each option Store the following in a file named getopts2.sh and execute it #!/bin/sh # # Usage: # # getopts2.sh [-P string] [-h] [file1 file2 ...] # # Example runs: # # getopts2.sh -h -Pxerox file1 file2 # getopts2.sh -hPxerox file1 file2 # # Will print out the options and file names given # # Initialize our variables so we don't inherit values # from the environment opt_P='' opt_h='' # Parse the command-line options while getopts 'P:h' option do case "$option" in "P") opt_P="$OPTARG" ;; "h") opt_h="1" ;; ?) echo "getopts2.sh: Bad option specified...quitting" exit 1 ;;


esac done shift èxpr $OPTIND - 1` if [ "$opt_P" != "" ] then echo "Option P used with argument '$opt_P'" fi if [ "$opt_h" != "" ] then echo "Option h used" fi if [ "$*" != "" ] then echo "Remaining command-line:" for arg in "$@" do echo " $arg" done fi Execute it several times with sh getopts2.sh -h -Pjunky sh getopts2.sh -hPjunky sh getopts2.sh -h -Pjunky /etc /tmp Can also implement actions inside case statement if desired

Using Functions

Functions Sequence of statements that can be called anywhere in script Used for

o Good organization o Create re-usable sequences of commands

Define a Function Define a function echo_it () { echo "In function echo_it" } Use it like any other command echo_it Put these four lines in a script and execute it

Function Arguments Functions can have command-line arguments echo_it () { echo "Argument 1 is $1" echo "Argument 2 is $2"


} echo_it arg1 arg2 When you execute the script above, you should see Argument 1 is arg1 Argument 2 is arg2 Create a script 'difference.sh' with the following lines: #!/bin/sh echo_it () { echo Function argument 1 is $1 } echo Script argument 1 is $1 echo_it Barney Execute this script using sh difference.sh Fred Notice that '$1' is echoed twice with different values The function has separate command-line arguments from the script's

Example With Functions Use functions to organize script read_inputs () { ... } compute_results () { ... } print_results () { ... } Main program very readable read_inputs compute_results print_results

Functions in Pipes Can use a function in a pipe ls_sorter () { sort -n +4 } ls -al | ls_sorter Function in pipe executed in new shell New variables forgotten when function exits

Inherited Variables Variables defined before calling script available to script func_y () { echo "A is $A" return 7 } A='bub' func_y if [ $? -eq 7 ] ; then ... Try it: is a variable defined inside a function available to the main program?

Functions -vs- Scripts Functions are like separate scripts Both functions and scripts can: Use command-line arguments echo First arg is $1


Operate in pipes echo "test string" | ls_sorter Return exit status func_y arg1 arg2 if [ $? -ne 0 ] ...

Libraries of Functions Common to store definitions of favorite functions in a file Then execute file with . file Period command executes file in current shell Compare to C shell's source command

Set: Shell Options Can change Bourne shell's options at runtime Use set command inside script set -v set +v set -xv Toggle verbose mode on and off to reduce amount of debugging output

Set: Split a Line Can change Bourne shell's options set -- word1 word2 echo $1, $2 word1, word2 Double dash important! Word1 may begin with a dash, what if word1 is '-x'? Double dash says "even if first word begins with '-', do not treat it as an option

to the shell

Example With Set Read a line from keyboard Echo words 3 and 5 read var set -- $var echo $3 $5 Best way to split a line into words

Trapping Signals


What are Signals? Signals are small messages sent to a process Process interrupted to handle signal Possibilities for managing signal:

o Terminate o Ignore o Perform a programmer-defined action

Common Signals Common signals are

o SIGINTR sent to foreground process by ^C o SIGHUP sent when modem line gets hung up o SIGTERM sent by kill -9

Signals have numeric equivalents 2 SIGINTR 9 SIGTERM

Send a Signal Send a signal to a process kill -2 PID kill -INTR PID

Trap Signals Handling Signals trap "echo Interrupted; exit 2" 2 Ignoring Signals trap "" 2 3 Restoring Default Handler trap 2

Where to Find List of Signals See file /usr/include/sys/signal.h

User Signals SIGUSR1, SIGUSR2 are for your use Send to a process with kill -USR1 PID Default action is to terminate process

Experiment With Signals Script that catches USR1 Echo message upon each signal trap 'echo USR1' 16 while : ; do date sleep 3 done Try it: does signal interrupt sleep?


Understanding Command Translation

Command Translation Common translations include

o Splitting at spaces, obey quotes o $HOME -> /users/us/freddy o `command` -> output of command o I/O redirection o File name wildcard expansion

Combinations of quotes and metacharacters confusing Resolve problems by understanding order of translations

Experiment With Translation Try wildcards in echo command echo b* b budget bzzzzz b* translated by sh before echo runs When echo runs it sees echo b budget bzzzzz Echo command need not understand wildcards!

Order of Translations Splits into words at spaces and tabs Divides commands at ; & | && || (...) {...} Echos command if -v Interprets quotes Performs variable substitution

Performs command substitution Implements I/O redirection and removes redirection characters Divides command again according to IFS Expands file name wildcards Echos translated command if -x Executes command

Exceptional Case Delayed expansion for variable assignments VAR=b* echo $VAR b b_file Wildcard re-expanded for each echo

Examples With Translation


Variables translated before execution Can store command name in variable command="ls" $command file1 file2 dir1 dir2... Variables translated before I/O redirection tempfile="/tmp/scriptname_$$" ls -al > $tempfile Delayed expansion of wildcards in variable assignment Output of this echo command changes when directory contents change (* is

re-evaluated each time the command is run) x=* echo $x Can view values stored in variables with set Try it: verify that the wildcard is stored in x without expansion

Wildcards expanded after redirection (assuming file* matches exactly one file):

cat < file* file*: No such file or directory Command in backquotes expanded fully (and before I/O redirection) cat < ècho file*` (contents of file sent to screen)

Eval Command Forces an extra evaluation of command eval cat \< file* (contents of matching file) Backslash delays translation of < until second translation

Writing Advanced Loops

While loops Execute statements while a condition is true i=0 while [ $i -lt 10 ] do echo I is $i i=èxpr $i + 1` done

Until loops Execute statements as long as a condition is false until grep "sort" dbase_log > /dev/null do sleep 10 done echo "Database has been sorted" Example executes until grep is unsuccessful


Redirection of Loops Can redirect output of a loop for f in *.c do wc -l $f done > loop.out Loop runs in separate shell New variables forgotten after loop Backgrounding OK, too

Continue Command Used in for, while, and until loops Skip remaining statements Return to top of loop for name in * do if [ ! -f $name ] ; then continue fi echo "Found file $name" done Example loops over files, skips directories

Break Command Used in for, while, and until loops Skip remaining statements Exit loop for name in * do if [ ! -r $name ] ; then echo "Cannot read $name, quitting loop" break fi echo "Found file or directory $name" done Example loops over files and directories, quits if one is not readable

Case Command Execute one of several blocks of commands case "string" in pattern1) commands ;; pattern2) commands ;; *) # Default case commands ;; esac Patterns specified with file name wildcards quit) ... qu*) ...


Example With Case Read commands from keyboard and interpret Enter this script 'case.sh' echo Enter a command while read cmd do case "$cmd" in list) ls -al ;; freespace) df . ;; quit|Quit) break ;; *) echo "$cmd: No such command" ;; esac done echo "All done" When you run it, the script waits for you to type one of: list freespace quit Quit Try it: modify the example so any command beginning with characters "free"

runs df

Infinite Loops Infinite loop with while while : do ... done : is no-op, always returns success status Must use break or exit inside loop for it to terminate

Forking Remote Shells

Remote Shells Rsh command rsh hostname "commands" Runs commands on remote system Must have .rhosts set up Can specify different login name rsh -l name hostname "commands"

Examples With rsh Check who's logged on rsh spooky "finger" Run several remote commands rsh spooky "uname -a; time" Executes .cshrc on remote system Be sure to set path in .cshrc instead of .login

Access Control with .Rhosts


May get "permission denied" error from rsh Fix this with ~/.rhosts on remote system Example: provide for remote shell from spunky to spooky spunky % rlogin spooky spooky % vi ~/.rhosts (insert "spunky login-name") spooky % chmod 600 ~/.rhosts spooky % logout spunky % rsh spooky uname -a spooky 5.5 sparc SUNW,Ultra-1 May also rlogin without password: security problem!

Remote Shell I/O Standard output sent to local host rsh spooky finger > finger.spooky Standard input sent to remote host cat local-file | rsh spooky lpr -

Return Status Get return status of rsh rsh mayer "uname -a" echo $? Returns 0 if rsh managed to connect to remote host Returns 1 otherwise

o Invalid hostname o Permission denied

Remote Return Status What about exit status of remote command? Have to determine success or failure from stdout or stderr

More Miscellaneous

Temporary Files Use unique names to avoid clashes tempfile=$HOME/Weq_$$ command > $tempfile $$ is PID of current shell Avoids conflict with concurrent executions of script Do not use /tmp!

Wait Command Wait for termination of background job command & pid=$! (other processing) wait $pid Allows overlap of two or more operations


Using exec for File I/O Open file and associate with a number (called a file descriptor) Write files with file descriptors #!/bin/sh exec 4> temp.out # Write a line to standard output, a line to temp.out, # and finally a line to standard error echo "Hello" echo "Test data" >&4 echo "Data to standard error" >&2 You can also read from files using file descriptors First, create data file temp.dat line 1 line 2 line 3 line 4 Now execute the script data.sh shown here: #!/bin/sh # Open file temp.dat using file descriptor 3 exec 3< temp.dat while read line <&3 # Read temp.dat line-by-line do echo Found line: $line done

Using Quotes

Quotes Provide control of collapsing of spaces and translation of variables Try it: run three examples No quotes (variables translated, spaces collapsed) echo Home: $HOME Home: /users/us/freddy Double quotes (no collapsing) echo "Home: $HOME" Home: /users/us/freddy Single quotes (no translation or collapsing) echo 'Home: $HOME' Home: $HOME Try it: single quotes within double quotes echo "Home directory '$HOME' is full..."

Metacharacters Characters with special meaning to shell " ' ` $ * [ ] ? ; > < & ( ) \ Avoid special meaning with quoting echo 'You have $20'


Backslash like single quotes Applies only to next character echo You have \$20

Examples With Quotes Bad command line: grep dog.*cat file Shell tries to expand dot.*cat as file name wildcard Use quotes to avoid translation grep 'dog.*cat' file Single quotes OK in this case because we don't need variable translation

More Examples With Quotes Read name and search file for name read name grep "$name" dbase Single quotes not OK because we need variable translation

Searching for Metacharacters Bad command line: search for dollar sign grep "Gimme.*$20" file Problem: shell translates variable $20 Solution 1: use backslash grep "Gimme.*\$20" file Solution 2: use single quotes grep 'Gimme.*$20' file

Shell Metacharacters There are a number of characters that have special meaning to the shell, which is what makes them metacharacters. Note that meta- is a generic prefix assuming different meanings for different programs. Be careful not to confuse their meanings or confuse

them with metakeys. Common tcsh metacharacters are (! $ ^ & * ( ) ~ [ ] | { } ' " ; < > ? [space] [tab]). The metacharacters are explained further in the man page for tcsh.

Table contains very commonly used metacharacters.

Table: Common Metacharacters

Metacharacter Function

; seperates multiple commands on the same line

http://www.cs.wisc.edu/csl/cs1000/node42.html#table:metachars


? matches any one character in a file name

matches zero or more characters in a file name

& runs the preceding command in the background

nullifies the special meaning of the next metacharacter

As an example,

[bbadger@demo01] (1)$ ls file.*

run on a directory containing the files file, file.c, file.lst, and myfile would list the files file.c and file.lst. However,

[bbadger@demo01] (1)$ ls file.?

run on the same directory would only list file.c because the ? only matches one character, no more, no less. This can save you a great deal of typing time. For example, if there is a file called california_cornish_hens_with_wild_rice and no other files whose names begin with `c', you could view the file without typing the whole name by typing this:

[bbadger@demo01] (1)$ more c*

because the c* matches that long file name.

Filenames containing metacharacters can pose many problems and should never be intentionally created. If you do find that you've created a file with metacharacters, and you would like to remove it, you have three options. You may use wildcards to match

metacharacter, use the to directly enter the filename, or put the command in double quotes (except in the case of double quotes within the file name, these must be captured with one of the first two methods). For example, deleting a file named `*|more' can be accomplished with:

[bbadger@demo01] (1)$ rm ??more

or

[bbadger@demo01] (1)$ rm * |more

or

[bbadger@demo01] (1)$ rm ``*|more''


Shell Variables and Environment Variables

Every UNIX process runs in a specific environment. An environment consists of a table of environment variables, each with an assigned value. When you log in certain login files are executed. They initialize the table holding the environment variables for the process. (Exactly which files run will be made clear later in this chapter.) When this file passes the process to the shell, the table becomes accessible to the shell. When a (parent) process starts up a child process, the child process is given a copy of the parent process' table. Environment variable names are generally given in upper case.

The shell maintains a set of internal variables known as shell variables. These variables cause the shell to work in a particular way. Shell variables are local to the shell in which they are defined; they are not available to the parent or child shells. Shell variable names are generally given in lower case in the C shell family and upper case in the Bourne shell family.

C Shell Family

The C shell family explicitly distinguishes between shell variables and environment variables.

Shell Variables

A shell variable is defined by the set command and deleted by the unset command. The main purpose of your .cshrc file (discussed later in this chapter) is to define such variables for each process. To define a new variable or change the value of one that is already defined, enter:

% set name=value

where name is the variable name, and value is a character string that is the value of the variable. If value is a list of text strings, use parentheses around the list when defining the variable, e.g.,

% set name=(value1 value2 value3)

The set command issued without arguments will display all your shell variables. You cannot check the value of a particular variable by using set name, omitting =value in the command; this will effectively unset the variable.

To delete, or unset, a shell variable, enter:

% unset name

To use a shell variable in a command, preface it with a dollar sign ($), for example $name. This tells the command interpreter that you want the variable's value, not its


name, to be used. You can also use ${name}, which avoids confusion when concatenated with text.

To see the value of a single variable, use the echo command:

% echo $name

If the value is a list, to see the value of the nth string in the list enter:

% echo $name[n]

The square brackets are required, and there is no space between the name and the opening bracket.

To prepend or append a value to an existing shell variable, use the following syntax:

% set name=prepend_value${name}

or

% set name=${name}append_value

[Missing image]Note that when a shell is started up, four important shell variables are automatically initialized to contain the same values as the corresponding environment variables. These are user, term, home and path. If any of these are changed, the corresponding environment variables will also be changed.

Environment Variables

Environment variables are set by the setenv command, and displayed by the printenv or env commands, or by the echo command as individual shell variables are. Some environment variables are set by default (e.g., HOME, PATH).

The formats of the commands are (note the difference between set and setenv):

% setenv [NAME value]% unsetenv NAME

where value is interpreted as a character string. If the string includes blanks (i.e. if it encompasses multiple values), enclose the string in double quotes ("), e.g.,

% setenv NAME "value1 value2 ..."

The current environment variable settings can be displayed using the setenv command with no arguments.

To use an environment variable in a command, preface it with a dollar sign ($), for example $NAME. This tells the command interpreter that you want the variable's value, not its name, to be used. You can also use ${NAME}, which avoids confusion when concatenated with text.


To prepend or append a value to an existing environment variable, use the following syntax:

% setenv NAME "prepend_value${NAME}"

or

% setenv NAME "${NAME}append_value"

If the pre- or appended value is the value of a preexisting environment variable, enclose the variable name in braces, too, e.g.,

% setenv NAME "${NAME}${XYZ_VAR}"

[Missing image]Appending and prepending is commonly used with the PATH variable, and a colon is used as a separator, e.g.,

% setenv PATH "${PATH}:${XYZ_DIR}"

Bourne Shell Family

The Bourne shell family does not really distinguish between shell and environment variables. When a shell starts up, it reads the information in the table of environment variables, defines itself a shell variable for each one, using the same name (also uppercase by convention), and copies the values. From that point on, the shell only refers to its shell variables. If a change is made to a shell variable, it must be explicitly "exported" to the corresponding environment variable in order for any forked subprocesses to see the change. Recall that shell variables are local to the shell in which they were defined.

Shell variables are defined by assignment statements and are unset by the unset command. The format of the assignment statement is:

$ NAME=value[; export NAME][41]

where there are no spaces around the equal sign (=). The unset command format is:

$ unset NAME

where NAME is the variable name, and value is a character string that is the value of the variable. If the string includes blanks (i.e. if it encompasses multiple values), enclose the string in double quotes, e.g.,

$ NAME="value1 value2 ..."

The values of all the current variables may be displayed with the set command.

To use a variable in a command, preface it with a dollar sign ($). This tells the command interpreter that you want the variable's value, not its name, to be used. For example, to see the value of a single variable, enter:

http://www.fnal.gov/docs/UNIX/unix_at_fermilab/htmldoc/rev1997/uatf-62.html#FOOTNOTE-41


$ echo $NAME

You can also use ${NAME}, which avoids confusion when concatenated with text.

To prepend or append a value to an existing environment variable, use the following syntax:

$ NAME=prepend_value$NAME

or

$ NAME=$NAMEappend_value

[Missing image]Appending and prepending is commonly used with the PATH variable, and a colon is used as a separator, e.g.,

$ PATH=${PATH}:${XYZ_DIR}

UNIX Utilities – awk

Definition: awk is a powerful Unix command. It allows the user to manipulate files that are structured as columns of data and strings.

Outline o General structure of awk scripts o Elementary awk programming

Elementary examples o Advanced awk programming

Advanced examples o Important things which will bite you

General structure of awk (Aho, Weinberg, and Kernighan) o awk, oawk, nawk, gawk, mawk

The original version, based on the first edition of The awk Programming Language was called awk

2nd edition of book led to nawk Unices usually ship with three different names for awk: oawk,

nawk, and awk; either oawk=awk or nawk=awk. gawk is the FSF version. mawk is a speedier rewrite which does a partial compilation

o The awk command line is:

awk [program|-f programfile] [flags/variables] [files]

Command line flags


o -f file -- Read the awk script from the specified file rather than the command line

o -F re -- Use the given regular expression re as the field separator rather than the default "white space"

o variable=value -- Initialize the awk variable with the specified

An awk program consists of one or more awk commands separated by either \n or semicolons.

The structure of awk commands o Each awk command consists of a selector and/or an action; both may

not be omitted in the same command. Braces surround the action. o selector [only] -- action is print o {action}[only] -- selector is every line o selector {action} -- perform action on each line where selector is true o Each action may have multiple statements separated from each other

by o semicolons or \n

Line selection o A selector is either zero, one, or two selection criteria; in the latter case

the criteria are separated by commas o A selection criterion may be either an RE or a boolean expression (BE)

which evaluates to true or false o Commands which have no selection criteria are applied to each line of

the input data set o Commands which have one selection criterion are applied to every line

which matches or makes true the criterion depending upon whether the criterion is an RE or a BE

o Commands which have two selection criteria are applied to the first line which matches the first criterion, the next line which matches the second criterion and all the lines between them.

o Unless a prior applied command has a next in it, every selector is tested against every line of the input data set.

Processing o The BEGIN block(s) is(are) run (mawk's -v runs first) o Command line variables are assigned o For each line in the input data set

It is read and NR, NF, $I, etc. are set For each command, its criteria are evaluated If the criteria is true/matches the command is executed

o After the input data set is exhausted, the END block(s) is(are) run

Elementary awk programming o Constants

Strings are enclosed in quotes (")


Numbers are written in the usual decimal way; non-integer values are indicated by including a period (.) in the representation.

REs are delimited by /

Variables o Need not be declared o May contain any type of data, their data type may change over the life

of the program o Are named as any token beginning with a letter and continuing with

letters, digits and underscores o As in C, case matters; since all the built-in variables are all uppercase,

avoid this form. o Some of the commonly used built-in variables are:

NR -- The current line's sequential number NF -- The number of fields in the current line FS -- The input field separator; defaults to whitespace and is

reset by the -F command line parameter Fields

o Each record is separated into fields named $1, $2, etc o $0 is the entire record o NF contains the number of fields in the current line o FS contains the field separator RE; it defaults to the white space RE,

/[<SPACE><TAB>]*/ o Fields may be accessed either by $n or by $var where var contains a

value between 0 and NF print/printf

o print prints each of the values of $1 through $NF separated by OFS then prints a \n onto stdout; the default value of OFS is a blank

o print value value ... prints the value(s) in order and then puts out a \n onto stdout;

o printf(format,value,value,...) prints the value(s) using the format supplied onto stdout, just like C. There is no default \n for each printf so multiples can be used to build a line. There must be as many values in the list as there are item descriptors in format.

o Values in print or printf may be constants, variables, or expressions in any order

Operators - awk has many of the same operators as C, excepting the bit operators. It also adds some text processing operators.

Built-in functions o substr(s,p,l) -- The substring of s starting at p and continuing for l

characters o index(s1,s2) -- The first location of s2 within s1; 0 if not found o length(e) -- The length of e, converted to character string if necessary,

in bytes o sin, cos, tan -- Standard C trig functions o atan2(x,y) -- Standard quadrant oriented arctangent function o exp, log -- Standard C exponential functions o srand(s), rand() -- Random number seed and access functions


Elementary examples and uses o length($0)>72 -- print all of the lines whose length exceeds 72 bytes o {$2="";print} -- remove the second field from each line o {print $2} -- print only the second field of each line o /Ucast/{print $1 "=" $NF} -- for each line which contains the string

'Ucast' print the first variable, an equal sign and the last variable (awk code to create awk code; a common trick)

o BEGIN{FS="/"};NF<4 -- using '/' as a field separator, print only those records with less than four fields; when applied to the output of du, gives a two level summary

o {n++;t+=$4};END{print n " " t} -- when applied to the output of an ls -l command provides a count and total size of the listed files; I use it as part of an alias for dir. Depending on your flavor of UNIX, the $4 may need to be changed to $5.

o $1==prv{ct++;next}{printf("%8d %s",ct,prv);ct=1;pr v=$0} -- prints each unique record with a count of the number of occurrences of it; presumes input is sorted

Advanced awk programming o Program structure (if, for, while, etc.)

if(boolean) statement1 else statement2 if the boolean expression evaluates to true execute statement1, otherwise execute statement 2

for(v=init;boolean;v change) statement Standard C for loop, assigns v the value of init then while the boolean expression is true executes the statement then the v change

for(v in array) statement Assigns to v each of the values of the subscripts of array, not in any particular order, then executes statement

while(boolean) statement While the boolean expression is true, execute the statement

do statement while(boolean) execute statement, evaluate the boolean expression and if true, repeat

statement in any of the above constructs may be either a simple statement or a series of statements enclosed in {}, again like C; a further requirement is that the opening { must be on the line with the beginning keyword (if, for, while, do) either physically or logically via \ .

break -- exit from an enclosing for or while loop continue -- restart the enclosing for or while loop from the top next -- stop processing the current record, read the next record

and begin processing with the first command exit -- terminate all input processing and, if present, execute the

END command

Arrays o There are two types of arrays in awk - standard and generalized o Standard arrays take the usual integer subscripts, starting at 0 and

going up; multidimensional arrays are allowed and behave as expected


o Generalized arrays take any type of variable(s) as subscripts, but the subscript(s) are treated as one long string expression.

o The use of for(a in x) on a generalized array will return all of the valid subscripts in some order, not necessarily the one you wished.

o The subscript separator is called SUBSEP and has a default value of comma (,)

o Elements can be deleted from an array via the delete(array[subscript]) statement

Built-in variables o FILENAME -- The name of the file currently being processed o OFS -- Output Field Separator default ' ' o RS -- Input Record Separator default \n o ORS -- Output Record Separator default \n o FNR -- Current line's number with respect to the current file o OFMT -- Output format for printed numbers default %.6g o RSTART -- The location of the data matched using the match built-in

function o RLENGTH -- The length of the data matched using the match built-in

function

Built-in functions o gsub(re,sub,str) -- replace, in str, each occurrence of the regular

expression re with sub; return the number of substitutions performed o int(expr) -- return the value of expr with all fractional parts removed o match(str,re) -- return the location in str where the regular expression

re occurs and set RSTART and RLENGTH; if re is not found return 0

o split(str,arrname,sep) -- split str into pieces using sep as the separator and assign the pieces in order to the elements from 1 up of arrname; use FS if sep is not given

o sprintf(format,value,value,...) -- write the values, as the format indicates, into a string and return that string

o sub(re,sub,str) -- replace, in str, the first occurrence of the regular expression re with sub; return 1 if successful, 0 otherwise

o system(command) -- pass command to the local operating system to execute and return the exit status code returned by the operating system

o tolower(str) -- return a string similar to str with all capital letters changed to lower case

Other file I/O o print and printf may have > (or >>) filename or | command

appended and the output will be sent to the named file or command; once a file is opened, it remains open until explicitly closed

o getline var < filename will read the next line from filename into var. Again, once a file is opened, it remains so until it is explicitly closed

o close(filename) explicitly closes the file named by the filename expression


Writing your own functions o A function begins with a function header of the form:

function name(argument(s), localvar(s)) { o and ends with the matching } o The value of the function is returned via a statement of the form:

return value o Functions do not have to return a value and the value returned by a

function (either built-in or written locally) may be ignored by just placing the function with its arguments as a whole, separate statement

o The local variables indicated in the localvars of the heading replace the global variables of the same name until the function completes, at which time the globals are restored

o Functions may have side effects such as updating global variables, doing I/O or running other functions with side effects; beware the frumious bandersnatch


UNIT – III


Introduction of System

The system is written in a high-level language making it easy to read, understand change , and move to other machines.

It has a simple user interface that has the power to provide the services that programs.

It uses a hierarchical file system that allows easy maintenance and efficient implementation.

It uses a consistent format for files , the byte stream , making application programs easier to write.

It provides a simple, consistent interface to peripheral devices. It is a multi-user , multiprocessing system; each user can execute several

process simultaneously. It hides the machine architecture from the user , making it easier to write

programs that run on different hardware implementation.

System Structure


User Perpesctive

File System

The UNIX File System is characterized by : A hierarchical Structure Consistent treatment of file data The ability to create and delete files Dynamic growth of files The protection of file data The treatment of peripheral devices( such as terminals and tape units) as files.

Example :- Program to copy a File

A user at a terminal invokes the program by typing :-copy oldfile newfile

This particular command takes three arguments such as” copy = argv[0]

oldfile = argv[1]newfile = argv[2]

File System

Processing Environment

Building Block Primitives


We can copy the contents of one file into another file with the help of command as well as program using C Language , Shell

Program

#include <fcntl.h>char buffer[2048];int version =1;main(argv,argv)

int argc;char *argv[];

{int fold, fnew;if (argc!=3){

printf(“need” ! Arguments for copy program \n”)exit(1);

}fold =open(argv 1], O_RDONLY); //Open source file read onlyif(fold== -1){

printf(“cannot open file %s\n” ,argc[1]);exit(1);

}

fdnew = create(ar , v[2] ,0666) /* Create target file rw for allif (fnew== -1){

printf(“Cannot create file %s \n” ,argv[2]);exit(1);

}copy(fdold , fdnew);exit(0);

} //Close Main Functioncopy(old,new)

int old,new;{

int count;while((count =read(old,buffer,sizeof(buffer)))>0)

write(new,buffer,count);}

Processing Environment

Program that Create a New Process to copy Filesmain(argc,argv)

int argc;char *argv[];


{/* assume 2 args : source file and target file */if (fork()==0)exec(“copy” ,”copy”, argv[1],argv[2] ,0);wait((int *) 0);printf(“Copy Done \n”);

}

Building Block Primitives

The UNIX System is to provide operating system primitives that enable users to write small ,modular programs that can be used as building blocks to build more complex programs.

The First building block is “Redirect I/O” Process conventionally have access to three files:-

They read from their standard input. Write to their standard output File. Write error messages to their standard error file.

Processes executing at a terminal typically use the terminal for these files ,

but each may be “redirected” independently.

E X A M P L E S

ls : lists all files is the current directory on the standard output.

ls>output : Redirects the standard output to the file called “output” in the current directory.

$cat output //for showing the result mail mjb < letter :

Opens the file “letter” for its standard input and mails its contents to the user named “mjb”.

P I P E S

The second Building Block Primitive is the “PIPE”. Example:-

The program grep searches a set of files for a given pattern$grep main a.c b.c c.c

Searches the three files a.c ,b.c and c,c for lines containing the string “main” and prints the lines that it finds onto standard output. Output may be:

a.c : main(argc,argv)


b.c :/* Here is the main loop in the program */c.c : main()

The program wc with the option –l counts the number of lines in the standard input file. The command line

$grep main a.c b.c c.c | wc –lcounts the no. of lines in the file that contain the string “main” the output from

grep is “piped” directly into the wc command.Output: 3

Operating System Services

Controlling the execution of processes by allowing their creation , termination or suspension and communication.

Scheduling process fairly for execution on the CPU. Processes share the CPU in time-shared manners. The CPU executes a process , the kernel suspends it when its time quantum elapses and kernel schedules another process to execute . The kernel later rescheduled the suspended process.

Allocating Main Memory for an executing process. If the system runs low on free memory , the kernel frees memory by writing a process temporarily to secondary memory called a swap device

If the kernel writes entire processes to a swap device , the implementation of the UNIX system is called a swapping system.

If it write pages of memory to a swap device , it is called a paging system. Allocating secondary memory for efficient storage and retrieval of user data. Allowing processes controlled access to peripheral devices such

as terminals , tape devices , disk drives and network devices.

Assumptions About Hardware

The execution of user processes on UNIX systems is divide into two levels: User Level Kernel Level

When a process executes a system call , the execution mode of the process changes from user mode to kernel mode.

Operating system executes and attempts to service the user request, returning an error code if it fails.

Difference Between User Mode & Kernel Mode

Processes in user mode can access their own instructions and data but not kernel instructions and data . Processes in kernel mode ,can access kernel and user address.Example : The virtual address space of a process may be divided between that are accessible only in kernel mode and addressed that are a accessible in either mode.

Some machine instructions are privileged and result in an error when executed in user mode. Processing execution in user mode should not have this capability. Put simply , the hardware views the world in terms of kernel mode and user mode and does not distinguish among the many users executing programs in those modes.


Figure :- Multiple Processes and Modes of Execution

The kernel and buffer cache Architecture of the UNIX Operating System

The UNIX system supports the illusions that the file system has “Places” and that processes have “life”.

The two entities files and processes are the two central concepts in the UNIX system model.

A file is also related to another by being part of a single hierarchical structure .

Moreover you too can be “placed” at a specific locations in this hierarchy and you can also “move” form one place to another.

The process , which is the given to a file when it is executed as a program. A process is simply the “time image” of an executable file. Files , Processes also belong to a separate hierarchical tree structure.


The three levels of system :User LevelKernel LevelHardware

The system call and library interface represent the border between user programs.

The set of system calls into those that interact with the file subsystem and those that interact with the process control subsystem.

The file subsystem manages files, allocating file space , administrating free space , controlling access to files and retrieving data for users.

Process interact with the file subsystem via a specific set of system calls such as open , close , read , write stat , chown and chmod.

The file subsystem accesses file data using a buffering mechanism that regulates data flow between the kernel and secondary storage devices.

The buffering mechanism interacts with block I/O device drivers to initiate data transfer to and from the kernel.

Device drivers are the kernel module that control the operation of peripheral devices.

The file subsystem also interacts directly with “raw” I/O device drivers without the intervention of a buffering mechanism. Raw devices sometimes called character devices, include all devices that are not block devices.


The process control subsystem is responsible for process synchronization , inter communication , memory management and process scheduling.

The file subsystem and the process control subsystem interact when loading a file into memory for execution .

Some of system call for controlling processes:-

The memory management module controls the allocating of memory. If at any time the system does not have enough physical memory for all processes , the kernel moves them between main memory and secondary memory so that all processes get a fair chance to execute (Swapping , Paging and Demand Paging for managing memory).

The scheduler module allocates the CPU to process. It schedules them to run in turn until they voluntarily relinquish the CPU while awaiting a resource or until the kernel preempts them when their recent run time exceeds a time quantum.

The scheduler then choose the highest priority eligible process to run , the original process will run again it is the highest priority eligible process available.

Finally , the H/W control is responsible of handling interrupts and for communicating with the machine.


An overview of the File Subsystem

The internal representation of a file is given by an inode , which contains a description of the disk layout of the file data and other information such as the file owner , access permission and access times.The term inode is a contraction of the term index node and is commonly used in literature on the UNIX system. Every file has one node but it may have several names , all of which map into the inode . Each name is called link.

The kernel contains two other data structure , the file table and the user file description table .


User File Descriptors Table

1. The User File Descriptor Table identifies all open files for a process.2. The Kernel returns a file descriptor for the open and create system calls ,

which is an index into the user file descriptor table.3. When executing read and write system calls , the kernel uses the file descriptor

to access the user file descriptor table, follows pointers to the file table and inode table entries and from the inode finds the data in the file.

The UNIX system keeps regular files and directories on block devices such as tapes or disks . B’coz of the difference in access time between the two , few , if any UNIX system installation use tapes for their file systems. For simplicity , however the ensuring text assumes the use of disks. An installation may have several physical disk units each containing one or more file systems. The kernel deals on a logical level with file system Rather than with disks , treating each one as a logical device identified by a logical device number .The conversion between logical device (File System) address and physical device (disk) addresses is done by the disk driver. A file system consists of a sequence of logical blocks , each containing 512 , 1024 , 2048 or any convenient multiple of 512 bytes , depending on the system implementation. The size of a logical block is homogeneous within a file system but may vary between different file system in a system configuration.


File Table

1. The file table is a global kernel structure but the user file descriptor table is allocated per process.

2. When a process opens or create a file, the kernel allocates an entry form each table , corresponding to file’s inode.

3. The file table keeps track of the byte offset in the file where the user’s next read or write will start and the access rights , allowed to the opening process.

Inode Table

Inode:- A structure maintained in a special area of the disk for storing the attribute of a file. This table contains for every file its permission, ownership details, time stamps and number of links .However ,it doesn’t contain the filename.inode number :- A number identifying an inode for a file which is unique in a single file system. It is displayed with the –i option of ls command.The inode contains the following attributes of a file :-

• File Type• File permission• Number of Links• The UID of the owner• The GID of the group owner• File size in bytes• Date and Time of lat change of the inode • An array of pointers that keep track of all disk blocks used by

the file.

Process

A process is the execution of a program and consists of a pattern of bytes that the CPU interrupts as machine instruction (Called “text” ) , data and stack.

A user compiles the source code of a program to create an executable file , which consists of several parts:-

1. A set of “headers” that describe the attributes of the file.2. The Program text.3. A machine language representation of data that has initial values when the

program starts execution and an indication of how much space the kernel should allocate for uninitialized data , called bss2 .

4. Other sections , such as symbol table information.


The kernel loads an executable file into memory during an exec system call and the loaded process consists of at least three parts, called regions: text , data and the stack.

The text and data regions correspond to the text and data-bss sections of the executable file , but the stack region is automatically created and its size is dynamically adjusted by the kernel at run time .

The stack consists of logical stack frames that are pushed when calling a function and popped when returning ; a special register called the stack pointer indicates the current stack depth . A stack frame contains the parameters to a function , its local variables and the data necessary to recover the previous stack frame , including the value of the program counter and stack pointer of the function call.

“It maintain the Process table”

The process table entry and the u area contain control and status information about he process. The u area is an extension of the Process Table entry (UNIT –V).Fields in the process table :-


• A state fields• Identifies indicating the user who owns the process(user IDs

or UIDs) . • An event descriptor set when a process is suspended ( Sleep

State)The u area contains information describing the process that needs to beaccessible only when the process is executing. The important field are:-

• A pointer to the process table slot of the currently executing process.

• Parameters of the current system call , return values and error codes.

• File descriptors for all open files.• Internal I/O parameters.• Current directory and current root.• Process and the file size limits.

The kernel can directly access fields of the u area of the executingprocess but not of the u area of other processes.


Buffer Cache

B U F F E R - H E A D E R


The buffer header contains a device number field and a block number field that specify the file system and block number of the data on disk and uniquely identify the buffer.


The device number is the logical file system number, not a physical device (disk) unit number.

The buffer header also contains a pointer to a data array for the buffer , whose size must be at least as big as the size of a disk block and a status field that summarizes the current status of the buffer.

The status of a buffer is a combination of the following conditions……………..

The buffer is currently locked . The buffer contains valid data. The kernel must write the buffer contents to disk before

reassigning the buffer ; this condition is known as “Delayed- Write”.

The kernel is currently reading or writing the contents of the buffer to disk.

A process is currently waiting for the buffer to become free.

Structure of The Buffer Pool

The kernel caches data in the buffer pool according to a least recently used algorithm : after it allocates a buffer to a disk block , it cannot use the buffer for another block until all other buffers have been used more recently.

The kernel maintains a free list of buffers that preserves the least recently used order. The free list is a doubly linked circular list of buffers with a dummy buffer header that marks its beginning and end.

Every buffer is put on the free list when the system is booted.


The kernel takes a buffer from the free list when it wants any free buffer , but it can take a buffer from the middle of the free list if it identifies a particular block in the buffer from the middle of the free list if it identifies a particular block in the buffer pool.


There are five scenarios , the kernel may follow in getblk to allocate a buffer for a disk block…

The kernel finds the block on its hash queue and its buffer is free. The kernel cannot find the block on the hash queue , so it allocates a buffer

from the free list. The kernel cannot find the block on the hash queue and in attempting to

allocate a buffer from the free list , finds a buffer on the free list that has marked “delayed write”. The kernel must write the “delayed write” buffer to disk and allocate another buffer.

The kernel cannot find the block on the hash queue and the free list of buffers is empty.

The kernel finds the block on the hash queue , but its buffer is currently busy.



Algorithm for Releasing a

algorithm brelse input: locked bufferoutput: none{

wakeup all pros: event , waiting for any buffer to become free;

wakeup all pros: event , waiting for this buffer to become free;

raise processor execution level to block interrupts;if (buffer contents valid and buffer not old)

enquence buffer at end of free listelse

enquence buffer at beginning of free listlower processor execution level to allow interrupts;unlock(buffer);





Reading and Writing Disk Blocks

algorithm bread /* Block Read */Input : file system block numberOutput : buffer containing data{

get buffer for block( algorithm getblk );if ( buffer data valid )

return buffer;initiate disk read;sleep (event disk read complete);return (buffer);

}

Algorithm for Block Read Ahead

algorithm breada /* Block Read and Read Ahead*/Input : (1) file system block number for immediate read (2) file system block number for asynchronous read Output : buffer containing data for immediate read{ if ( first block not in cache )


{ get buffer for first block (algorithm getblk); if (buffer data not read; }

if (second block not in cache){get buffer for second block (algorithm getblk);

if (buffer data valid)release buffer (algorithm brelse);

elseinitiate disk read;

}if ( first block was originally in cache ){

read first block (algorithm read);return buffer;

}sleep (event first buffer contains valid data);return buffer;

}

Algorithm for Writing a Disk Block

algorithm bwrite /* Block Write */input : bufferoutput : none{

initiate disk drive;if (I/O synchronous){

sleep( event I/O complete);release buffer (algorithm brelse);

}else if (buffer marked for delayed write)

mark buffer to put at head of free list;}

Advantages and Disadvantages of the Buffer Cache

The use of buffers allows uniform disk access , because the kernel does not need to know the reason for the I/O. Instead , it copies data to and from buffers , regardless of whether the data is part of a file , an inode or a super block.

The system places no data alignment restrictions on user processes doing I/O , because the kernel aligns data internally. Hardware implementation frequently require a particular alignment of data for disk I/O , such as aligning the data on a two-byte boundary or on a four-byte boundary in memory.


The kernel eliminates the need for special alignment of user buffers , making user programs simpler and more portable.

Use of the buffer cache can reduce the amount of disk traffic , thereby increasing overall system throughput and decreasing response time.Process reading from the file system may find data blocks in the cache and the need for disk I/O.

The kernel frequently uses “delayed write” to avoid unnecessary disk writes , leaving the block in the buffer cache and hoping for a cache hit on the block.

If too much memory is used for buffers , the system may slow down because of excessive process swapping or paging.

The buffer algorithm help insure file system integrity , because they maintain a common , single image of disk blocks contained in the cache.

If two processes simultaneously attempt to manipulate one disk block , the buffer algorithm s serialize their access , preventing data corruption.

Reduction of disk traffic is important for good throughput and response time , but the cache strategy also introduces several disadvantages.

Use of the buffer cache requires an extra data copy when reading and writing to and from user processes. A process writing data copies the data into the kernel and the kernel copies the data to disk ; a process reading data has the data read from disk into the kernel and from the kernel to the user process.


UNIT-IV


INODES

Inodes exist in a static form on disk and the kernel reads them into an in-core inode to manipulate them. Disk inodes consist of the following fields:

File Owner identifier File Type File Access permissions File Access times , giving the time the file was last modified Number of links Disk addresses of data in a file File Size

IN-CORES

The in-core copy of the inode contains the following fields in addition to the fields of the disk inode :-

Status The inode is Locked A process is waiting for the inode to become unlocked The in-core representation of the inode The in-core representation of the file The file is a mount point

Logical device Inode number Pointer to other in-core inodes A reference count

Accessing Inodes

algorithm iget input : File System inode numberoutput : locked inode { while (not cache)

{ if(inode in inode cache){

sleep(event inode becomes unlocked);;continue;

}if (inode on inode free list)

remove from free list;increament inode reference count;return(inode);

}if (no inodes on free list)

return (error);remove new inode from the list;


reset inode number and file system;remove inode from old hash queue, place on new one;read inode from disk(algorithm bread);initializing inode;return(inode);

}

Formula

block number =((inode number -1) /number of inodes per block) + Start block of inode list

byte offset = ((inode number -1) modulo(number of inodes per block))*size of disk inode

Releasing Inodesalgorithm iput input : pointer to in-core inode output : none{

lock inode if not already locked;decreament inode reference count;if(reference count ==0){

if(inode link count ==0){

free disk blocks for file;set file type to 0;free inode;

}if (file accessed or inode changed or file changed)

update disk inode;put inode on free list;

}release inode lock;

}




D I R E C T O R I E S

Byte Offset in directory Inode Number(2 byte)

File Name

0 83 .

16 2 ..

32 1798 init

48 1276 fsck

64 85 clri

80 1268 motd

96 1799 mount

112 88 mknod

128 2114 passwd

144 1717 umount

160 1851 checklist

CONVERSION OF A PATH NAME TO AN INODE

algorithm namei /* convert path name to inode */input : path nameoutput : locked inode {

if (path name starts from root)working inode = root inode (algorithm iget);

elseworking inode = current directory inode (algorithm iget);

while (there is more path name){

read next path name component from input;verify that working inode is of directory,access permissions OK;

if ( working inode is of root and component is “..”)continue;

read directory(working inode) by repeated use of algorithms bmap,bread,and brelse;If(component matches an entry in directory (working inode))


{get inode number for matched component;release working inode (algorithm iput);working inode = inode of matched component(algorithm

iget);}else /*component not in directory */

rerurn (no inode);}return (working inode);} SUPER BLOCK

The super block consists of the following fields……. The size of the file system The number of free block available on the file system The index of the next free block in the free block list The size of the inode list The number of free inodes in the file system A list of free inodes in the file system The index of the next inode in the free inode list Lock fields for the free block and free inode lists A flag indicating that the super block has been

modified.

INODE ASSIGNMENT TO NEW FILE

The kernel uses algorithm iget to allocate a known inode , one whose (file System) inode number was previously determined.

In algorithm namei for instance , the kernel determines the inode number by matching a path name component to a name component to a name in a directory.

Another algorithm ialloc assigns a disk inode to a newly created file.

algorithm ialloc /* allocate inode */Input : file systemOutput : locked inode {

while( not done) {

if( super block locked){

sleep( event super block becomes free)continue; /*while loop*/

}if (inode list in super block is empty){

lock super block;


get remembered inode for free inode search;search disk for free inodes until super block full , oron more free inodes(algorithm bread and brelse);unlock super block;wake up( event super block becomes free);if ( no free inodes found on disk)

return (no inode);set remembered inode for next free inode search; } /* there are inodes in super block inode list */get inode number from super block inode list;get inode (algorithm iget);if (inode not free after all){

write inode to disk;release inode(algorithm iput);continue;

}initialize inode; /* inode is free */ write inode to disk;decreament file system free inode count;return(inode);}



ALLOCATION OF DISK BLOCKS

When a process writes data to a file, the kernel must allocate disk blocks from the file system for direct data blocks and sometimes, for indirect blocks.The file system super block contains an array that is used to cache the number of free disk blocks in the files system. The utility program mkfs ( make file system) organizes the data blocks of a file system in a linked list, such that each link of the list is a disk block that contains an array of free disk block numbers and one array entry is the number of the next block of the linked list.


algorithm alloc /* File System Block Allocation */input : file system numberoutput : buffer for new block{

while ( super block locked )sleep ( event super block not locked );remove block from super free list;if ( removed last block from free list ){lock super block;read block just taken free list( algorihtm bread)copy block numbers in block into super block;release block buffer ( algorithm brelse );unlock super block;wake up processes ( event super block not locked);}get buffer for block removed from super block list (algorithm getblk);zero buffer contents;decrement total count of free blocks;mark super block modified;return buffer;

}


System calls for the File System

OPEN READ WRITE CLOSE FILE AND RECORD LOCKING FILE CREATION CREATION OF SPECIAL FILES CHANGE DIRECTORY AND CHANGE ROOT CHANGE OWNER AND CHANGE MODE STAT AND FSTAT PIPES MOUNTING AND UNMOUNTING FILE SYSTEM LINK UNLINK

OPEN

The open system call is the first step a process must take to access The data in a file. The syntax for the open system call is:

fd = open(pathname, flags, modes);

Where pathname is a file name, flags indicate the type of open( such as for reading or writing), and modes give the file permissions if the file is being created. The open system call return an integer Called the user file descriptor. Other operation , such as reading , writing, seeking, duplicating the file descriptor, setting file I/O Parameters, determining file status, and closing the file, use the file Descriptor that the open system call returns.

algorithm openinputs: file name

type of openfile permission (for creation type of open)

output: file descriptor{

convert file name to inode(algorithm namei);if (file does not exist or not permitted access);

return (error);allocate file table entry for inode, initialize count, offset;allocate user file descriptor entry, set pointer to file table entry;if (type of open specifies truncate file)

free all file blocks (algorithm free);unlock(inode)return(user file descriptor);

}


fd1 = open(“/etc/passwd”, O_RDONLY);O_RDONLY : OPEN IN READ ONLY MODE

fd2 = open(“local”, O_RDWR);O_RDWR : OPEN READ AND WRITE

MODEfd3 = open(“/etc/passwd”, O_WRONLY);

O_WRONLY : OPEN WRITE ONLY MODE

READ

The Syntax of the read system call is:

number = read(fd, buffer, count)

Where fd is the file descriptor returned by open, buffer is the address of a data structure in the user process that will contain the read data on successful completion of the call, count is the number of byte the user wants to read, and number if the number of bytes actually read.

algorithm readinputs: user file descriptor

address of buffer in user processnumber of bytes to read

output: count of bytes copied into user space{

get file table entry from user file descriptor;check file accessibility;set parameters in a u area for user address, byte count, I/O to user;get inode from file table;lock inode;set byte offset in u area from file table offset;while (count not satisfied){

convert file offset to disk block (algorithm )calculate offset into block, number of bytes to read;break;read block ( algorithm breada if with read ahead, algorithm

bread otherwise);copy data from system buffer to user address;update u area fields for file byte offset, read count, address to write into user space;release buffer;}unlock inode;update file table offset for next read;return (total number of bytes read);


}

Mode

count

offset

address

flag

Indicates read or write

Count of bytes to read or write

Byte offset in file

Target address to copy data, in user or kernel memory

Indicates if address is in user or kernel memory

#include <fctrl.h>main(){int fd;char lilbuf[20], bigbuf[1024];

fd = open(“etc/passwd”, O_RDONLY);read(fd, lilbuf, 20);read(fd, bigbuf, 1024);read(fd, lilbuf, 20);}

It starts reading at :

Byte 20 in the fileByte 20 + 1024 = 1044 in the fileByte 1044 + 20 = 1064 on the file, finding that value in the offset field in the file table for the descriptor

WRITE

The Syntax of the write system call is:

number = write(fd, buffer, count)

Where fd is the file descriptor returned by open, buffer is the address of a data structure in the user process that will contain the read data on successful completion of the call, count is the number of byte the user wants to read, and number if the number of bytes actually read. OR “Same as read System Call” .

#include <fctrl.h>/* process A */main(){

int fd;char buf[512];fd = open(“etc/passwd”, O_RDONLY);read(fd, buf, sizeof(buf)); /*read1*/read(fd, buf, sizeof(buf)); /*read2*/


}

#include <fctrl.h>/* process B */main(){

int fd ,i;char buf[512];for (i=0;i<sizeof(buf); i++)

buf[i] = ‘a’;

fd = open(“etc/passwd”, O_WRONLY);write(fd, buf, sizeof(buf)); /*write1*/write(fd, buf, sizeof(buf)); /*write2*/

}

FILE AND RECORD LOCKING

The original UNIX system developed by Thompson and Ritchie did not have an internal mechanism by which a process could insure exclusive access to a file. A locking mechanism was considered unnecessary because, as Ritchie notes, “ We are not faced with large, single-file databases maintained by independent processes”.File locking is the capability to prevent other processes from reading or writing any part of an entire file and record locking is the capability to prevent other processes from reading or writing particularrecords.

CLOSE

A process closes an open file when it no longer wants to access it. The syntax for the close system callis :

close (fd);

A process closes an open file when it no longer wants to access it. The syntax for the close system call is :

close (fd);

FILE CREATION

The open system call gives a process access to an existing file, but the creat system call creates a new file in the system. The syntax : for the creat system call is:

fd = creat (pathname, modes);

Where the variables pathname, modes, and fd mean the same as they do in the open System call.


algorithm creat input: file name

permission settingsoutput: file descriptor{

get inode for the file name(algorithm namai)if (file already exists){

release inode (algorithm iput);return (error);

}} else /* file does not exist yet */

{assign free inode from file system( algorithm ialloc);create new directory entry in parent directory:inlcude new file name and newly assigned inode numeber;

}allocate file table entry for inode, initialize count;if (file did exist at time of create)free all file block (algorithm free);unlock (inode);return (user file descriptor);

}

Creation of Special Files

The system call mknod creates special files in the system, including named pipes, device files, and directories. It is similar to create in that the kernel allocates an inode for the file. The syntax of the mknod system call is:

mknod(pathname, type and permissions, dev)

Where pathname is the name of the inode to be created, type and permission give the inode type (directory, for example) and access permission for the new file to be created, and dev specifies the major device numbers for block and character special files.

algorithm make new node

inputs : node (file name)file typepermissionsmajor, minor device number (for block, character special files)

output : none


{if (new node not named pipe and user not super user)

return (error);get inode of parent of new node (algorithm namei);if (new node already exists){

release parent inode(algorithm iput);return(error);

}assign free inode from the file system for new node(algorithm ialloc);create new directory entry in parent directory :include new node name

and newly assigned inode number;release parent directory inode (algorithm input);If (new node is block or character special file );

write major, minor numbers into inode structure ;release new node( algorithm input);}

CHANGE DIRECTORY AND CHANGE ROOT

The algorithm chdir(figure below) changes the current directory of a process. The syntax for the chdir system call is

chdir(pathname)Where pathname is the directory that becomes the new current directory of the process. The kernel parse the name of the name of the target directory using algorithm namei and checks that the target file is a directory and the process owner has access permission to the directory.

Algorithm change directory

Input: new directory nameOutput: none{

get inode for new directory name(algorithm namei);if (inode not that of directory or process not permitted access to

file){release inode( algorithm iput);return(error);}unlock inode;release “old” current directory inode (algorithm iput);place new inode into current directory slot in u area;

}


CHANGE ROOT

Syntax : chroot(pathname)

CHANGE OWNER

Syntax : chown(pathname, owner, group)

CHANGE MODE

Syntax : chmod(pathname, mode)

STAT AND FSTAT

The system call stat and fstat allow processes to query the status of files, returning information such as the file type, file owner, access permissions, file size, number of links , inode number and file access time.Syntax:-

stat(pathname, statbuffer);fstat(fd, statbuffer);

Where pathname is a file name, fd is a file descriptor returned by a Previous open call, and statbuffer is the address of a data structure in the user process that will contain the status information of the file on completion of the call. The system calls simply write the fields of the inode into statbuffer.

PIPE

Pipes allow transfer of data between processes in a first-in-first-out manner( FIFO), and they allow synchronization of process execution.

Their implementation allows processes to communicate even though they do not know what processes are on the other end of the pipe.

The traditional of pipes uses the file system for data storage. There are two kinds of pipes: named pipes and unnamed pipes, which are

identical except for the way that a process initially accesses them. Processes use the open system call for named pipes, but the pipe system call

to create an unnamed pipe. Afterwards, processes use the regular system calls for files, such as read, write and close when manipulating pipes.

Only related processes, descendants of a process that issued the pipe call, can share access to unnamed pipes.


Pipes System Call

Syntax :pipe(fdptr)

Where fdptr is the file descriptor pointer to an integer array that will contain the two file descriptors for reading and writing the pipe. Because the kernel implements pipes in the file system and because a pipe does not exist before its use. The kernel must assign an inode for it on creation. It also allocates a pair of user file descriptors and corresponding file table entries for the pipe: One file descriptor for reading from the pipe and the other for writing to the pipe.

algorithm pipe

input : noneoutput : read file descriptor

write file descriptor{

assign new inode from pipe device( algorithm ialloc);allocate file table entry for reading, another for writing;initialize file table entries to point to new inode;allocate user file descriptor for reading, another for writing,

initialize to point to respective file table entries;set inode reference count to 2;


initialize count of inode readers, writer to 1;}The kernel assign an inode for a pipe from a file system designated the pipe device using algorithm ialloc. A pipe device is just a file system from which the kernel can assign inodes and data blocks for pipes.

Opening a Named Pipe

A named pipe is a file whose semantics are the same as those of an unnamed pipe, except that it has a directory entry and is accessed by a path name.

Processes open named pipes in the same way that they open regular files and, hence, processes that are not closely related can communicate.

Named pipes permanently exist in the file system hierarchy, but unnamed pipes are transient. When all processes finish using the kernel reclaims its inode.

The algorithm for reading a named pipe is identical to the algorithm for opening a regular file. However before completing the system call the kernel increments the read or write counts in the inode, indicating the number of processes that have the named pipe open for reading or writing.

If a process opens a named pipe for reading and a writing process exists, the open call completes.

Reading and Writing Pipes

A pipe should be viewed as if processes write into one end of the pipe and read from the other end.

Processes access data from a pipe in FIFO manner, meaning that the order that data is written into a pipe is the order that it is read from the pipe.

The number of processes reading for a pipe do not necessarily equal the number of processes writing the pipe; if the number of readers or writers is greater than 1, they must coordinate use of the pipe with other mechanisms.

The kernel accesses the data for a pipe exactly as it accesses data for a regular file. It stores data on the pipe device and assigns blocks to the pipe as needed during writing calls.

The difference between storage allocation for a pipe and a regular file is that uses only direct blocks of the inode for greater efficiency, although this place a limit on how much data a pipe can hold at a time.






Closing Pipes

When closing a pipe , a process follows the same procedure it would follow for closing a regular file, except that the kernel does special processing before releasing the pipe’s inode.

The kernel decrements the number of pipe readers or writers, according to the type of the file descriptor.

Syntax:-

pipe (fdptr) Where fdptr is the file descriptor pointer to an integer array that will contain the two file descriptors for reading and writing the pipe. Because the kernel implements pipes in the file system and because a pipe does not exist before its use. The kernel must assign an inode for it on creation. It also allocates a pair of user file descriptors and corresponding file table entries for the pipe: One file descriptor for reading from the pipe and the other for writing to the pipe.

Mounting & Unmounting File System

Syntax for the mount system call is :mount (special pathname, directory pathname , options);


Where special pathname is the name of the device special file of the disk section containing the file system to be mounted, directory pathname is the directory in the existing hierarchy where the file system will be monted, options indicates whether the file system should be monuted “read only”.

Eg:- mount (“/dev/dsk1” , “/usr” , o)


UNIT V

Contents

Process States and Transition Layout of System Memory The Context of a Process Manipulation of Process Address Space Sleep Process Creation/Termination The USER ID of a Process Changing the size of a process The Shell


Process States and Transitions

PROCESS

A process is the execution of a program and consists of a pattern of bytes that the CPU interprets as machine instructions (called “text”) ,data and stack.

Many of process appear to execute simultaneously as the kernel schedules them for execution and several processes any be instances of the program.

A process executes by following a strict sequence of instructions that is self-contained and does not jump to that of another process process; it reads and writes its data and stack sections, but it cannot read or write the data and stack of other processes.

Process communicate with other processes and with the rest of the world via system calls.

State


Transition

Node Edge A state transition diagram is a directed graph whose nodes represent the states

a process can enter and whose edges represent the events that cause a process to move from state to another.

They are legal between two states if there exists an edge from the first state to the second.

It may emanate from a state, but a process will follow one and only one transition depending on the system event that a occurs.

The following list contains the complete set of process states:-◦ The process is executing in user mode. ◦ The process is executing in kernel mode. ◦ The process is not executing but is ready to run as soon as the kernel

schedules it..◦ The process is sleeping and resides in main memory.◦ The process is ready to run, but the swapper must swap the process

into memory before the kernel can schedule it to execute.◦ The process is sleeping, and the swapper has swapped the process to

secondary storage to make room for other processes in main memory.◦ The process is returning from the kernel to user mode, but the kernel

preempts it and does a context switch to schedule another process. ◦ The process is newly created and is in a transition state; the process

exists, but it is not ready to run , nor is it sleeping. This state is the start state for all processes except process 0.

◦ The process executed the exit system call and is in zombie state . The process no longer exits, but it leaves a record containing an exit code


and some timing statistics for its parent process to collect. The zombie state is the final state of a process.

Process States

1. User Running (User Mode) 2. Kernel Running (Kernel Mode)3. Ready to Run in memory4. Asleep in memory5. Ready to run , Swapped6. Sleep , Swapped7. Preempted8. Created9. Zombie (Exit)

Two kernel data structures describe the state of a process: ◦ The process table entry ◦ u area.

The process table entry contains fields that must always be accessible to the Kernel, but the u area contains fields that need to be accessible onlyto the running process.The fields in the process table are the following :

1. The state field identifies the process state.2. The process table entry contains fields that allow the kernel to locate

the process and its u area in main memory or in secondary storage. The


kernel uses the information to do a context switch to the process when the process moves from State “ready to run in memory” to the state “kernel running” or from the state “preempted to the state “user running”.

3. UID , It determine various process privileges.4. PID , specify the relationship of processes to each other.5. The process table entry contains an event descriptor when the process

is in the “sleep” state.6. Scheduling parameters allow the kernel to determine the order in

which processes move to the states “kernel running” and “user running”.

7. A signal field enumerates the signals sent to a process but not yet handled.

8. Various timers give process execution time and kernel resource utilization, used for process accounting and for the calculation of process scheduling priority.

The u area contains the following fields :

A pointer to the process table identifies the entry that corresponds to the u area.

The real and effective UIDs determine various privileges allowed the process such as file access rights.

Timer fields record the time the process spent executing in user mode and in kernel mode.

An array indicates how the process wishes to react to signals. The control terminal fields identifies the “login terminal” associated with the

process, if one exists. An error fields records errors encountered during a system call. A return value field contains the result of system calls. I/O parameters Current working directory and current root describe the file system

environment of the process. The user file descriptor table records the files the process has open. Limit fields restrict the size of a process and the size of a file it can write. A permission modes field masks mode setting on files the process creats.

Layout of System Memory

The logical memory The physical memory A process on the UNIX system consists of 3 logical sections :

Text - Contains the set of instructions the machine execute for the process; address in the text section include text address (For branch instructions or subroutine calls)

Data (For access to global variables) Stack (For access to data structures local to a subroutine)

The virtual memory


The subsystem of the kernel and the hardware that co-operate to translate virtual to physical address comprise the memory management subsystem.

1. Regions

The system V kernel divides the virtual address space of a process into logical regions.

A regions is a contiguous area of the virtual address space of a process that can be treated as a distinct object to be shared or protected.

Text , data and stack usually from separate regions of a process . Several process can share a region. Several processes may execute the same program and it is natural that they

share on copy of the text region. Similarly, several processes may cooperate to share a common shared-memory region.

The kernel contains a region table and allocates an entry from the table for each active region in the system.

The concept of the region is independent of the memory management policies implemented by the Operating System

2. Pages and Page Tables


In a memory management architecture based on pages, the memory management hardware divides physical memory into a set of equal-sized blocks called pages.

Typical page size range from 512 bytes to 4K Bytes and are defined by the hardware.

Every addressable location in memory in contained in a page and consequently, every memory location can be addressed by a

( page number, byte offset in page ) pair.

For Example: If a machine has 232 bytes of physical memory and a page size of 1K bytes , it has 222 pages of physical memory ; every 32-bit address can be treated as a pair consisting of a 22-bit page number and a 10-bit offset into the page.


3. Layout of the kernelThe virtual memory mapping associated with the kernel is independent of all processes.The code and data for the kernel reside in the system permanently, and all processes share it.When the system is brought into service (booted) , it load the kernel code into memory and setup the necessary tables and registers to map its virtual addresses into physical memory address.



The Context of a Process

The context of a process consists of the contents of its (user) address space and the contents of hardware registers and kernel data structures that relate to the process.

Formally, the context of a process is the union of its user-level context register context system-level context.

User-level context

The user-level context consists of the process text, data, user stack, and shared memory that occupy the virtual address space of the process.

Parts of the virtual address space of a process that periodically do not reside in main memory because of swapping or paging still constitute a part of the user-level context.


Register Level Context

The program counter specifies the address of the next instruction the CPU will execute; the address is a virtual address in kernel or in user memory space.

The processor status register (PS) specifies the hardware status of the machine as it relate to the process.

The stack pointer contains the current address of the next entry in the kernel or user stack, determined by the mode of execution.

The general purpose registers contain data generated by the process during its execution.

System Level Context

The process table entry of a process defines the state of a process and contains control information that is always accessible to the kernel.

The U area of a process contains process control information that need be accessed only in the context of the process.

General control parameters such as the process priority are stored in the process table because they must be accessed outside the process context.

Region entries, region tables and page tables, define the mapping from virtual to physical address and therefore define the text, data, stack and other regions of a process.

If several processes share common regions, the regions are considered part of the context of each process, b’coz each process manipulate the regions independently.

The kernel stack contains the stack frames of kernel procedures as process execute in kernel mode. The kernel stack is empty when the process executes in user mode.

The dynamic part of the system-level context of a process consists of a set of layers, visualized as a LIFO stack.


Sleep

This chapter has covered all the low-level functions that are executed for the transitions to and from the state “Kernel Running” except for the functions that move a process into the sleep state.

It will conclude with a presentation of the algorithms for sleep which changes the process state from “Kernel Running” to “asleep in memory” and wakeup, which changes the process state from “asleep” to “ready to run” in memory or swapped.


1. Sleep Events and Address


Algorithm sleep

Input: (1) sleep address(2) priority

Output: 1 if process awakened as a result of a signal that process catches.long jump algorithm of process awakened as a result of a signal that it

does not catch.0 otherwise;

{raise processor execution level to block all interrupts;set process state to sleep;put process on sleep hash queue , based on sleep address;save sleep address in process table slot;set process priority level to input priority;if ( process sleep is NOT interruptible ){

do context switch;reset processor priority level to allow interrupts as when

process went to sleep;return (0);

}If ( no signal pending against process ){

do context switch;if ( no signal pending against process )


{reset processor priority level to what it was when process went to sleep;return(0);

}}Removes process from process hash queue , if still there;Reset processor priority level to what it was when process went to sleep;If ( process sleep priority set to catch signals)

return (1)do long jump algorithm;

}

Algorithm wakeup

Algorithm wakeup /* wake up a sleeping process */Input: sleep address

Output: none{

raise processor execution level to block all interrupts;find sleep hash queue for sleep address;for ( every process asleep on sleep address){

remove process from hash queue;mark process state “ready to run”;put process on scheduler list of process ready to run;clear field in process table entry for sleep address;if ( process not loaded in memory )

wake up swapper process (0);else if (awakened process is more eligible to run than currently

running process); set scheduler flag;

}restore processor execution level to original level;

}

Process Creation

The only way for a user to create a new process in the UNIX operating system is to invoke the fork system call.

The process that invokes fork is called the parent process and the newly created process is called the child process.

The syntax for the fork system call is pid = fork()

On return from the fork system call, the two processes have identical copies of their user-level context except for the return value pid.in parent process, pid the child process pid is 0. Process 0 created internally by the kernel when the system is booted, is the only process not created via fork.


The kernel does the following sequence of operations for fork :

It allocates a slot in the process table for the new process. It assign a unique ID number to the child process. It makes a logical copy of the context of the parent process. It increments file and inode table counters for files associated with the process. It returns the ID number of the table counters for files associated with the

process.

Algorithm forkInput : noneOutput : to parent process, child PID number

to child process, 0{

check for available kernel resources;get free proc table slot, unique PID number;check that user not running too many processes;marks child state “being created”;copy data from parent proc table slot to new slot;increment counts on current directory inode and changed root;increment open file counts in file table;make copy of parent context ( u area, text , data , stack ) in memory;push dummy system level context layer onto child system level context;

if ( executing process is parent process ){ change child state to “ready to run”

return (child ID); }else{ initialize u area timing fields;

return ( 0 ); }


Process Termination

Process on a UNIX system terminate by executing the exit system call. An exiting process enters the zombie state, relinquishes its recourses and dismantles its context except for its slot in the process table. The Syntax for the call is

exit (Status); Where the value of status is returned to the parent process for its examination. Process may call exit explicitly or implicitly at the end of a program.

Algorithm exit

Input : return code for processOutput : none{ ignore all signals;

if ( process group leader with associated control terminal ){


send hang up signal to all members of process group;reset process group for all members to 0;

}close all open files (internal version of algorithm close);release current directory ( algorithm iput);release current ( changed ) root, if exists ( algorithm iput);free regions, memory associated with process ( algorithm freereg);write accounting record;make process state zombie…..;send death of child signal to parent process;context switch;

}

Manipulation of process address space

Locking and Unlocking a Region Allocating a Region Attaching a Region to a Process Changing the size of a Region Loading a Region Freeing a Region Detaching a region from a Process Duplicating a Region

Complete Notes

Documents

set auto indent

string includes

eliminating

encompasses

replaces single

unix operating

file named

current working