Introduction to UNIX CS 2204 Class meeting 1 *Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003. (C) Doug Bowman, Virginia Tech, 2001- 2 What is UNIX? A modern computer operating system Operating system: “a program that acts as an intermediary between a user of the computer and the computer hardware” Software that manages your computer’s resources (files, programs, disks, network, …) e.g. Windows, MacOS Modern: features for stability, flexibility, multiple users and programs, configurability, etc. (C) Doug Bowman, Virginia Tech, 2001- 3 Why UNIX? Used in many scientific and industrial settings Huge number of free and well-written software programs Open-source OS Internet servers and services run on UNIX Largely hardware-independent Based on standards (C) Doug Bowman, Virginia Tech, 2001- 4 Brief history of UNIX Ken Thompson & Dennis Richie originally developed the earliest versions of UNIX at Bell Labs for internal use in 1970s Borrowed best ideas from other Oss Meant for programmers and computer experts Meant to run on “mini computers” (C) Doug Bowman, Virginia Tech, 2001- 5 Early UNIX History Thompson also rewrote the operating system in high level language of his own design which he called B. The B language lacked many features and Ritchie decided to design a successor to B which he called C. They then rewrote UNIX in the C programming language to aid in portability. (C) Doug Bowman, Virginia Tech, 2001- 6 UNIX variants Two main threads of development: Berkeley software distribution (BSD) Unix System Laboratories System V Sun: SunOS, Solaris GNU: Linux (many flavors) SGI: Irix FreeBSD Hewlett-Packard: HP-UX Apple: OS X (Darwin) …
55
Embed
What is UNIX? Introduction to UNIX - Virginia Techcourses.cs.vt.edu/~cs2204/fall2004/resources/CS2204_notes_Summer_2004.pdf · “a program that acts as an intermediary between a
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Introduction to UNIX
CS 2204Class meeting 1
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003. (C) Doug Bowman, Virginia Tech, 2001- 2
What is UNIX?A modern computer operating systemOperating system:
“a program that acts as an intermediary between a user of the computer and the computer hardware” Software that manages your computer’s resources (files, programs, disks, network, …)e.g. Windows, MacOS
Modern: features for stability, flexibility, multiple users and programs, configurability, etc.
(C) Doug Bowman, Virginia Tech, 2001- 3
Why UNIX?Used in many scientific and industrial settingsHuge number of free and well-written software programsOpen-source OSInternet servers and services run on UNIXLargely hardware-independentBased on standards
(C) Doug Bowman, Virginia Tech, 2001- 4
Brief history of UNIXKen Thompson & Dennis Richieoriginally developed the earliest versions of UNIX at Bell Labs for internal use in 1970s
Borrowed best ideas from other OssMeant for programmers and computer expertsMeant to run on “mini computers”
(C) Doug Bowman, Virginia Tech, 2001- 5
Early UNIX HistoryThompson also rewrote the operating system in high level language of his own design which he called B.The B language lacked many features and Ritchie decided to design a successor to B which he called C.They then rewrote UNIX in the C programming language to aid in portability.
(C) Doug Bowman, Virginia Tech, 2001- 6
UNIX variantsTwo main threads of development:
Berkeley software distribution (BSD)Unix System Laboratories System V
Sun: SunOS, SolarisGNU: Linux (many flavors)SGI: IrixFreeBSDHewlett-Packard: HP-UXApple: OS X (Darwin)…
Standard Utility Programs(shell, editors, compilers, etc.)
Users
System Interface calls
Library Interface
User Interface
User Mode
Kernel Mode
(C) Doug Bowman, Virginia Tech, 2001- 8
UNIX StructureThe kernel is the core of the UNIX system, controlling the system hardware and performing various low-level functions. The other parts of the UNIX system, as well as user programs, call on the kernel to perform services for them.
(C) Doug Bowman, Virginia Tech, 2001- 9
UNIX StructureThe shell is the command interpreter for the UNIX system. The shell accepts user commands and is responsible for seeing that they are carried out.
(C) Doug Bowman, Virginia Tech, 2001- 10
UNIX StructureOver two hundred utility programs or toolsare supplied with the UNIX system. These utilities (or commands) support a variety of tasks such as copying files, editing text, performing calculations, and developing software.This course will only attempt to introduce a limited number of these utilities or tools!
(C) Doug Bowman, Virginia Tech, 2001- 11
Getting started
(C) Doug Bowman, Virginia Tech, 2001- 12
Logins and passwordsYou must have an “account” on the UNIX machine you’re trying to use“login” is your user name (usually some variant of your real name)Your password will not echo as you typeRemember good password practices
3
(C) Doug Bowman, Virginia Tech, 2001- 13
The shell promptAfter logging in, you get some information about the system, then a shell promptshell: The program you use to send commands to the UNIX systemPrompts take many forms:
$%username@hostname>hostname # %
Must beat the “Tyranny of the blank screen”(C) Doug Bowman, Virginia Tech, 2001- 14
Entering commandsAnytime you see a prompt, you can enter a command for the shellSome commands are a single word
whodatels
Others use additional informationcat textfilels -l
(C) Doug Bowman, Virginia Tech, 2001- 15
Command syntaxCommands must be entered exactly. If you make a mistake before entering, delete/backspace to fix it. Be careful!command options argument(s)Options: modify command’s executionArguments: often filenames that tell a command what to operate on
(C) Doug Bowman, Virginia Tech, 2001- 16
Example commands: ls (list)ls –lls –a –lls –alls –a; ls -lls textfilels folderls textfile1 textfile2ls –al textfile
(C) Doug Bowman, Virginia Tech, 2001- 17
If you don’t get a normal shell prompt…
A program is probably runningIf you see a special program prompt, try to quit the program (quit, bye, exit)If you see nothing, you can
Stop the program with CTRL-z (program will wait until started again)Interrupt the program with CTRL-c (program will usually die)
(C) Doug Bowman, Virginia Tech, 2001- 18
Ending your sessionAlways log out when you are doneUse the command exit (also sometimes logout or CTRL-d)Note: if you are running a window system, logging out of the shell only ends that shell. You must also log out of the window system using a menu.
1
UNIX filesystem
CS 2204Class meeting 2
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
(C) Doug Bowman, Virginia Tech, 2001 2
UNIX filesystemThe filesystem is your interface to
physical storage (disks) on your machinestorage on other machinesoutput devicesetc.
Everything in UNIX is a file (programs, text, peripheral devices, terminals, …)There are no drive letters in UNIX! The filesystem provides a logical view of the storage devices
(C) Doug Bowman, Virginia Tech, 2001 3
Working directoryThe current directory in which you are workingpwd command: outputs the absolute path (more on this later) of your working directoryUnless you specify another directory, commands will assume you want to operate on the working directory
(C) Doug Bowman, Virginia Tech, 2001 4
Home directoryA special place for each user to store personal filesWhen you log in, your working directory will be set to your home directoryYour home directory is represented by the symbol ~ (tilde)The home directory of “user1” is represented by ~user1
(C) Doug Bowman, Virginia Tech, 2001 5
UNIX file hierarchyDirectories may contain plain files or other directoriesLeads to a tree structure for the filesystemRoot directory: /No file extensions!
/
tmpusersbin
user1 user2
cs2204joketxt
lab2txtlab1txt
(C) Doug Bowman, Virginia Tech, 2001 6
Some Standard Directories/ - root directory/bin – Home of all binaries/dev – device directory/etc – host-specific files and directories/home – users home directories
/home/grads/sgifford/lib – libraries for various languages/sbin – System administration utilities, tools,…/tmp – temporary files/var – Variable data that is changing
2
(C) Doug Bowman, Virginia Tech, 2001 7
Path namesSeparate directories by /Absolute path
start at root and follow the treee.g. /users/user1/joketxte.g. ~user1/joketxte.g. ~/joketxt
/
tmpusersbin
user1 user2
cs2204joketxt
lab2txtlab1txt
(C) Doug Bowman, Virginia Tech, 2001 8
Path names (cont’d)Relative path
start at working directory.. refers to level above;. refers to working dir.If /users/user1/cs2204is working dir, these both refer to the same file:../joketxt../../user1/joketxt
/
tmpusersbin
user1 user2
cs2204joketxt
lab2txtlab1txt
(C) Doug Bowman, Virginia Tech, 2001 9
Changing directoriesChange the working directory with the cd commandcd <directory_path>Use absolute or relative path namescd without a path is equivalent to cd ~
e.g. cd ../../user1/e.g. cd /users/user1/
(C) Doug Bowman, Virginia Tech, 2001 10
Output of ls -lFtotal 4lrwxr-xr-x 1 user1 user 18 Aug 28 13:41 home ->
/usr/people/bowman/ -rw-r--r-- 1 user1 user 94 Aug 28 13:42 nothing.txtdrwxr-xr-x 2 user1 user 9 Aug 28 13:40 test_dir/
We’ll keep coming back to this slide!File type
Permissions Owner Group Modify date Filename
Size in bytes
(C) Doug Bowman, Virginia Tech, 2001 11
Types of filesPlain (-)
Most filesIncludes binary and text files
Directory (d)A directory is actually a filePoints to another set of files
Link (l): A pointer to another file or directorySpecial: e.g. peripheral devices
Creating and removing directoriesmkdir <directory_name>
Create a subdirectory of the current directoryrmdir <directory _name>
Remove a directory (only works for empty directories)
rm –r <directory _name>Remove a directory and all of its contents, including subdirectories
(C) Doug Bowman, Virginia Tech, 2001 15
Creating linksln –s <existing_file> <link_name>This command creates a symbolic linkThe file “link_name” will be a pointer to the “existing_file” which may be in another directory or even on another physical machine
(C) Doug Bowman, Virginia Tech, 2001 16
File ownershipEach file has a single ownerchown command can be used to change the owner (usually only root user can use this command)There are also various groups to which users can belongGroups may have different permissions than everyone else
(C) Doug Bowman, Virginia Tech, 2001 17
File permissionsPermissions used to allow/disallow access to file/directory contentsRead (r), write (w), and execute (x)For owner, group, and world (everyone)chmod <mode> <file(s)>chmod 700 filetxtchmod g+rw filetxt
(C) Doug Bowman, Virginia Tech, 2001 18
File modification dateLast time the file was changedUseful information when
There are many copies of a fileMany users are working on a file
touch command can be used to update the modification date to the current date (or to create a file if it doesn’t exist)
4
(C) Doug Bowman, Virginia Tech, 2001 19
Looking at file contentscat <filename(s)>
“concatenate”output the contents of the file all at once
more <filename(s)>Output the contents of a file one screen at a timeAllows forward and backward scroll and search
(C) Doug Bowman, Virginia Tech, 2001 20
Wildcards in file namesAll of the commands covered here that take file names as arguments can also use wildcards* for any string, e.g. *txt, obj*, a*.*? for any character, e.g. doc?[] around a range of characters, e.g. [a-c]*
(C) Doug Bowman, Virginia Tech, 2001 21
Getting help on UNIX commands
These notes only give you the tip of the iceberg for these basic commandsman <command_name> shows you all the documentation for a commandapropos <keyword> shows you all the commands with the keyword in their description
1
Text editing and more basic commands
CS 2204Class meeting 3
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
(C) Doug Bowman, Virginia Tech, 2001 2
Text editingSo far, we’ve manipulated files in the file system and viewed their contentsNow we need to know how to modify the contents of filesMost UNIX editors work on plain ASCII text files, not special formatsCommon text editors: vi, emacs, picoWindow systems have own window-based editors
(C) Doug Bowman, Virginia Tech, 2001 3
Why vi?Available on practically all UNIX systemsCan be used with a remote login (no windows)Balance of simplicity and power
That’s what your professor had to learn, so, darn it, you’re going to learn it too! ☺
(C) Doug Bowman, Virginia Tech, 2001 4
vi basicsInvoke with: vi [filename(s)]The vi editor takes over your screenVarious modes:
command mode – typed characters send commands to systeminsert modeappend mode typing changes the textchange mode
(C) Doug Bowman, Virginia Tech, 2001 5
Command modevi starts in this modeFrom other modes, enter command mode by typing EscCommands for:
Cursor movementEditingFile operationsSearchEntering other modes
(C) Doug Bowman, Virginia Tech, 2001 6
Cursor movementMove the cursor with j (down), k (up), l (right), h (left), or with arrow keysLarger movements:
4j – move four lines downCTRL-F, CTRL-B – page down, page upw – move to beginning of next word]], [[ - next/previous section0, $ - beginning/end of current line:n – move to line n
2
(C) Doug Bowman, Virginia Tech, 2001 7
Editing commandsu – undo last typingx – delete current character*dd – delete current line*dw – delete current word*rx – replace current char. with x*yy – copy current line*p – paste copied/deleted itemsJ – join two lines* can be preceded by a number to do operation multiple times
(C) Doug Bowman, Virginia Tech, 2001 8
File operationsZZ, :wq – save and quit:w – save:w filename – save as filename:q – quit:q! – quit without saving:e <filename> - load another file:n – load next file
(C) Doug Bowman, Virginia Tech, 2001 9
Search/string – search forward for string?string – search backward for stringn – repeat previous searchN – repeat search opposite direction% - find match of current (, [, {
(C) Doug Bowman, Virginia Tech, 2001 10
Entering other modesi – insert text before current char.I – insert text at beginning of linea – append text after current char.A – append text at end of lineO, o – open line above/below currentcw – change (overwrite) current wordC – change text after cursor
(C) Doug Bowman, Virginia Tech, 2001 11
Additional infoUNIX in a nutshell, ch. 8 lists all vi commandsTrivia: vi uses all lowercase characters except ‘v’ (and many uppercase and punctuation characters) for commands
Be careful!Use ‘u’, ‘U’, and ‘:q!’ if you mess up
UIAN, ch. 7 covers emacs
(C) Doug Bowman, Virginia Tech, 2001 12
More shell commands: communicationtalk – interactive chat with another user (not on lab machines)Mail – text-based email programftp – text-based FTP programtelnet, ssh – connect to other machines directlylynx – text-based web browser
3
(C) Doug Bowman, Virginia Tech, 2001 13
More shell commands: processesps – list current processestop – dynamic display of system’s utilization by processeskill – terminate a processtime – keep timing information for a process
1
I/O Redirection & Regular Expressions
CS 2204Class meeting 4
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
(C) Doug Bowman, Virginia Tech, 2001 2
Redirecting stdoutInstead of writing to the terminal, you can tell a program to print its output to another file using the > operator>> operator is used to append to a fileExamples:man ls > ls_help.txtEcho $PWD > current_directorycat file1 >> file2
(C) Doug Bowman, Virginia Tech, 2001 3
Redirecting stdinInstead of reading from the terminal, you can tell a program to read from another file using the < operatorExamples:Mail [email protected] < messageinteractive_program < command_list
(C) Doug Bowman, Virginia Tech, 2001 4
Pipes and filtersPipe: a way to send the output of one command to the input of anotherFilter: a program that takes input and transforms it in some way
wc - gives a count of words/lines/charsgrep - searches for lines with a given stringmoresort - sorts lines alphabetically or numerically
A patternDefines a set of strings of charactersAny string in the set is said to be “matched” by the RE (the RE matches the string)
2
(C) Doug Bowman, Virginia Tech, 2001 7
Why REs?Pattern matching is a useful tool in many real-world situations:
Search for a file on a filesystemFind and replace strings in a fileExtract particular data elements from a database
REs are an important part of formal languages - one of the basic CS theory disciplines
(C) Doug Bowman, Virginia Tech, 2001 8
UNIX programs that use REsgrep (search within files)egrep (grep but with extended RE’s)vi/emacs (text editors)ex (line editor)sed (stream editor)awk (pattern scanning language)perl (scripting language)
(C) Doug Bowman, Virginia Tech, 2001 9
Characters vs. metacharacters
In patterns, characters can be any character except a newlineMetacharacters are special characters that have a special meaningTo use metacharacters as regular characters in a pattern, quote them with the ‘\’ character
(C) Doug Bowman, Virginia Tech, 2001 10
Basic vs. Extended RE’sIn basic regular expressions themetacharacters ?, +, {, |, (, and ) have no special meaning (grep).
To make them special, instead use thebackslashed versions \?, \+, \{, \|, \(, and \)
For extended regular expressions these are interpreted as specialGrep –E egrep
(C) Doug Bowman, Virginia Tech, 2001 11
Using egrepegrep pattern filename(s)To be safe, put quotes around your patternExamples:
Asterisk (*) matches zero or moreoccurrences of the previous RE
Not the same as wildcards in the shell!“ab*c” matches ac, abc, abbc, abbbc, …“.*” matches any string
3
(C) Doug Bowman, Virginia Tech, 2001 13
Metacharacters 2Plus (+): matches one or more occurrences of the preceding RE
“ab+c” matches abc, abbc, but not acQuestion mark (?): matches zero or one occurrences of the preceding RE
“ab?c” matches ac, abc but not abbcLogical or (|): matches RE before or RE after bar
“abc|def” matches abc or def
(C) Doug Bowman, Virginia Tech, 2001 14
Metacharacters 3Caret (^): means beginning of line
“^D” matches all strings starting with DDollar sign ($) means end of line
“d$” matches all strings ending with dBackslash (\): used to quote othermetacharacters
“file\.txt” matches file.txt but not fileatxt
(C) Doug Bowman, Virginia Tech, 2001 15
Metacharacters 4Square brackets ([ ]) indicate a set/range of characters
Any characters in the set will match^ before the set means “not”- between characters indicates a rangeExamples:
“[fF]un” matches fun, Fun“b[aeiou]g” matches bag, beg, big, bog, bug“[A-Z].*” matches any string beginning with a captial letter“[^abc].*” matches any string not starting with a, b, or c
(C) Doug Bowman, Virginia Tech, 2001 16
Metacharacters 5Parentheses ( ): used to group REs when using other metacharacters
“a(bc)*” matches a, abc, abcbc, abcbcbc, …“(foot|base)ball” matches football, baseball
Braces ({ }): specify the number of repetitions of an RE
“[a-z]{3}” matches three lowercase letters“m.{2,4}” matches strings starting with m between three and five letters
(C) Doug Bowman, Virginia Tech, 2001 17
What do these mean?egrep “^B.*s$” fileegrep “ [0-9]{3} ” fileegrep “num(ber)? [0-9]+” fileegrep “word” file | wc -legrep “[A-Z].*\?” fileWhat about if they were grep?
(C) Doug Bowman, Virginia Tech, 2001 18
PracticeConstruct egrep commands that find in file:
Lines beginning with a word of at least 10 charactersLines containing a student ID number in standard 3-part formNumber of lines with 2 consecutive capitalized wordsNumber of lines not ending in an alphabetic characterLines containing a word beginning with a vowel at the end of a sentence
1
UNIX window systems
CS 2204Class meeting 5
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
(C) Doug Bowman, Virginia Tech, 2001 2
Why window systems?Increased usability due to
VisibilityGraphical representation of programsSee multiple environments at once
Direct manipulationEnables powerful graphics programs
(C) Doug Bowman, Virginia Tech, 2001 3
Window systems and UNIXMost UNIX users can be considered experts, and are fiercely protective of the command line (for good reason)However, all current UNIX systems have a built-in window system - the advantages are inescapable
(C) Doug Bowman, Virginia Tech, 2001 4
X WindowsPractically all UNIX window systems are based on X WindowsThe standard version is called X11Very complex system with many partsBasically, X11:
Manages the screen spaceDraws simple graphicsAssigns rectangular regions to various programs
(C) Doug Bowman, Virginia Tech, 2001 5
X’s client-server architectureX is actually meant to work over the networkX server: software that runs on the machine where the program’s output will be displayedX client: program running on the same or another machineClient sends drawing and other X commands to the server, which displays the results
(C) Doug Bowman, Virginia Tech, 2001 6
Historical use of XUsers sat at “X terminals” - graphical terminals that only knew how to run an X serverThey logged in to other UNIX machines remotely and ran X clients thereThis gave users the benefits of a window system without the need for a full-featured computer on every desk
2
(C) Doug Bowman, Virginia Tech, 2001 7
Features of XTransparent remote executionGives programs their own virtual screenIncludes important windowing concepts:
Window damageWindow reveal eventsBacking store
X11 programs are highly portable
(C) Doug Bowman, Virginia Tech, 2001 8
Window managersNot part of X11 itselfRun on top of X11Place borders, decorations on windowsHandle input from usersThere are many, many choices with different “look & feel”
(C) Doug Bowman, Virginia Tech, 2001 9
Desktop environmentsYet another layer, running on top of window managersComplete the desktop metaphor with:
Iconic access to files and directoriesOverall system menus / toolbarsTrash canetc.
GNOME is one example
1
UNIX shell environments
CS 2204Class meeting 6
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
(C) Doug Bowman, Virginia Tech, 2001 2
Shell characteristicsCommand line interface between the user and the systemIs simply a program that automatically starts when you loginWaits for user to type in commands
(C) Doug Bowman, Virginia Tech, 2001 3
Main shell featuresInteractivity
aliasesfile-name completion
Scripting languageAllows programming (shell scripting) within the shell environmentUses variables, loops, conditionals, etc.Next lecture
(C) Doug Bowman, Virginia Tech, 2001 4
Various UNIX shellssh (Bourne shell, original UNIX shell)ksh (Korn shell)csh (C shell, developed at Berkeley)tcshbash (Bourne again SHell)…Differences mostly in level of interactivity support and scripting details
(C) Doug Bowman, Virginia Tech, 2001 5
The Bourne again SHell (bash)We will be using bash and ksh as the standard shells for this classThis will be important for shell scripting assignmentsSuperset of the Bourne shell (sh)Borrows features from sh, csh, tcsh & kshCreated by Free Software Foundation
(C) Doug Bowman, Virginia Tech, 2001 6
Changing your shellOn most UNIX machines (and lab):which bash (note path)chsh
On the some machines:which ksh (note path /bin/bash)ypchsh
2
(C) Doug Bowman, Virginia Tech, 2001 7
Environment variablesA set of variables the shell uses for certain operationsVariables have a name and a valueCurrent list can be displayed with the envcommandA particular variable’s value can be displayed with echo $<var_name>
Setting environment variablesSet a variable with <name>=<value>Examples:
TERM=vt100PS1=myprompt>PS1=$USER@$HOSTNAME:PS1=“multiple word prompt> “PATH=$PATH:$HOME/binPATH=$PATH:~
DATE=`date`
(C) Doug Bowman, Virginia Tech, 2001 10
AliasesAliases are used as shorthand for frequently-used commandsSyntax: alias <shortcut>=<command>Examples:
alias ll=“ls -lF”alias la=“ls -la”alias m=morealias up=“cd ..”alias prompt=“echo $PS1”
(C) Doug Bowman, Virginia Tech, 2001 11
Repeating commandsUse history to list the last 16 commandsUse fc -l <m> <n> to list previously typed commands m through n
(C) Doug Bowman, Virginia Tech, 2001 12
Editing on the command lineSome command lines can be very long and complicated - if you make a mistake you don’t want to start all over againYou can interactively edit the command line in several ways if using ksh
set -o vi allows you to use vi commands to edit the command lineset -o vi-tabcomplete also lets you complete commands/filenames by entering a TAB
3
(C) Doug Bowman, Virginia Tech, 2001 13
Login scriptsYou don’t want to enter aliases, set environment variables, set up command line editing, etc. each time you log inAll of these things can be done in a script that is run each time the shell is started
(C) Doug Bowman, Virginia Tech, 2001 14
Login scripts (continued)For bash, order of files is:/etc/profile ~/.bash_profile~/.bash_login~/.profile
After logout~/.bash_logout
(C) Doug Bowman, Virginia Tech, 2001 15
Example .bash_profile (partial)# .bash_profile: executed by bash(1) for login
shellsumask 022
# include .bashrc if it exists
if [ -f ~/.bashrc ]; thensource ~/.bashrc
fi# some ls aliasesalias ll='ls -l'alias la='ls -A'alias l='ls -CF'
Script executionProvide script as an argument to the shell program (e.g. bash my_script)Or specify which shell to use within the script
First line of script is #!/bin/bashMake the script executable using chmodMake sure the PATH includes the current directoryRun directly from the command line
No compilation is necessary!
(C) Doug Bowman, Virginia Tech, 2001 5
Simple example script#!/bin/bash
echo “Hello world!”cd ~pwd Output:
Hello world!/home/grads/sgifford
(C) Doug Bowman, Virginia Tech, 2001 6
QuotingQuoting is necessary to use special characters in a variable’s value or string”” - shell only interprets $ and ‘‘ and \
String variablesIf you do not use the declare keyword with option –i when using a variable for the first time, it will be a stringvar=100 makes var the string ‘100’.
However, (( var=100 )) will treat var as an integer even though it is a string
(C) Doug Bowman, Virginia Tech, 2001 10
Array variablesArray is a list of values
Don’t have to declare sizeReference a value by ${name[index]}
${array[@]} or ${array[*]} refers to the entire array contentsecho ${moresports[*]} produces
football basketball tennis
(C) Doug Bowman, Virginia Tech, 2001 12
Command line argumentsIf arguments are passed to a script, they can be referenced by $1, $2, $3, …$0 refers to the name of the script$@ - array filled with arguments excluding $0$# - number of arguments
3
(C) Doug Bowman, Virginia Tech, 2001 13
Exporting variablesThe export command, when used with a variable name, allows child processes of the shell to access the variable
(C) Doug Bowman, Virginia Tech, 2001 14
OutputWe have already seen echoMore common in other shells including ksh is print (does not exist in bash)echo –n does not print newline after output
(C) Doug Bowman, Virginia Tech, 2001 15
Return valuesScripts can return an integer valueUse return NThe variable $? will contain the return value of the last command runCan be used to test conditions
(C) Doug Bowman, Virginia Tech, 2001 16
ConditionsIf using integers: (( condition ))If using strings: [[ condition ]]Examples:
Other important g++ options-g: include debugging symbols in the output-l<name>: include a library called libname.a
(C) Doug Bowman, Virginia Tech, 2001 10
Include and library pathsThere are default directories in which g++ looks for include files and libraries-I<path>: also look for include files in this directory-L<path>: also look for library files in this directory
(C) Doug Bowman, Virginia Tech, 2001 11
Defines in g++Often programs contain conditional parts based on defines:#ifdef DEBUGprintf(“value of var is %d”, var);#endif
You can set preprocessor defines on the command lineg++ -DDEBUG -o prog prog.c
(C) Doug Bowman, Virginia Tech, 2001 12
Using make in compilationWith medium to large software projects containing many files, it’s difficult to:
Type commands to compile all the files correctly each timeKeep track of which files have been changedKeep track of files’ dependencies on other files
Basic operation of makeReads a file called [Mm]akefile, which contains rules for building a “target”If the target depends on a file, then that file is builtIf that file depends on a third file, then the third file is built, and so on…Works backward through the chain of dependenciesTargets only built if they are older than the files they depend on
(C) Doug Bowman, Virginia Tech, 2001 15
Types of lines in MakefilesDependency or rules linesCommandsMacro assignmentsComments
(C) Doug Bowman, Virginia Tech, 2001 16
Dependency/rules linesSpecify a target and a list of prerequisites (optional) for that target
target : prereq1 prereq2 prereq3 …
(C) Doug Bowman, Virginia Tech, 2001 17
Command linesFollow dependency linesMUST start with a TAB!Any command that can be run in the shell can be placed here
target : prereq1 prereq2command1command2Special variables in commands:
$@ represents the target$? represents prereqs that are newer than target
Comments begin with a ‘#’Can be placed at the beginning of a line or after a non-comment line
Lines that are too long can be continued on the next line by placing a ‘\’ at the end of the first line
(C) Doug Bowman, Virginia Tech, 2001 22
Invoking makeBe sure that your description file:
is called makefile or Makefileis in the directory with the source files
make (builds the first target in the file)make target(s) (builds target(s))Important options:-n: don’t run the commands, just list them-f file: use file instead of [Mm]akefile
Suffix rulesIt’s still tedious to specifically tell makehow to build each .o file from a .c/.cc fileSuffix rules can be used to generalize such situationsA default suffix rule turns .c/.cc files into .o files by running the command:
${CC} ${CFLAGS} -c $<$< refers to the prerequisite (file.cc)
5
(C) Doug Bowman, Virginia Tech, 2001 25
Simplifying the example Makefile again
OBJS = main.o iodat.o dorun.oCC = /usr/bin/g++
program : ${OBJS}${CC} -o $@ ${OBJS}
(C) Doug Bowman, Virginia Tech, 2001 26
Other useful Makefile tipsInclude a way to clean up your mess
No dependencies!clean:/bin/rm -f *.o coreInclude a target to build multiple programs
all:program1 program2 program3
1
Development: Revision control
CS 2204Class meeting 9
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
(C) Doug Bowman, Virginia Tech, 2001 2
Overview of software development process
Creation of source files (.c, .h, .cpp)RCS, this lecture
Compilation (e.g. *.c *.o) and linkingg++ and make, last lecture
Running and testing programsgdb debugger, next lecture
(C) Doug Bowman, Virginia Tech, 2001 3
What is revision control?A way to ensure that edits on files are
Logged / archivedConsistent
Especially applicable when projects include
Multiple filesMultiple editors (developers)Maintenance (corrective and enhancing)
(C) Doug Bowman, Virginia Tech, 2001 4
Why do you need revision control?
Software development projects are normally done in teamsMultiple people may be responsible for code in a single fileYou may want to freeze a working version and create a new revision branchYou may want to roll back development to a previous point in time
(C) Doug Bowman, Virginia Tech, 2001 5
Overview of RCSMaintains complete revision information for a set of files (not limited to source code)Uses major and minor revision numbers (e.g. 1.1, 2.3)Supports locking files so that only one user can edit at a timeSupports merging two edits of the same fileCan automatically place revision information within the file itself
(C) Doug Bowman, Virginia Tech, 2001 6
Key RCS commandsrcs (administer project)ci (check in file)co (check out file)rlog (view the log)rcsdiff (see changes since last revision)For more, see UIAN ch. 19
2
(C) Doug Bowman, Virginia Tech, 2001 7
Basic RCS usageCreate a subdirectory RCS of the directory containing the files in questionCheck in the files to create an initial revision 1.1Check out and lock files to edit themCheck files back in to create a new revision and allow others to edit them
(C) Doug Bowman, Virginia Tech, 2001 8
Checking out filesAssume we start with file main.cchecked in (RCS/main.c,v)co main.c
Creates read-only file main.c in pwdco -l main.c
Creates writable file main.c in pwdLocks file so others can’t check it out
(C) Doug Bowman, Virginia Tech, 2001 9
Checking in filesAssume we’ve locked and edited main.cci main.c
Creates a new revision (e.g. 1.2)Removes main.c from pwd
ci -u main.cCreates a new revisionLeaves read-only main.c in pwd
ci -l main.cCreates a new revision (checkpoint)Allows you to keep editing
(C) Doug Bowman, Virginia Tech, 2001 10
Setting the revision numberPerhaps we’ve done major revisions to revision 1.4 of main.c and we want to start a new branch (2) on the revision treeci -r2 main.c
(C) Doug Bowman, Virginia Tech, 2001 11
Using keyword substitutionEach time you check in a new revision, RCS keeps information about the author, date/time, a log message (which you provide), etc.You can show this information in files by placing special tags within the files when setting up RCS
(C) Doug Bowman, Virginia Tech, 2001 12
Keyword example/** Last revision: $Revision$ by $Author$ on $Date$
* $Log$*/
main(){
...}
/** Last revision: $Revision:
1.1 $ by $Author: bowman $ on $Date: 2001/10/10 20:29:27 $
* $Log: main.c,v $* Revision 1.1 2001/10/10
20:29:27 bowman* Initial revision**/
main(){
...}
3
(C) Doug Bowman, Virginia Tech, 2001 13
RCS hintsUsually use ci -u file to keep a read-only copy of file for compilation, etc.Use co -l -rR file to retrieve revision number R of file instead of most recent revision
1
Development: Debugging
CS 2204Class meeting 9-10
*Notes by Mir Farooq Ali and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
2
Example 1TYPE: Accidental
for (i=0; i<numrows; i++)for (j=0; j<numcols; j++);
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
(C) Doug Bowman, Virginia Tech, 2001 2
Overview of software development process
Creation of source files (.c, .h, .cpp)RCS, last lecture
Compilation (e.g. *.c *.o) and linkingg++ and make, a previous lecture
Running and testing programsgdb debugger, this lecture
(C) Doug Bowman, Virginia Tech, 2001 3
Why use a debugger?No one writes perfect code first time, every timeDesk checking code can be tedious and error-pronePutting print statements in the code requires re-compilation and a guess as to the source of the problemDebuggers are powerful and flexible
(C) Doug Bowman, Virginia Tech, 2001 4
Common debugger functionsRun programStop program at breakpointsExecute one line at a timeDisplay values of variablesShow sequence of function callsCatch signals
(C) Doug Bowman, Virginia Tech, 2001 5
The GNU debugger (gdb)Free, like all GNU softwareCommand line debuggerMost common ways to invoke:gdb executablegdb executable coregdb executable process_id
(C) Doug Bowman, Virginia Tech, 2001 6
Execution commandslist or l (list code)
listlist mainlist 56
run or r (run program from beginning)runrun file.txt file2.txt
next or n (execute next line, stepping over function calls)step or s (execute next line, stepping into function calls)
2
(C) Doug Bowman, Virginia Tech, 2001 7
Breakpoint commandsbreak or b (set a breakpoint)break mainbreak 10
delete or d (delete a breakpoint)deletedelete 2
continue or c (continue execution when stopped)
(C) Doug Bowman, Virginia Tech, 2001 8
Program information commandsprint or p (print value)print xprint x*yprint function(x)
display (continuously display value)undisplay (remove displayed value)where (show current function stack)
(C) Doug Bowman, Virginia Tech, 2001 9
Miscellaneous commandsset (change a value)set n=3
help or h (display help text)helphelp stephelp breakpoints
quit or q (quit gdb)
1
System programming: file management
CS 2204Class meeting 11
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
(C) Doug Bowman, Virginia Tech, 2001 2
Parts of the UNIX OS
Shell
Kernel
Tools & Apps
H/W
(C) Doug Bowman, Virginia Tech, 2001 3
UNIX system programmingProgramming that uses special features of the UNIX system (the Kernel)Programs make system calls via librariesTypes of system calls
File I/OProcess managementInter-process communication (IPC)Signal handling
(C) Doug Bowman, Virginia Tech, 2001 4
Basic file I/ORemember everything in UNIX is a fileProcesses keep a list of open filesFiles can be opened for reading, writing
Note: Some of the system calls used to interact with the UNIX kernel are also available in other OS’. However, they may be (probably are) implemented much differently and some are not available at all.
(C) Doug Bowman, Virginia Tech, 2001 5
Basic file I/OEach file is referenced by a file descriptor(integer)Three files are opened automatically
FD 0: standard inputFD 1: standard outputFD 2: standard error
When a new files is opened, it is assigned the lowest available FDman 2 <systemcall> (for more information)
(C) Doug Bowman, Virginia Tech, 2001 6
File I/O system calls - open()fd = open(path, flags, mode)path: char*, absolute or relative pathbasic flags:
O_RDONLY - open for readingO_WRONLY - open for writingO_RDWR - open for reading and writingO_CREAT - create the file if it doesn’t existO_TRUNC - truncate the file if it exists (effectively delete existing file)O_APPEND - only write at the end of the file
2
(C) Doug Bowman, Virginia Tech, 2001 7
File I/O system calls - open()fd = open(“myfile”,O_CREAT, 00644)
some other flags:O_NDELAY - open immediatelyO_EXCL – create file if it does not existO_SYNC – delay the opening of the file until there is an attempt to write to file.
mode: specify permissions if using O_CREAT
(C) Doug Bowman, Virginia Tech, 2001 8
File I/O system calls - close()return_val = close(fd)
Close an open file descriptorReturns 0 on success, -1 on error
(C) Doug Bowman, Virginia Tech, 2001 9
File I/O system calls - read()bytes_read = read(fd, buffer, count)
Read up to count bytes from file and place into bufferfd: file descriptorbuffer: pointer to arraycount: number of bytes to readReturns number of bytes read or -1 if error
(C) Doug Bowman, Virginia Tech, 2001 10
File I/O system calls -write()
bytes_written = write(fd, buffer, count)Write count bytes from buffer to a filefd: file descriptorbuffer: pointer to arraycount: number of bytes to writeReturns number of bytes written or -1 if error
(C) Doug Bowman, Virginia Tech, 2001 11
File I/O system calls -lseek()
retval = lseek(fd, offset, whence)Move file pointer to new locationfd: file descriptoroffset: number of byteswhence:
SEEK_SET - offset from beginning of fileSEEK_CUR - offset from current offset locationSEEK_END - offset from end of file
Returns offset from beginning of file or -1
(C) Doug Bowman, Virginia Tech, 2001 12
Simple file I/O example
3
(C) Doug Bowman, Virginia Tech, 2001 13
File I/O using FILEsMost UNIX programs use higher-level I/O functions
fopen()fclose()fread()fwrite()fseek()
These use the FILE datatype instead of file descriptorsNeed to include stdio.h
(C) Doug Bowman, Virginia Tech, 2001 14
File I/O using FILEsfopen/fclose – simular to open/close
fopen returns a FILE* mode is ‘r’, ‘w’, etc.man fopen
The function fread/fwrite reads/write nmembelements of data, each size bytes long, from the stream pointed to by stream, storing them at the location given by ptr.
man fread
(C) Doug Bowman, Virginia Tech, 2001 15
Simple file I/O example using FILEs
(C) Doug Bowman, Virginia Tech, 2001 16
Using datatypes with file I/OAll the functions we’ve seen so far use raw bytes for file I/O, but program data is usually stored in meaningful datatypes (int, char, float, etc.)fprintf(), fputs(), fputc() - used to write data to a filefscanf(), fgets(), fgetc() - used to read data from a fileman <function> (for more info)
(C) Doug Bowman, Virginia Tech, 2001 17
Detour: printf/scanfOutput to screen: printfSyntax
printf (control string , argument list )Function Prototype:
int printf( const char, *format, ... ) ;
(C) Doug Bowman, Virginia Tech, 2001 18
Control String Formatterschar conversion
%d %i decimal integer%u unsigned decimal integer%o unsigned octal integer%x %X unsigned hexadecimal integer%c character%s string or character array%f float%e %E double (scientific notation)%g %G float or double format%% outputs a '%' character
4
(C) Doug Bowman, Virginia Tech, 2001 19
Learning by Exampleprintf("The sum of %d, %d, and %d is %d\n", 65, 87, 33, 65+87+33);
The sum of 65, 87, and 33 is 185printf("Character code %c has ASCII code %d.\n", 'A', 'A');
Character code A has ASCII code 65.printf("Error %s occurred at line %d \n",emsg, lno);
Error invalid variable occurred at line 27Note: emsg and Ino are variables!
(C) Doug Bowman, Virginia Tech, 2001 20
Learning by Exampleprintf("Octal form of %d is %o \n", 59, 59);
Octal form of 59 is 73printf("Hexadecimal form of %d is %x \n", 59, 59);
Hexadecimal form of 59 is 3Bprintf("Square root of 2 is %f \n", sqrt(2.0));
Square root of 2 is 1.414214
(C) Doug Bowman, Virginia Tech, 2001 21
Learning by Exampleprintf("Square root of 157 is %e \n",sqrt(157.0));
Square root of 157 is 1.252996e+01printf("You scored %d out of %d for %d%%.\n",17, 25, 68);
You scored 17 out of 25 for 68%.
(C) Doug Bowman, Virginia Tech, 2001 22
Input: Using scanfSyntax
scanf ( control string , argument list )Function Prototype: int scanf( const char, *format, ... ) ;
SemanticsReturns number of variables assigned values or EOF if an error.Argument list expressions are pointer expressions converted according to the corresponding formatter in the control string.Standard input file stdin (keyboard) supplies the input stream.
(C) Doug Bowman, Virginia Tech, 2001 23
Learning by Examplescanf(“%c%c%s”, &c1, &c2, s);
/* Input echo */printf(“\“%c%c %s\””, c1, c1, s);Input Data...into that good night
Output
“.. .into that good night”
(C) Doug Bowman, Virginia Tech, 2001 24
Final Thoughts:printf() & scanf() have options other than those covered herein.knowledge of printf() & scanf() is required to understand legacy C code, but they should be avoided when possible, however, we are using them for practice!fprintf and fscanf are the file versions of the same calls: see website for slides
5
(C) Doug Bowman, Virginia Tech, 2001 25
Simple file I/O example using FILEs and meaningful
datatypes
1
System programming: process management and IPC
CS 2204Class meeting 12
*Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright 2001-2003.
(C) Doug Bowman, Virginia Tech, 2001 2
UNIX system programmingProgramming that uses special features of the UNIX systemPrograms make system callsTypes of system calls
File I/OProcess managementInter-process communication (IPC)Signal handling
(C) Doug Bowman, Virginia Tech, 2001 3
Processes in UNIXProcess: basic unit of execution
Executing instance of a programHas a process ID (PID)Is in a hierarchy of processes (parents, children)Has its own state/context/memory
Shell commands dealing with processes: ps, top, kill, nice, …
(C) Doug Bowman, Virginia Tech, 2001 4
Process managementSystem calls dealing with:
Process creationSetting the program a process executesWaiting for a process to terminateProcess terminationSending signals to a process
(C) Doug Bowman, Virginia Tech, 2001 5
Process creationpid = fork()Creates a new child process that is an exact copy of the current process
Same program running at same locationSame variable valuesSame open files
Only difference: child has new PIDReturns 0 in the child processReturns child’s PID in the parent process
(C) Doug Bowman, Virginia Tech, 2001 6
Setting the program a process executesexec family of functionse.g. execlp(executable_name, arg0, arg1, …);Replaces the current process with a new process image running the executable specified with the arguments listedNew process retains old PID and any open filesOther functions: execl, execle, execv, execvp, execve
2
(C) Doug Bowman, Virginia Tech, 2001 7
Waiting for a process to terminatepid = wait(&status)
Suspends execution of the calling process until any child process terminatesReturns PID of terminating childPuts exit status (int) of child in status
pid = waitpid(pid, &status, options)
Suspends execution of the calling process until a specific child terminates
(C) Doug Bowman, Virginia Tech, 2001 8
Using fork/exec/wait togetherpid = fork();if(pid == 0)execl(“./program”, “program”,
arg1, NULL);elsepid = wait(&status);
continue execution
(C) Doug Bowman, Virginia Tech, 2001 9
Process terminationexit(status)Terminates the calling processCloses all open file descriptorsReturns status to the parent process
(C) Doug Bowman, Virginia Tech, 2001 10
Sending signals to a processretval = kill(pid, signal)Some common signals a user program might send:
Information passing between processesTwo basic paradigms
Message passing: processes send information back and forth in messages/packetsShared Memory: processes share a chunk of physical memory and read/write data there to share that information
(C) Doug Bowman, Virginia Tech, 2001 12
IPC with pipesExample of message passingint fds[2]; retval = pipe(fds);Creates two file descriptors (a pipe is a file), the first for reading, and the second for writingHow does another process connect to this pipe?
NOTES:data is written/read in order (FIFO)reads block until there’s something to readwrites block if the pipe is fullclosing the writing fd causes EOF to be read on the other end
1
sed, awk, & perl
CS 2204Class meeting 13
*Notes by Mir Farooq Ali and other members of the CS faculty at Virginia Tech. Copyright 2003.
Use patterns/regular expressions rather than explicitly specifying line numberssed –n ‘/^From: /p’ $HOME/mbox
retrieve all the sender lines from the mailbox file“For each line, if that line starts with ‘From’, print it.” Note that the / / mark the beginning and end of the pattern to matchls –l | sed –n ‘/^.....w/p’“For each line, if the sixth character is a W, print”
Tedious to type in commands at the prompt, especially if commands are repetitiveCan put commands in a file and sed can use themsed –f cmds.sed data.txt
Powerful pattern scanning and processing languageNames after its creators Aho, Weinberger and Kernighan (Don’t you love how commands are named?)Most commands operate on entire line
"Practical Extraction and Reporting Language"written by Larry Wall and first released in 1987rumour: name came first, then the acronym"Perl is a language for easily manipulating text, files and processes": originally aimed at systems administrators and developers
enables quick development of programsno need to define variable typesportableextensible (module import/export mechanism)powerful "regular expression" capabilitiessimple I/O modelmany modulessupport for static scopingbuilt-in debugger
"bang path" convention for scripts:can invoke Perl at the command line, oradd #!/public/bin/perl at the beginning of the scriptexact value of path depends upon your platform (use "which perl" to find the path)
From the command line:% perlprint "Hello, World!\n";CTRL-DHello, World!
kinds of variable:scalars, lists, "hashes" (also called "associative arrays" or "dictionaries")some rudimentary support for object-orientation, but not really designed as an OOP languageadvanced perl supports pointers, user-defined structures, subroutine references
/etc/inittab and /etc/init.d/inittab: configuration file for the init process
Defines “run levels”init.d: Directory containing system initialization scripts
Script rc <n> is run at run level nStarts and stops various services
(C) Doug Bowman, Virginia Tech, 2001 9
Configuring the systemOften by directly editing files with a text editorIn some cases there are programs that modify the files for youMany systems also have nice graphical user interfaces that let you manipulate these files indirectly
(C) Doug Bowman, Virginia Tech, 2001 10
File System Implementation
A possible file system layout
(C) Doug Bowman, Virginia Tech, 2001 11
Internal View of a File SystemBoot Block:
The first block in a UNIX file system, contains the boot program and other initialization information or unused.
Super BlockAlways the second block, contains the complete “catalog” of specific information about the file system, including lists of free memory
(C) Doug Bowman, Virginia Tech, 2001 12
Internal View of a File SystemInode list blocks
List of inodes for the file system, contiguous and always follows the super block. The number ofinodes is specified by the system administrator
File access and type information, collectively known as the mode. File ownership information. Time stamps for last modification, last access and last mode modification. Link count. File size in bytes. Addresses of physical blocks.
3
(C) Doug Bowman, Virginia Tech, 2001 13
Internal View of a File SystemData blocks
OS files, user data and program files etc.
(C) Doug Bowman, Virginia Tech, 2001 14
File system Commandsmount
mount a file systemumount
unmount a file systemfsck
check and repair a Linux file systemsync
flush filesystem buffers
(C) Doug Bowman, Virginia Tech, 2001 15
crontabUseful to have a script or command executed without human intervention
a script to verify that the networks are working correctly
cron daemon reads cron configuration files called “crontab” short for “cron table”
parse crontabsfind the soonest command to be rungo to sleep until the command’s execution time has arrived
(C) Doug Bowman, Virginia Tech, 2001 16
What’s cron and crontabs?Under UNIX, periodic execution is handled by the cron daemon
read one or more configuration files containing as following
command linestimes at which they are to be invoked(on some systems) login names under which they are to run
crontabs/etc/crontab
(C) Doug Bowman, Virginia Tech, 2001 17
Format of Crontab filesSeven or six fields
minute hour day month weekday [username] command
an asterisk matches all possible values, a single integer matches that exact value, a list of integers separated by commas (no spaces) used to match any one of the values two integers separated by a dash (a range)used to match any value within the range
Read all required materialFinished all labsCompleted all assignmentsAttended all lecturesReviewed all notes
(C) S. S. Gifford, Virginia Tech, 2003- 3
Study Hints cont.When writing the final, I
Focused on the details that support major topicsDoes not mean there are no detailsDoes mean that I didn’t try to major on the minors
Did not focus on all commands & optionsDid not focus on C syntaxTried to cover things not in the projectsA few questions assume you attended class!A few questions are from the labsA few questions are from the readings
(C) S. S. Gifford, Virginia Tech, 2003- 4
Study Hints cont.3questions, 5.?? points each, total=~190Multiple choice, true/false, and other innovative scantron questions:
5.) What is the first digit of the number between 14 and 16?6.) What is the second digit of the number between 14 and 16?7.) How many of the following choices are correct? (Enter a number between 0-9.)
(C) S. S. Gifford, Virginia Tech, 2003- 5
No Extra Credit on Exam2 hour test, most will complete in <1h(You do have the extra credit assignment you can do, remember?)
(C) S. S. Gifford, Virginia Tech, 2003- 6
What should you know?What is UNIX vs. LinuxWho is Ken Thompson and Dennis Richie?All commands that were covered outside of just the readings