CIS 191: Linux and Unix Class 1 January 28 th, 2014.

CIS 191: Linux and Unix

Class 1January 28th, 2014

Outline

Ubuntu Basics

Text Editors

More Command Line Tools

Piping and Redirection

xkcd/456/

How To: Install Software

• Ubuntu uses dpkg to install software. This is meant to run on .deb package files. In Ubuntu, this is low-level.

• Users can interact with the package management system via apt and/or aptitude. – apt is installed by default.

• Example: Installing aptitude (requires root priveleges)– $ sudo apt-get install aptitude

• Example: Details about the gvim package (w/ aptitude)– $ aptitude search gvim– This does not require root.

The Ubuntu Software Center

• A graphical tool for interacting with packages– synaptic -> mid-level interface for dpkg– Ubuntu Software Center is a high-level way to install official

Ubuntu packages– Some fun programs in here; take a look!

Interacting with the Terminal

• Two popular programs:– gnome-terminal is installed by default on Ubuntu (and is very

pretty)– xterm is available on most Unix derivatives

• There are of course plenty of other options…• Terminal emulators tend to come with standard features

– Shift-PgUp/Shift-PgDn – scroll up/scroll down– tab is used for automatic completion

• supercali<tab> -> supercalifragilisticexpealadocious– Supports keybindings from many editors (with proper

configurations!)

Outline

Ubuntu Basics

Text Editors

More Command Line Tools

Piping and Redirection

The Age-Old Debate

xkcd.comm/378

Vi vs. Emacs

• Two of the most popular text editors• These have sparked much debate over the years• Originally…

– VI was a lightweight mode-based text editor– emacs was designed to be easy to use from the start and

feature-complete while being easily extensible

• Each now incorporates many features found in the other

Why use a Text Editor?

• To save time!– Save seconds to do small tasks

• If you spend a lot of time editing text (you will!), this adds up…

• Impressive macro support in both vi and emacs• Many libraries available to include in both editors as well• Not about typing text; about editing text• Customization through add-ons and configuration files

– .vimrc or .emacs

Emacs basics

• Uses modifier keys to tell it explicit commands– Control (abbreviated C): Ctrl– Meta (Abbreviated M): Alt or Esc

• Hold Alt and press key or press and release Esc, then press key

• Common tasks:– Type to enter text– C-f to move forwards, C-b for back, C-p to prev line, C-n to next– C-_ to undo– C-x C-s saves current buffer– C-x C-c to quit emacs

• Get to tutorial with C-h t

Why Emacs?

• Has the default keybindings for many command line programs– C-r will do a reverse search of your bash history, for example

• Emacs is non-modal– The same command will have the same behavior, always

• Emacs commands have a hierarchical structure, kind of• Emacs commands are named• Emacs contains a lisp implementation• Running code from within emacs is easy

In contrast - Vim

• Vim is very much a modal text editor– All commands vary based on which mode you are in!

• Keybindings arranged to make actions easy to type– Typical actions require typing one key in normal mode

• Available by default on highly slimmed-down distributions (more commonly than emacs)

• man, less, and other programs use vi-like keybindings• vi-like keybindings are available on several web services

– Gmail– Duck duck go

Vi modes

• Four modes (actually more, but we’ll stick with these)– Normal, awaiting commands (movement, pasting, deleting…)– Insert, awaiting text to be inserted– Visual, used for selecting blocks of text (like highlighting)– Replace, used to enter text without affecting formatting

• “A good way to get gibberish is to give a vi window to someone who doesn’t know how to use it”

• Press Esc or Ctrl-[ to return to Normal mode– If you are ever unsure of what’s happening, press Esc a bunch of

times!

Vim Philosophy

• Your hands should move as little as possible– Move with h,j,k,l

• Remain in Normal mode as much as possible– You spend most of the time moving around a document…– You’ll see this is true if you pay attention!

Learning Vim

• Official vim tutor program can be invoked from the command line with vimtutor.

• Access vim help system by typing :help [topic] within vim• There are even games to help you learn vim keybindings!

– vim-adventures.com, for example (in your homework this week)

Outline

Ubuntu Basics

Text Editors

More Command Line Tools

Piping and Redirection

pwd – Where you are in the file tree

• This simply prints the absolute path of the current working directory to the command line

• Useful for remembering where you are (because if you’re anything like me you’ll jump around and kinda get lost)

less and more – view files

• less – browse through files– Features forward and backward browsing– Can open large files by only loading the necessary parts into the

buffer for display• For large files, using less can be faster than an editor like vim!

– In fact, the man command uses less to display file contents!

• more is worse (less is more – haha get it??)– Doesn’t let you move backward– Has to load the entire file into the buffer before starting

• This can be very slow for big files

head and tail – see the top/end

• head lets you see the top 10 lines of a file– Specify more or less with the –n option

• tail lets you see the bottom 10 lines of a file– Specify more or less with the –n option– In BSD (like Mac OSX) systems, tail has a –r option which lets

you print the output in reverse order• This does not hold true in debian (like Ubuntu) based systems!• Debian-based systems have tac (see man pages)

file – Diagnose File Type

• Guesses the file type of a given file (passed in as command-line arguments)– This is done by looking at the “magic number” of a given file– “Magic numbers” are essentially a marker at the beginning of a

file in unix systems which indicate the type of the file– Think of them as the unix equivalent of file extensions in

Microsoft Word, but a little different

wc – count the number of words

• wc counts the number of words in a given input file or string

• It can also count number of characters, number of lines, etcetera

• See the man page for more details

nl – number lines

• Takes the lines in an input file, or as passed in on standard input, and numbers each line in increasing order– Starting from 1

sort – sort input

• sort is a simple tool for sorting lines of text files or text given in standard input

• See the man page for more details

find – Search Recursively for a File

• Executing this command on a directory will search for a file with constraints passed in as flags/arguments– There are many of these! See the man page for more details

• Some examples– Find files in dir with the name file.txt

• $ find dir –name ‘file.txt’• Use the –iname flag instead for case insensitive searches

– Find files ending with .jpg in both dir1 and dir2• find dir1 dir2 –iname ‘.jpg’

– Find files in dir that were modified in the last 7 days• find dir –mtime -7

Aside – Regular Expression Preview

• A means of specifying a formula for a language…– Regular expressions are to a grammar what a class is to an

object (kindasorta)

• * means any number of any character in a language…– So *.jpg can be cat.jpg or cats.jpg or fatpants.jpg– Or we can do cat*s.jpg

• Can be catdogs.jpg or catfs.jpg or…

• More on regular expressions in a later lecture• For now, make use of the useful * in your file commands!

grep – Search a File

• grep allows you to apply a basic regular expression to the contents of one or more input files, as specified in the arguments passed in to grep

• The given regular expression will be applied to each line in the file, and matching lines will be printed to standard output– One match on the line should be enough to make it print

• Can specify case insensitive with the –i flag• Example

– grep ‘*cheezburger’ lolcats.txt

Wildcards

• In the shell, you can refer to multiple files or directories at once by using “wildcards”

• In particular, you can use– *

• If you want to refer to “any amount of any character”– ?

• If you want to refer to “any character, but just one”

Outline

Ubuntu Basics

Text Editors

More Command Line Tools

Piping and Redirection

A bit about shells

• The Shell is responsible for executing and managing programs, as well as displaying their output– You can think of it as a line of communication between user and

Operating System

• Provides an environment and parameters to each program it runs/is running

Shells – A history

• In the beginning, users could only interact with the operating system via the shell!

• Nowadays, we use bash (Bourne-Again Shell)• In a graphical user interface (GUI), you will interact with

the shell via a terminal (e.g. gnone-terminal or xterm)

Lines of communication

• Four main channels of communication between processes– Not including mutexes, semaphores, and signals!

1. Command Line Arguments– Provided only when a program starts up

2. Standard Input– A source of input requested at runtime

3. Standard Output– Where to send (most) output

4. Standard Error– Reserved by convention for error-related output

More on CLI Arguments

• Programs can also accept file paths or an input string as a command line argument

• Command line arguments can be thought of as strings that are passed in to a program when it is invoked

• Ever write a program that accepts command line arguments, and then if they are not available, ask for input from the user via the keyboard?– Maybe in Java?

• Same concept applies with unix executable programs

Aside: An important signals

• Signals – like a message sent to a running process (more on these later)

• Two important signals:– C-c sends SIGINT, which asks a program to stop executing– C-\ sends QUIT, which tells a program to stop executing and

dump its core• If C-c is a baseball bat, C-\ is a medieval mace with spikes on it

We’ve seen these in action already

• CLI Arguments– ls me.html treats me.html as a command line argument

• Standard Output– echo prints on standard output…

• Standard Input– Invoking cat without any arguments opens up an interaction

session which reads from standard input• C-d (EOF character) to stop the session

• Standard Error– $ rm nonexistant

• rm: cannot remove ‘nonexistant’:• No such file or directory

Pipes to connect two programs

• The concept of pipes is a fundamental one…• Connects the Standard Output of one program into the

Standard Input of another!• Consider the Unix hypothesis

– Have a bunch of simple command line tools that do one thing well• grep, find, cat, wc (word count), cut (cut out bits of an input), sort

(sort input)

• Consider connecting many simple pieces together to get an infinitely complex pipeline of jobs…

Pipes: An example

• The pipe character is |, the vertical bar above the \ on standard QWERTY keyboards

• To concatenate two files list1 and list2, then sort the result…– $ cat list1 list2 | sort

• This literally concatenates list1 and list2 into a single string, which is then given to sort as standard input– To reiterate – this is not the same as a command-line argument!

sort receives its input via standard input, not via command-line arguments

Another piping example

• Count the number of visible files in a directory• $ ls | wc –l

– ls outputs a list of files to standard output, which is then passed to wc as standard input

– wc then counts the number of words on standard input, and the –l option prints only the number of newlines found

• Viola! The number of visible directories.

An interesting example…

• See the list of users currently logged onto Eniac• cis191@plus:~> who | cut –d‘ ‘ –f1 | sort | uniq

– who prints a list of users logged on to standard output– cut receives this list as input, and prints only the first column

• -d‘ ‘ says to use spaces as a delimeter• -f1 asks for only the first column• See the man page for more details

– sort lexicographically (default) sorts the list given by cut, so that repeated entries are adjacent

– uniq receives sort’s output as input, and filters out all non-unique names

Redirection

• The standard streams (input, output, and error) can all be redirected to or from files

• More on this in the following slides

Redirecting Standard Output

• This can be accomplished with >• Sometimes it’s useful to place program output in a file…• $ echo “in here” > file

– This will overwrite file if it exists, and will create it if it does not– The contents of file will now be “in here”

• If we want to append instead of overwrite, we can use the >> operator

Redirecting Standard Input

• This can be accomplished with <• Rather than typing input, it can come from a file!• Redirecting standard input substitutes the contents of

the source file for what would be typed on standard input– Standard input does not affect command-line arguments!

• Example:– sort < file1.txt behaves the same as sort file1.txt

• Except that in the redirection example, sort does not know which file it is reading from…

Redirecting Standard Error

• This can be accomplished with 2>• Sometimes it’s useful to separate error messages from

normal program output– Especially when you don’t want to see any error messages!

• $ make 2> /dev/null• This sends anything directed as standard error to

/dev/null• Alternatively, redirect standard error to standard output

with 2>&1– Literally, redirects file descriptor 2 (standard error) to file

descriptor 1 (standard output)

/dev/null

• The null device…• Think of it like a black hole that swallows up everything!• http://devnull-as-a-service.com/

Some More Examples

• The following are (mostly) equivalent– cat file | wc– wc < file– wc file

• We can also use pipes in conjunction with redirects– head –n 17 < file | tail –n 7 – ls | sort >> filelist– wc < file > my_wc– grep hello file | nl > my_hello

A note on shell programming

• Almost always, fundamental data types are numbers and compound structures

• In shell scripting, fundamentals are strings, lists of strings, and files!– Bad news: arithmetic is weird…– More on that later

• Because of this, certain programs lend themselves to shell scripting!

• Connecting input and output of multiple programs is also very simple because of the piping and redirection tools

Mapping Input and Output

• In the following graph, to convert from the left side to the right side– Redirections are in black– Other shell constructs are in blue– And programs are in green

• The following diagram is borrowed from Sam Panzer

Mapping Input and Output