UNIX! Landon Cox September 3, 2012. Dealing with complexity How do you reduce the complexity of large programs? Break functionality into modules Goal.

UNIX!

Landon CoxSeptember 3, 2012

Dealing with complexity

• How do you reduce the complexity of large programs?• Break functionality into modules• Goal is to “decouple” unrelated functions• Narrow the set of interactions between modules• Hope to make whole system easier to reason about

• How do we specify interactions between code modules?• Procedure calls (or objects = data + procedure calls)• int foo(char *buf)

• Procedure calls reduce complexity by • Limiting how modules can interact with one another• Hiding implementation details

Dealing with complexity

int main () { cout << “input: ”; cin >> input; output = sqrt (input); output = pow (output,3); cout << output << endl;}

int main () { getInput (); computeResult (); printOutput ();}

void getInput () { cout << “input: ”; cin >> input;}

void printOutput () { cout << output << endl;}

void computeResult () { output = sqrt (input); output = pow (output,3);}

int P(int a){…}

void C(int x){ int y=P(x); }

How do C and P share information?

Via a shared, in-memory stack

int P(int a){…}

void C(int x){ int y=P(x); }

What info is stored on the stack?

C’s registers, call arguments, RA, P's local vars

Review of the stack

• Each stack frame contains a function’s• Local variables• Parameters• Return address• Saved values of calling function’s registers

• The stack enables recursion

const1=1const2=0main

tmp=1RA=0x804838cA

RA=0x8048361B

const=0RA=0x8048354C

tmp=0RA=0x8048347A

0xfffffff

0x0

Memory

void C () { A (0);}

void B () { C ();}

void A (int tmp){ if (tmp) B ();}

int main () { A (1); return 0;}

0x8048347

0x8048354

0x8048361

0x804838c

Code Stack

…

SP

SP

SP

SP

SP

const1=3const2=0main

bnd=3RA=0x804838cA

bnd=2RA=0x8048361A

bnd=1RA=0x8048361A

bnd=0RA=0x8048361A

0xfffffff

0x0

Memory

void A (int bnd){ if (bnd) A (bnd-1);}

int main () { A (3); return 0;}

0x8048361

0x804838c

Code Stack

…

SP

SP

SP

SP

SP

How can recursion go wrong?Can overflow the stack …Keep adding frame after frame

…

wrd[3]wrd[2]wrd[1]wrd[0]

const2=0

main

b= 0x00234RA=0x804838ccap

0xfffffff

0x0

Memory

void cap (char* b){ for (int i=0; b[i]!=‘\0’; i++) b[i]+=32;}int main(char*arg) { char wrd[4]; strcpy(arg, wrd); cap (wrd); return 0;}

0x8048361

0x804838c

Code Stack

…SP

SP

0x00234What can go wrong?Can overflow wrd variable …Overwrite cap’s RA

…

int P(int a){…}

void C(int x){ int y=P(x); }

Can think of this as a contract

P agrees to returnP agrees to resume where C left offP agrees to restore the stack pointerP agrees to leave rest of stack alone

int P(int a){…}

void C(int x){ int y=P(x); }

Is the call contract enforced?

At a low level, NO!P can violate all terms of the contractSources of violations: attacks + bugs

int P(int a){…}

void C(int x){ int y=P(x); }

Enforcing the contract is feasible

Interaction is purely mechanicalProgrammers intention is clearNo semantic gap to cross

int P(int a){…}

void C(int x){ int y=P(x); }

How does Java enforce the call contract?

Language restricts expressivenessProgrammers can’t access the stackSpecial “invoke” instruction expresses intentJVM trusted to transfer control between C, P

int P(int a){…}

void C(int x){ int y=P(x); }

Awesome, so why not run only Java programs?

Lower-level languages are faster(trusted JVM interposes on every instr)Restricts programmer’s choice(maybe, I hate programming in Java)

int P(int a){…}

void C(int x){ int y=P(x); }

Another approach to enforced modularity

Put C and P in separate processesCode is fast when processes not interactingTrust kernel to handle control transfersKernel ensures transitions are correct

int P(int a){…}

void C(int x){ int y=P(x); }

Key question:What should the interface be?

Put C and P in separate processesWant a general interface for inter-process communication (IPC)Should be simple and powerful (i.e., elegant)

UNIX philosophy

• OS by programmers for programmers• Support high-level languages (C and scripting)• Make interactivity a first-order concern (via shell)• Allow rapid prototyping

• How should you program for a UNIX system?• Write programs with limited features

• Do one thing and do it well• Support easy composition of programs

• Make data easy to understand• Store data in plaintext (not binary formats)• Communicate via text streams Thompson and Ritchie

Turing Award ‘83

UNIX philosophy

ProcessC

ProcessP

Kernel

?

What is the core abstraction?

Communication via files

UNIX philosophy

ProcessC

ProcessP

Kernel

What is the interface? Open: get a file reference (descriptor)Read/Write: get/put dataClose: stop communicating

File

UNIX philosophy

ProcessC

ProcessP

Kernel

Why is this safer than procedure calls? Interface is narrower

Access file in a few well-defined waysKernel ensures things run smoothly

File

UNIX philosophy

ProcessC

ProcessP

Kernel

How do we transfer control to kernel? Special system call instruction!

CPU pauses process, runs kernelKind of like Java’s invoke instruction

File

UNIX philosophy

ProcessC

ProcessP

Kernel

Key insight: Interface can be used for lots of thingsPersistent storage (i.e., “real” files)Devices, temporary channels (i.e., pipes)

File

UNIX philosophy

ProcessC

ProcessP

Kernel

Two questions (1) How do processes start running?(2) How do we control access to files?

File

Course administration

• Heap manager project• Due a week from Friday• Sorry, but I can’t help you …• Questions for Vamsi?

• Piazza• Should have received account info• Email Jeff if not

• Other questions?

UNIX philosophy

ProcessC

ProcessP

Kernel

Two questions (1) How do processes start running?(2) How do we control access to files?

File

UNIX philosophy

ProcessC

ProcessP

Kernel

Two questions (1) How do processes start running?

File

UNIX philosophy

ProcessC

ProcessP

Kernel

Maybe P is already running?

Could just rely on kernel to start processes

File

UNIX philosophy

ProcessC

ProcessP

Kernel

File

What might we call such a process?

Basically what a server isA process C wants to talk to that someone else launched

UNIX philosophy

ProcessC

ProcessP

Kernel

All processes shouldn’t be servers

Want to launch processes on demandC needs primitives to create P

File

UNIX Shell

Shell

Kernel

Program that runs other programs

Interactive (accepts user commands)Essentially just a line interpreterAllows easy composition of programs

UNIX shell

• How does a UNIX process interact with a user?• Via standard in (fd 0) and standard out (fd 1)• These are the default input and output for a program• Establishes well-known data entry and exit points for a program

• How do UNIX processes communicate with each other?• Mostly communicate with each other via pipes• Pipes allow programs to be chained together• Shell and OS can connect one process’s stdout to another’s stdin

• Why do we need pipes when we have files?• Pipes create unnamed temporary buffers between processes• Communication between programs is often ephemeral• OS knows to garbage collect resources associated with pipe on exit• Consistent with UNIX philosophy of simplifying programmers’ lives

UNIX shell

• Pipes simplify naming• Program always receives input on fd 0• Program always emits output on fd 1• Program doesn’t care what is on the other end of fd• Shell/OS handle input/output connections

• How do pipes simplify synchronization?• Pipe accessed via read system call• Read can block in kernel until data is ready• Or can poll, checking to see if read returns enough data

How kernel starts a process1. Allocates process control block (bookkeeping data structure)2. Reads program code from disk3. Stores program code in memory (could be demand-loaded too)

4. Initializes machine registers for new process5. Initializes translator data for new address space

• E.g., page table and PTBR• Virtual addresses of code segment point to correct physical locations

6. Sets processor mode bit to “user”7. Jumps to start of program

Needhardwaresupport

Creating processes

• Through what commands does UNIX create processes?• Fork: create copy child process• Exec: initialize address space with new program

• What’s the problem of creating an exact copy process?• Child needs to do something different than parent• i.e., child needs to know that it is the child

• How does child know it is child?• Pass in return point• Parent returns from fork call, child jumps into other region of code• Fork works slightly differently now

Fork

• Child can’t be an exact copy• Is distinguished by one variable (the return value of fork)

if (fork () == 0) { /* child */ execute new program} else { /* parent */ carry on }

Creating processes

• Why make a complete copy of parent?• Sometimes you want a copy of the parent• Separating fork/exec provides flexibility• Allows child to inherit some kernel state• E.g., open files, stdin, stdout• Very useful for shell

• How do we efficiently copy an address space?• Use “copy on write”• Make copy of page table, set pages to read-only• Only make physical copies of pages on write fault

Copy on writePhysical memory

Parent memory

Child memory

What happens if parent writes to a page?

Copy on write

Child memory

Have to create a copy of pre-write page for the child.

Physical memory

Parent memory

Alternative approach

• Windows CreateProcess• Combines the work of fork and exec

• UNIX’s approach • Supports arbitrary sharing between parent and child

• Window’s approach• Supports sharing of most common data via params

Shells (bash, explorer, finder)

• Shells are normal programs• Though they look like part of the OS

• How would you write one?while (1) { print prompt (“crocus% “) ask for input (cin) // e.g., “ls /tmp” first word of input is command // e.g., ls fork a copy of the current process (shell) if (child) { redirect output to a file if requested (or a pipe) exec new program (e.g., with argument “/tmp”) } else { wait for child to finish or can run child in background and ask for another command }}

Shell demo

UNIX philosophy

ProcessC

ProcessP

Kernel

Two questions (1) How do processes start running?(2) How do we control access to files?

File

UNIX philosophy

ProcessC

ProcessP

Kernel

Two questions (1) How do processes start running?(2) How do we control access to files?

File

Access control

• Where is most trusted code located?• In the operating system kernel

• What are the primary responsibilities of a UNIX kernel?• Managing the file system• Launching/scheduling processes• Managing memory

• How do processes invoke the kernel?• Via system calls• Hardware shepherds transition from user process to kernel• Processor knows when it is running kernel code• Represents this through protection rings or mode bit

Access control

• How does kernel know if system call is allowed?• Looks at user id (uid) of process making the call• Looks at resources accessed by call (e.g., file or pipe)• Checks access-control policy associated with resource• Decides if policy allows uid to access resources

• How is a uid normally assigned to a process?• On fork, child inherits parent’s uid

MOO accounting problem

• Multi-player game called Moo• Want to maintain high score in a file

• Should players be able to update score?• Yes

• Do we trust users to write file directly?• No, they could lie about their score

High score

Game client(uid y)

Game client (uid x)

“x’s score = 10”

“y’s score = 11”

MOO accounting problem

• Multi-player game called Moo• Want to maintain high score in a file

• Could have a trusted process update scores

• Is this good enough?

High score

Game client(uid y)

Game client (uid x)

Game server

“x’s score = 10”

“y’s score = 11”

“x:10y:11”

MOO accounting problem

• Multi-player game called Moo• Want to maintain high score in a file

• Could have a trusted process update scores

• Is this good enough?• Can’t be sure that reported score is genuine• Need to ensure score was computed correctly

High score

Game client(uid y)

Game client (uid x)

Game server

“x’s score = 100”

“y’s score = 11”

“x:100y:11”

Access control

• Insight: sometimes simple inheritance of uids is insufficient• Tasks involving management of “user id” state• Logging in (login)• Changing passwords (passwd)

• Why isn’t this code just inside the kernel?• This functionality doesn’t really require interaction w/ hardware• Would like to keep kernel as small as possible

• How are “trusted” user-space processes identified?• Run as super user or root (uid 0)• Like a software kernel mode• If a process runs under uid 0, then it has more privileges

Access control

• Why does login need to run as root?• Needs to check username/password correctness• Needs to fork/exec process under another uid

• Why does passwd need to run as root?• Needs to modify password database (file)• Database is shared by all users

• What makes passwd particularly tricky?• Easy to allow process to shed privileges (e.g., login)• passwd requires an escalation of privileges

• How does UNIX handle this?• Executable files can have their setuid bit set• If setuid bit is set, process inherits uid of image file’s owner on exec

MOO accounting problem

• Multi-player game called Moo• Want to maintain high score in a file

• How does setuid solve our problem?• Game executable is owned by trusted entity• Game cannot be modified by normal users• Users can run executable though• High-score is also owned by trusted entity

• This is a form of trustworthy computing• Only trusted code can update score• Root ownership ensures code integrity• Untrusted users can invoke trusted code

High score

Game client(root)

Gameclient (root)

Shell(uid y)

Shell(uid x)

“fork/execgame”

“fork/execgame”

“x’s score = 10”

“y’s score = 11”