Page 1
Networking
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
1 / 43
Page 2
Networking and Operating Systems
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
2 / 43
■ Many different pieces■ Some pieces are in the kernel; others are in
user space■ Apart from the division, we need appropriate
interfaces
Page 3
Kinds of Networking
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
3 / 43
■ Different protocols; different protocol suites■ User versus kernel consumption■ Synchronous versus asynchronous
Page 4
Applications
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
4 / 43
■ Service■ Servers■ Clients■ Peer-to-peer■ Which are part of the OS?
Page 5
The Stack
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
5 / 43
■ Seven layers: physical, link, network, transport,session, presentation, application
■ Well, not really, on the Internet■ Link layer — device drivers■ Network: IP (Internet Protocol)■ Transport (and a bit of session): TCP, UDP■ Presentation, application: applications■ All but the last are in the kernel
Page 6
Protocol Suites
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
6 / 43
■ TCP/IP — The Internet■ Subclass: IPv6■ OSI■ Novell IPX■ Appletalk■ NetBIOS■ Many others that have faded from the scene
Page 7
Connection-Oriented or
Connectionless?Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
7 / 43
■ Some protocols are connection-oriented —once things are set up, you always talk to asingle endpoint
■ Example: TCP■ Others are connectionless — each packet can
go to or come from a different place■ Example: UDP, as used in the DNS
Page 8
Mux/Demux
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
8 / 43
■ Many layers of multiplex/demultiplex■ Link layer selects different network layers (and
maybe different protocol suites)■ Network layer selects different transports■ Transport layer selects different applications■ User space can read all layers
Page 9
Layers
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
9 / 43
Link
Network Network
TransportTransport
Application
Page 10
Application Access
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
10 / 43
■ Why do applications have access to all layers?■ Debugging■ Implement layer at user level
Page 11
Example: tcpdump at Link Layer
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
11 / 43
# tcpdump -v -v -s 1500 not ip and not
arp
tcpdump: listening on bge0, link-type
EN10MB (Ethernet), capture size 1500
bytes 23:05:57.642505 00:48:54:71:ce:32
> Broadcast null I (s=0,r=0,C) len=42
Page 12
Example: DHCP
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
12 / 43
■ DHCP is used to assign IP addresses to hosts■ At the time a host issues a DHCP request, it
has no IP address, so it can’t speak IP■ Both the DHCP client and the DHCP server
have to listen – and speak – at link level
Page 13
Example: OSPF
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
13 / 43
■ The OSPF routing protocol runs directly ontop of IP
■ To implement this at user level, the programhas to read IP packets
Page 14
Example: ping
Networking
Networking andOperating Systems
Kinds of Networking
Applications
The Stack
Protocol SuitesConnection-Orientedor Connectionless?
Mux/Demux
Layers
Application Access
Example: tcpdumpat Link Layer
Example: DHCP
Example: OSPF
Example: ping
Interfaces
Applications
Middleware
The Global Grid
14 / 43
■ ping uses ICMP messages■ ICMP lives directly on top of IP■ In other words, ICMP packets have to be
available both in the kernel and at user level
Page 15
Interfaces
Networking
Interfaces
InterfacesNetworkConnections Aren’tFiles
The Socket Interface
Sockets and Layers
ConnectionlessNetworksCouldn’t We JustOpen /dev/tcp?
Interfaces —Summary
Applications
Middleware
The Global Grid
15 / 43
■ What is the interface to the networking stack?■ Is it file-like? Something special?■ The answer, of course, is “it depends”
Page 16
Network Connections Aren’t Files
Networking
Interfaces
InterfacesNetworkConnections Aren’tFiles
The Socket Interface
Sockets and Layers
ConnectionlessNetworksCouldn’t We JustOpen /dev/tcp?
Interfaces —Summary
Applications
Middleware
The Global Grid
16 / 43
■ We always do much more than just read orwrite
■ We often have to pass extra information, suchas source or destination address forconnectionless protocols
■ But sometimes, we do just read and write. . .
Page 17
The Socket Interface
Networking
Interfaces
InterfacesNetworkConnections Aren’tFiles
The Socket Interface
Sockets and Layers
ConnectionlessNetworksCouldn’t We JustOpen /dev/tcp?
Interfaces —Summary
Applications
Middleware
The Global Grid
17 / 43
■ Due originally to Berkeley, circa 1983■ Network access is initiated by the socket()
system call■ On Unix, socket() returns an ordinary file
descriptor; you can (eventually) doread()/write()/close()
■ On Windows, it returns a special type of filedescriptor; you can only do special socketoperations, such as send()/recv()
Page 18
Sockets and Layers
Networking
Interfaces
InterfacesNetworkConnections Aren’tFiles
The Socket Interface
Sockets and Layers
ConnectionlessNetworksCouldn’t We JustOpen /dev/tcp?
Interfaces —Summary
Applications
Middleware
The Global Grid
18 / 43
■ Sockets are used at all layers — parameters onthe socket() call specify the layer andprotocol
■ The semantics of the returned file descriptorare layer-dependent
■ A variety of options can be set withsetsockopt() and ioctl()
■ Sockets are also used for configuration control,such as assigning IP addresses to interfaces
■ You control the routing tables by writing to aspecial socket
Page 19
Connectionless Networks
Networking
Interfaces
InterfacesNetworkConnections Aren’tFiles
The Socket Interface
Sockets and Layers
ConnectionlessNetworksCouldn’t We JustOpen /dev/tcp?
Interfaces —Summary
Applications
Middleware
The Global Grid
19 / 43
■ Every packet written needs a destinationaddress as well as data
■ Every packet read contains a source address aswell as data
■ Use sendto()/recvfrom()■ (Address format will vary, depending on the
type of network; some networks even usevariable-length addresses
Page 20
Couldn’t We Just Open /dev/tcp?
Networking
Interfaces
InterfacesNetworkConnections Aren’tFiles
The Socket Interface
Sockets and Layers
ConnectionlessNetworksCouldn’t We JustOpen /dev/tcp?
Interfaces —Summary
Applications
Middleware
The Global Grid
20 / 43
■ On some systems, such as Solaris, you can!■ But — we still need special operations■ setsockopt() could just be an ioctl, but
accept() returns a new file descriptor■ Connectionless networks use special data
format
Page 21
Interfaces — Summary
Networking
Interfaces
InterfacesNetworkConnections Aren’tFiles
The Socket Interface
Sockets and Layers
ConnectionlessNetworksCouldn’t We JustOpen /dev/tcp?
Interfaces —Summary
Applications
Middleware
The Global Grid
21 / 43
■ Network connections aren’t really files■ Given all the special stuff that has to go on
anyway, there’s little advantage to using thefile system instead of sockets
■ There may be some advantage, though, togetting a normal file descriptor
Page 22
Applications
Networking
Interfaces
Applications
Applications
Service Applications
Service Dispatchers
The Port Mapper
The Port Mapper
Other Applications?
Telnet and SSH
Anonymous FTP
Middleware
The Global Grid
22 / 43
■ Service■ Servers■ Clients■ Peer-to-peer
Page 23
Service Applications
Networking
Interfaces
Applications
Applications
Service Applications
Service Dispatchers
The Port Mapper
The Port Mapper
Other Applications?
Telnet and SSH
Anonymous FTP
Middleware
The Global Grid
23 / 43
■ Service applications function as an extension ofthe OS
■ Example: with NFS, the client and server arein the kernel, but other pieces are at user level:mounting and unmount the file system,locking, etc.
■ Example: Routing — IP is pretty uselesswithout it
Page 24
Service Dispatchers
Networking
Interfaces
Applications
Applications
Service Applications
Service Dispatchers
The Port Mapper
The Port Mapper
Other Applications?
Telnet and SSH
Anonymous FTP
Middleware
The Global Grid
24 / 43
■ Certain applications exist just to run otherapplications
■ ineted — Runs most TCP and UDPapplications
■ (Original idea was to keep the process tablesmall, to improve performance. Now, itpermits (some) servers to just usestdin/stdout)
■ Example: portmapper — dispatches inboundRPC requests
Page 25
The Port Mapper
Networking
Interfaces
Applications
Applications
Service Applications
Service Dispatchers
The Port Mapper
The Port Mapper
Other Applications?
Telnet and SSH
Anonymous FTP
Middleware
The Global Grid
25 / 43
■ Subprocedures — or rather, their stubs —register with the portmapper
■ The caller’s stubs contact the port mapper tofind out the actual port number for thesubprocedure
■ This is not quite invisible to the applicationprogrammer; at the least, the registration hasto be set up
Page 26
The Port Mapper
Networking
Interfaces
Applications
Applications
Service Applications
Service Dispatchers
The Port Mapper
The Port Mapper
Other Applications?
Telnet and SSH
Anonymous FTP
Middleware
The Global Grid
26 / 43
$ rpcinfo -p cluster.cs.columbia.edu
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100005 3 tcp 32782 mountd
100003 2 udp 2049 nfs
and many more besides
Page 27
Other Applications?
Networking
Interfaces
Applications
Applications
Service Applications
Service Dispatchers
The Port Mapper
The Port Mapper
Other Applications?
Telnet and SSH
Anonymous FTP
Middleware
The Global Grid
27 / 43
■ Most applications have nothing to do with theOS
■ A Web server could run, almost unchanged,just as well on Windows as on Unix
■ (Minor differences in a few system calls)■ But some do authentication
Page 28
Telnet and SSH
Networking
Interfaces
Applications
Applications
Service Applications
Service Dispatchers
The Port Mapper
The Port Mapper
Other Applications?
Telnet and SSH
Anonymous FTP
Middleware
The Global Grid
28 / 43
■ Telnet and SSH permit remote logins■ They have to authenticate users, using
OS-specific mechanisms■ Telnet can invoke login; SSH, which can do
its own authentication, cannot
Page 29
Anonymous FTP
Networking
Interfaces
Applications
Applications
Service Applications
Service Dispatchers
The Port Mapper
The Port Mapper
Other Applications?
Telnet and SSH
Anonymous FTP
Middleware
The Global Grid
29 / 43
■ Ordinary FTP has to authenticate users; seeabove
■ Anonymous FTP has a different problem:confining the remote user
■ Must use OS-specific sandbox mechanism
Page 30
Middleware
Networking
Interfaces
Applications
Middleware
Middleware
Goals of Middleware
Types of Middleware
CORBA
Publish-Subscribe
The Global Grid
30 / 43
■ Common middle layer between applicationsand the stack
■ I.e., common middle ground betweenapplications and the OS
■ Arguably as much a part of the OS as the Cruntime library
Page 31
Goals of Middleware
Networking
Interfaces
Applications
Middleware
Middleware
Goals of Middleware
Types of Middleware
CORBA
Publish-Subscribe
The Global Grid
31 / 43
■ Common interface■ Naming■ Replication■ Access
Page 32
Types of Middleware
Networking
Interfaces
Applications
Middleware
Middleware
Goals of Middleware
Types of Middleware
CORBA
Publish-Subscribe
The Global Grid
32 / 43
■ Document-based: the Web■ File-based: distributed file system■ Object-based: CORBA■ Publish-subscribe■ More. . .
Page 33
CORBA
Networking
Interfaces
Applications
Middleware
Middleware
Goals of Middleware
Types of Middleware
CORBA
Publish-Subscribe
The Global Grid
33 / 43
■ CORBA: Common Object Request Broker
Architecture
■ RPC extended procedure calls to the net■ CORBA extends objected-oriented
programming to the net■ Instead of referencing files or web pages, you
invoke methods■ The CORBA Object Broker handles naming,
location, access method, etc.
Page 34
Publish-Subscribe
Networking
Interfaces
Applications
Middleware
Middleware
Goals of Middleware
Types of Middleware
CORBA
Publish-Subscribe
The Global Grid
34 / 43
■ Processes that have information publish it■ On a LAN, implemented as a broadcast■ Information router forwards such broadcasts to
other interested LANs■ If a process subscribes to a topic, its local
information broker tells other LANs it wantssuch data
Page 35
The Global Grid
Networking
Interfaces
Applications
Middleware
The Global Grid
The Global Grid
It’s a Distributed OSActually, There AreMany Grids
Scenarios
Security Issues
Sandboxing
Storage Space
Accounting
Other EssentialComponents
35 / 43
■ Share computing resources around the world■ (The original goal of the ARPANET!)■ Built on high performance computers and high
performance networks
Page 36
It’s a Distributed OS
Networking
Interfaces
Applications
Middleware
The Global Grid
The Global Grid
It’s a Distributed OSActually, There AreMany Grids
Scenarios
Security Issues
Sandboxing
Storage Space
Accounting
Other EssentialComponents
36 / 43
■ The Grid functions like a large-scale distributedoperating system
■ It has to solve all the distributed OS problemswe talked about
■ These include security, scheduling, locking,communications, and more
Page 37
Actually, There Are Many Grids
Networking
Interfaces
Applications
Middleware
The Global Grid
The Global Grid
It’s a Distributed OSActually, There AreMany Grids
Scenarios
Security Issues
Sandboxing
Storage Space
Accounting
Other EssentialComponents
37 / 43
■ The Grid is a concept and a set of protocols,not a single Internet-wide virtual machine
■ You and your friends can pool your ownmachines
■ Each Grid sets up its own access control policy
Page 38
Scenarios
Networking
Interfaces
Applications
Middleware
The Global Grid
The Global Grid
It’s a Distributed OSActually, There AreMany Grids
Scenarios
Security Issues
Sandboxing
Storage Space
Accounting
Other EssentialComponents
38 / 43
■ Specific computer; local I/O only — simplyhave to authenticate to host computer
■ Specific computer; remote I/O — mustdelegate security credentials to host OS toallow file retrieval and update
■ Run on “best” computer — must talk toscheduler and delegate credentials; schedulermust verify acceptability of credentials on eachcandidate machine
■ Multiprocessing — each remote job needscredentials to talk to other pieces
Page 39
Security Issues
Networking
Interfaces
Applications
Middleware
The Global Grid
The Global Grid
It’s a Distributed OSActually, There AreMany Grids
Scenarios
Security Issues
Sandboxing
Storage Space
Accounting
Other EssentialComponents
39 / 43
■ User authentication — straight-forward■ Must protect grid computers from remote
users — standard operating system problem■ Must have large-scale, secure, distributed file
system■ Users want to protect their data from the
remote OS — hard!
Page 40
Sandboxing
Networking
Interfaces
Applications
Middleware
The Global Grid
The Global Grid
It’s a Distributed OSActually, There AreMany Grids
Scenarios
Security Issues
Sandboxing
Storage Space
Accounting
Other EssentialComponents
40 / 43
■ If untrusted users are running jobs on yourmachine, you may want to sandbox them
■ But each user needs different resources; youneed a flexible sandbox
■ You want to isolate each remote user fromevery other, and (if possible) from your localusers
Page 41
Storage Space
Networking
Interfaces
Applications
Middleware
The Global Grid
The Global Grid
It’s a Distributed OSActually, There AreMany Grids
Scenarios
Security Issues
Sandboxing
Storage Space
Accounting
Other EssentialComponents
41 / 43
■ The Grid is for big problems; these take a lotof storage space
■ How do we manage allocation?■ How do we transfer that much over the net?■ How do we encrypt that much data during
storage?
Page 42
Accounting
Networking
Interfaces
Applications
Middleware
The Global Grid
The Global Grid
It’s a Distributed OSActually, There AreMany Grids
Scenarios
Security Issues
Sandboxing
Storage Space
Accounting
Other EssentialComponents
42 / 43
■ Someone has to pay for resources used■ The Grid includes accounting standards —
record CPU, disk, and RAM usage (andperhaps bandwidth)
Page 43
Other Essential Components
Networking
Interfaces
Applications
Middleware
The Global Grid
The Global Grid
It’s a Distributed OSActually, There AreMany Grids
Scenarios
Security Issues
Sandboxing
Storage Space
Accounting
Other EssentialComponents
43 / 43
■ Name space■ File I/O and format conversion■ File replication and cache engine■ Brokers, which talk to schedulers■ These are middleware