Chapter 2: Application layer

Chapter 2: Application layer2.1 Principles of network applications2.2 Web and HTTP2.3 FTP 2.4 Electronic mail

SMTP, POP3, IMAP2.5 DNS

2.6 P2P applications2.7 Socket programming

with TCP2.8 Socket programming

with UDP

Application 2-1

FTP: the file transfer protocol

transfer file to/from remote host client/server model

client: side that initiates transfer (either to/from remote)

server: remote host ftp: RFC 959 ftp server: port 21

file transfer FTPserver

FTPuser

interfaceFTP

client

local filesystem

remote filesystem

user at host

Application 2-2

FTP: separate control, data connections

FTP client contacts FTP server at port 21, TCP is transport protocol

client authorized over control connection

client browses remote directory by sending commands over control connection.

when server receives file transfer command, server opens 2nd TCP connection (for file) to client

after transferring one file, server closes data connection.

FTPclient

FTPserver

TCP control connection,

server port 21

TCP data connection,server port 20

server opens another TCP data connection to transfer another file.

control connection: “out of band”

FTP server maintains “state”: current directory, earlier authentication

Application 2-3

FTP commands, responsessample commands: sent as ASCII text over control channel USER username PASS password LIST return list of file in current directory RETR filename retrieves (gets) file STOR filename stores (puts) file onto remote host

sample return codes status code and phrase (as in HTTP) 331 Username OK, password required 125 data connection already open; transfer

starting 425 Can’t open data connection 452 Error writing file

Application 2-4

Chapter 2: Application layer2.1 Principles of network applications2.2 Web and HTTP2.3 FTP 2.4 Electronic Mail

SMTP, POP3, IMAP2.5 DNS

2.6 P2P applications2.7 Socket programming

with TCP2.8 Socket programming

with UDP

Application 2-5

Electronic MailThree major

components: user agents mail servers simple mail transfer

protocol: SMTP

User Agent a.k.a. “mail reader” composing, editing,

reading mail messages e.g., Outlook, elm, Mozilla

Thunderbird, iPhone mail client

outgoing, incoming messages stored on server

user mailbox

outgoing message queue

mailserver

useragent

useragent

useragentmail

server

useragent

useragent

mailserver

useragent

SMTP

SMTP

SMTP

Application 2-6

Electronic Mail: mail serversMail Servers mailbox contains

incoming messages for user

message queue of outgoing (to be sent) mail messages

SMTP protocol between mail servers to send email messages client: sending mail

server “server”: receiving

mail server

mailserver

useragent

useragent

useragentmail

server

useragent

useragent

mailserver

useragent

SMTP

SMTP

SMTP

Application 2-7

Electronic Mail: SMTP [RFC 2821]

uses TCP to reliably transfer email message from client to server, port 25

direct transfer: sending server to receiving server three phases of transfer

handshaking (greeting) transfer of messages closure

command/response interaction commands: ASCII text response: status code and phrase

messages must be in 7-bit ASCII

Application 2-8

Scenario: Alice sends message to Bob1) Alice uses UA to compose

message and “to” [email protected]

2) Alice’s UA sends message to her mail server; message placed in message queue

3) Client side of SMTP opens TCP connection with Bob’s mail server

4) SMTP client sends Alice’s message over the TCP connection

5) Bob’s mail server places the message in Bob’s mailbox

6) Bob invokes his user agent to read message

useragent

mailserver

mailserver user

agent

1

2 3 4 56

Application 2-9

Sample SMTP interaction S: 220 hamburger.edu C: HELO crepes.fr S: 250 Hello crepes.fr, pleased to meet you C: MAIL FROM: <[email protected]> S: 250 [email protected]... Sender ok C: RCPT TO: <[email protected]> S: 250 [email protected] ... Recipient ok C: DATA S: 354 Enter mail, end with "." on a line by itself C: Do you like ketchup? C: How about pickles? C: . S: 250 Message accepted for delivery C: QUIT S: 221 hamburger.edu closing connection

Application 2-10

Try SMTP interaction for yourself:

telnet servername 25 see 220 reply from server enter HELO, MAIL FROM, RCPT TO, DATA, QUIT

commands above lets you send email without using email

client (reader)

Application 2-11

SMTP: final words SMTP uses persistent connections SMTP requires message (header & body) to be in 7-bit

ASCII SMTP server uses CRLF.CRLF to determine end of

message

comparison with HTTP: HTTP: pull SMTP: push both have ASCII command/response interaction, status

codes HTTP: each object encapsulated in its own response

msg SMTP: multiple objects sent in multipart msg

Application 2-12

Mail message formatSMTP: protocol for exchanging email msgsRFC 822: standard for text message format: header lines, e.g.,

To: From: Subject:different from SMTP commands!

body the “message”, ASCII characters only

header

body

blankline

Application 2-13

Mail access protocols

SMTP: delivery/storage to receiver’s server mail access protocol: retrieval from server

POP: Post Office Protocol [RFC 1939]• authorization (agent <-->server) and

download IMAP: Internet Mail Access Protocol [RFC 1730]

• more features (more complex)• manipulation of stored msgs on server

HTTP: gmail, Hotmail, Yahoo! Mail, etc.

useragent

sender’s mail server

useragent

SMTP SMTP accessprotocol

receiver’s mail server

Application 2-14

POP3 protocolauthorization phase client commands:

user: declare username

pass: password server responses

+OK -ERR

transaction phase, client: list: list message

numbers retr: retrieve message

by number dele: delete quit

C: list S: 1 498 S: 2 912 S: . C: retr 1 S: <message 1 contents> S: . C: dele 1 C: retr 2 S: <message 1 contents> S: . C: dele 2 C: quit S: +OK POP3 server signing off

S: +OK POP3 server ready C: user bob S: +OK C: pass hungry S: +OK user successfully logged on

Application 2-15

POP3 (more) and IMAPmore about POP3 previous example

uses “download and delete” mode.

Bob cannot re-read e-mail if he changes client

“download-and-keep”: copies of messages on different clients

POP3 is stateless across sessions

IMAP keeps all messages

in one place: at server

allows user to organize messages in folders

keeps user state across sessions: names of folders and

mappings between message IDs and folder name

Application 2-16

Chapter 2: Application layer2.1 Principles of network applications 2.2 Web and HTTP2.3 FTP2.4 Electronic Mail

SMTP, POP3, IMAP2.5 DNS

2.6 P2P applications2.7 Socket programming

with TCP2.8 Socket programming

with UDP

Application 2-17

Pure P2P architecture no always-on server arbitrary end systems

directly communicate peers are

intermittently connected and change IP addresses

Three topics: file distribution searching for

information case Study: Skype

peer-peer

Application 2-18

File Distribution: Server-Client vs P2PQuestion : How much time to distribute file

from one server to N peers?

us

u2d1 d2u1

uN

dN

Server

Network (with abundant bandwidth)

File, size F

us: server upload bandwidthui: peer i upload bandwidthdi: peer i download bandwidth

Application 2-19

File distribution time: server-client

us

u2d1 d2u1

uN

dN

Server

Network (with abundant bandwidth)

F server sequentially

sends N copies: NF/us time

client i takes F/di time to download

increases linearly in N(for large N)

= dcs = max { NF/us, F/min(di) }i

Time to distribute F to N clients using

client/server approach

Application 2-20

File distribution time: P2P

us

u2d1 d2u1

uN

dN

Server

Network (with abundant bandwidth)

F server must send one

copy: F/us time client i takes F/di time

to download NF bits must be

downloaded (aggregate) fastest possible upload rate: us + Sui

dP2P = max { F/us, F/min(di) , NF/(us + Sui) }i

Application 2-21

0

0.5

1

1.5

2

2.5

3

3.5

0 5 10 15 20 25 30 35

N

Min

imum

Dis

tribu

tion

Tim

e P2PClient-Server

Server-client vs. P2P: example

Client upload rate = u, F/u = 1 hour, us = 10u, dmin ≥ us

Application 2-22

File distribution: BitTorrent

tracker: tracks peers participating in torrent

torrent: group of peers exchanging chunks of a file

obtain listof peers

trading chunks

peer

P2P file distribution

Application 2-23

BitTorrent (1)

file divided into 256KB chunks. peer joining torrent:

has no chunks, but will accumulate them over time

registers with tracker to get list of peers, connects to subset of peers (“neighbors”)

while downloading, peer uploads chunks to other peers.

peers may come and go once peer has entire file, it may (selfishly) leave

or (altruistically) remain

Application 2-24

BitTorrent (2)Pulling Chunks at any given time,

different peers have different subsets of file chunks

periodically, a peer (Alice) asks each neighbor for list of chunks that they have.

Alice sends requests for her missing chunks rarest first

Sending Chunks: tit-for-tat Alice sends chunks to

four neighbors currently sending her chunks at the highest rate re-evaluate top 4 every 10

secs every 30 secs: randomly

select another peer, starts sending chunks newly chosen peer may

join top 4 “optimistically unchoke”

Application 2-25

BitTorrent: Tit-for-tat(1) Alice “optimistically unchokes” Bob

(2) Alice becomes one of Bob’s top-four providers; Bob reciprocates(3) Bob becomes one of Alice’s top-four providers

With higher upload rate, can find better trading partners & get file faster!

Application 2-26

Distributed Hash Table (DHT) DHT: distributed P2P database database has (key, value) pairs;

key: ss number; value: human name key: content type; value: IP address

peers query DB with key DB returns values that match the key

peers can also insert (key, value) peers

Application 2-27

DHT Identifiers assign integer identifier to each peer in

range [0,2n-1]. Each identifier can be represented by n bits.

require each key to be an integer in same range.

to get integer keys, hash original key. e.g., key = h(“Led Zeppelin IV”) this is why they call it a distributed “hash” table

Application 2-28

How to assign keys to peers? central issue:

assigning (key, value) pairs to peers. rule: assign key to the peer that has the

closest ID. convention in lecture: closest is the

immediate successor of the key. e.g.,: n=4; peers: 1,3,4,5,8,10,12,14;

key = 13, then successor peer = 14 key = 15, then successor peer = 1

Application 2-29

1

3

4

5

810

12

15

Circular DHT (1)

each peer only aware of immediate successor and predecessor.

“overlay network”Application 2-30

Circular DHT (2)

0001

0011

0100

0101

10001010

1100

1111

Who’s resp for key 1110 ?

I am

O(N) messageson avg to resolvequery, when thereare N peers

1110

1110

1110

1110

1110

1110

Define closestas closestsuccessor

Application 2-31

Circular DHT with Shortcuts

each peer keeps track of IP addresses of predecessor, successor, short cuts.

reduced from 6 to 2 messages. possible to design shortcuts so O(log N) neighbors,

O(log N) messages in query

13

4

5

810

12

15

Who’s resp for key 1110?

Application 2-32

Peer Churn

peer 5 abruptly leaves Peer 4 detects; makes 8 its immediate successor;

asks 8 who its immediate successor is; makes 8’s immediate successor its second successor.

What if peer 13 wants to join?

1

3

4

5

810

12

15 To handle peer churn, require

each peer to know the IP address of its two successors.

Each peer periodically pings its two successors to see if they are still alive.

Application 2-33

P2P Case study: Skype inherently P2P: pairs

of users communicate.

proprietary application-layer protocol (inferred via reverse engineering)

hierarchical overlay with SNs

Index maps usernames to IP addresses; distributed over SNs

Skype clients (SC)

Supernode (SN)

Skype login server

Application 2-34

Peers as relays problem when both

Alice and Bob are behind “NATs”. NAT prevents an

outside peer from initiating a call to insider peer

solution: using Alice’s and Bob’s

SNs, relay is chosen each peer initiates

session with relay. peers can now

communicate through NATs via relay

Application 2-35

Chapter 2: Application layer

2.1 Principles of network applications2.2 Web and HTTP2.3 FTP 2.4 Electronic Mail

SMTP, POP3, IMAP2.5 DNS

2.6 P2P applications2.7 Socket programming

with TCP2.8 Socket programming

with UDP

Application 2-36

Socket programming

Socket API introduced in BSD4.1

UNIX, 1981 explicitly created, used,

released by apps client/server paradigm two types of transport

service via socket API: unreliable datagram reliable, byte stream-

oriented

a host-local, application-created,

OS-controlled interface (a “door”) into which

application process can both send and

receive messages to/from another

application process

socket

Goal: learn how to build client/server application that communicate using sockets

Application 2-37

Socket-programming using TCPSocket: a door between application process and

end-end-transport protocol (UCP or TCP)TCP service: reliable transfer of bytes from one

process to another

process

TCP withbuffers,

variables

socket

controlled byapplicationdeveloper

controlled byoperating

system

host orserver

process

TCP withbuffers,

variables

socket

controlled byapplicationdeveloper

controlled byoperatingsystem

host orserver

internet

Application 2-38

Socket programming with TCP

Client must contact server server process must first

be running server must have

created socket (door) that welcomes client’s contact

Client contacts server by: creating client-local TCP

socket specifying IP address,

port number of server process

when client creates socket: client TCP establishes connection to server TCP

when contacted by client, server TCP creates new socket for server process to communicate with client allows server to talk

with multiple clients source port numbers

used to distinguish clients (more in Chap 3)

TCP provides reliable, in-order transfer of bytes (“pipe”) between client and server

application viewpoint

Application 2-39

Client/server socket interaction: TCP

wait for incomingconnection requestconnectionSocket =welcomeSocket.accept()

create socket,port=x, forincoming request:welcomeSocket =

ServerSocket()

create socket,connect to hostid, port=xclientSocket =

Socket()

closeconnectionSocket

read reply fromclientSocket

closeclientSocket

Server (running on hostid) Client

send request usingclientSocketread request from

connectionSocket

write reply toconnectionSocket

TCP connection setup

Application 2-40

outT

oSer

ver

to network from network

inFr

omS

erve

r

inFr

omU

ser

keyboard monitor

Process

clientSocket

inputstream

inputstream

outputstream

TCPsocket

Clientprocess

client TCP socket

Stream jargon stream is a sequence of characters that flow into or out

of a process. input stream is attached to some input source for the

process, e.g., keyboard or socket. output stream is attached to an output source, e.g.,

monitor or socket.

Application 2-41

Socket programming with TCPExample client-server

app:1) client reads line from

standard input (inFromUser stream) , sends to server via socket (outToServer stream)

2) server reads line from socket

3) server converts line to uppercase, sends back to client

4) client reads, prints modified line from socket (inFromServer stream)

Application 2-42

Example: Java client (TCP)import java.io.*; import java.net.*; class TCPClient {

public static void main(String argv[]) throws Exception { String sentence; String modifiedSentence;

BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in));

Socket clientSocket = new Socket("hostname", 6789);

DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());

createinput stream

create clientSocket object

of type Socket, connect to server

createoutput stream

attached to socket

Application 2-43

This package defines Socket() and ServerSocket() classes

server port #

server name,e.g., www.umass.edu

Example: Java client (TCP), cont.

BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));

sentence = inFromUser.readLine();

outToServer.writeBytes(sentence + '\n');

modifiedSentence = inFromServer.readLine();

System.out.println("FROM SERVER: " + modifiedSentence);

clientSocket.close(); } }

createinput stream

attached to socket

send lineto serverread line

from server

Application 2-44

close socket(clean up behind yourself!)

Example: Java server (TCP)import java.io.*; import java.net.*;

class TCPServer {

public static void main(String argv[]) throws Exception { String clientSentence; String capitalizedSentence;

ServerSocket welcomeSocket = new ServerSocket(6789); while(true) { Socket connectionSocket = welcomeSocket.accept();

BufferedReader inFromClient = new BufferedReader(new InputStreamReader(connectionSocket.getInputStream()));

wait, on welcomingsocket accept() method

for client contact create, new socket on return

Application 2-45

createwelcoming socket

at port 6789

create inputstream, attached

to socket

Example: Java server (TCP), cont

DataOutputStream outToClient = new DataOutputStream(connectionSocket.getOutputStream());

clientSentence = inFromClient.readLine();

capitalizedSentence = clientSentence.toUpperCase() + '\n';

outToClient.writeBytes(capitalizedSentence); } } }

read in linefrom socket

create outputstream,

attached to socket

write out lineto socket

end of while loop,loop back and wait foranother client connection

Application 2-46

Chapter 2: Application layer

2.1 Principles of network applications2.2 Web and HTTP2.3 FTP 2.4 Electronic Mail

SMTP, POP3, IMAP2.5 DNS

2.6 P2P applications2.7 Socket programming

with TCP2.8 Socket programming

with UDP

Application 2-47

Socket programming with UDP

UDP: no “connection” between client and server

no handshaking sender explicitly attaches

IP address and port of destination to each packet

server must extract IP address, port of sender from received packet

UDP: transmitted data may be received out of order, or lost

application viewpoint:UDP provides unreliable transfer

of groups of bytes (“datagrams”) between client and server

Application 2-48

Client/server socket interaction: UDP

Server (running on hostid)

closeclientSocket

read datagram fromclientSocket

create socket,clientSocket = DatagramSocket()

Client

Create datagram with server IP andport=x; send datagram via clientSocket

create socket,port= x.serverSocket = DatagramSocket()

read datagram fromserverSocket

write reply toserverSocketspecifying client address,port number

Application 2-49

Example: Java client (UDP)

send

Pac

ket

to network from network

rece

iveP

acke

t

inFr

omU

ser

keyboard monitor

Process

clientSocket

UDPpacket

inputstream

UDPpacket

UDPsocket

Output: sends packet (recallthat TCP sent “byte stream”)

Input: receives packet (recall thatTCP received “byte stream”)

Clientprocess

client UDP socket

Application 2-50

Example: Java client (UDP)import java.io.*; import java.net.*; class UDPClient { public static void main(String args[]) throws Exception { BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in)); DatagramSocket clientSocket = new DatagramSocket(); InetAddress IPAddress = InetAddress.getByName("hostname"); byte[] sendData = new byte[1024]; byte[] receiveData = new byte[1024]; String sentence = inFromUser.readLine();

sendData = sentence.getBytes();

createinput stream

create client socket

translate hostname to IP

address using DNS

Application 2-51

Example: Java client (UDP), cont.

DatagramPacket sendPacket = new DatagramPacket(sendData, sendData.length, IPAddress, 9876); clientSocket.send(sendPacket); DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length); clientSocket.receive(receivePacket); String modifiedSentence = new String(receivePacket.getData()); System.out.println("FROM SERVER:" + modifiedSentence); clientSocket.close(); }

}

create datagram with data-to-send,

length, IP addr, port

send datagramto server

read datagramfrom server

Application 2-52

Example: Java server (UDP)import java.io.*; import java.net.*; class UDPServer { public static void main(String args[]) throws Exception { DatagramSocket serverSocket = new DatagramSocket(9876); byte[] receiveData = new byte[1024]; byte[] sendData = new byte[1024]; while(true) { DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length);

serverSocket.receive(receivePacket);

createdatagram socket

at port 9876

create space forreceived datagram

receivedatagra

mApplication 2-53

Example: Java server (UDP), cont

String sentence = new String(receivePacket.getData()); InetAddress IPAddress = receivePacket.getAddress(); int port = receivePacket.getPort(); String capitalizedSentence = sentence.toUpperCase();

sendData = capitalizedSentence.getBytes(); DatagramPacket sendPacket = new DatagramPacket(sendData, sendData.length, IPAddress, port); serverSocket.send(sendPacket); } }

}

get IP addrport #, of

sender

write out datagramto socket

end of while loop,loop back and wait foranother datagram

create datagramto send to client

Application 2-54

Chapter 2: Summary

application architectures client-server P2P hybrid

application service requirements: reliability, bandwidth,

delay Internet transport

service model connection-oriented,

reliable: TCP unreliable, datagrams:

UDP

our study of network apps now complete! specific protocols:

HTTP FTP SMTP, POP, IMAP DNS P2P: BitTorrent, Skype

socket programming

Application 2-55

Chapter 2: Summary

typical request/reply message exchange: client requests info or

service server responds with

data, status code message formats:

headers: fields giving info about data

data: info being communicated

most importantly: learned about protocols

Important themes: control vs. data msgs

in-band, out-of-band

centralized vs. decentralized

stateless vs. stateful reliable vs. unreliable

msg transfer “complexity at

network edge”Application 2-56