Twisted Training

8/2/2019 Twisted Training

http://slidepdf.com/reader/full/twisted-training 1/200

Plan

5min - Introductions

7min - Network

7min - reactor intro

15min - uppercase

10min - proxy1

10min - deferred

10min - proxy2

15m - BREAK

Saturday, 18 June 2011



Twistedan introductory training




About me- @orestis


http://orestis.gr/

http://orestis.gr/



About youRaise your hand if...




Some setup TODO


http://twisted-talk-url/

http://twisted-talk-url/



Test




Network programming

Server listens

on a port




Network programming

Server listens

on a port

Client connects

to port




Service request

Server reads/

writes data

Client reads/

writes data




While servicing...

Server reads/

writes data

Client reads/

writes data




While servicing...

Server reads/

writes data

Client reads/

writes data

Client connects

to port




Timeout!

Server reads/

writes data

Client reads/

writes data

Client connects

to port




Timeout!

Server reads/

writes data

Client reads/

writes data

Client connects

to port




Must not block!

Server listens

on a port




Must not block!

Server listens

on a port

Client connects

to port




Handle request elsewhere

Server listens

on a port

Client connects

to port





Server listens

on a port

Client reads/

writes data





Server listens

on a port

Client reads/

writes data





Fork a new process

Have worker threads



http://slidepdf.com/reader/full/twisted-training 20/200Saturday, 18 June 2011







Twisted?

Twisted is a networking engine written in Python,

supporting numerous protocols. It contains a web

server, numerous chat clients, chat servers, mail

servers, and more.




Twisted!

Text




Twisted!




Twisted

Twisted is a networking engine written in Python,

supporting numerous protocols. It contains a web

server, numerous chat clients, chat servers, mail

servers, and more.




twisted.internet Asynchronous I/O and Events.


i d i !!!



twisted.internet !!!


i d i !!!



twisted.internet !!!

defer

endpoints

error

protocol

reactor

task




twisted.internet reactor“the loop which drives applications using Twisted”







Most of the

time we arewaiting







What if we

could...




Eliminate Blocking?




Eliminate Blocking?

Reactor loop Callback functions




Reactor loop

Event happens




Twisted reactor loop




Listen on a port, then do

something


Li h d



Listen at a port, then do

something





factory listens

on a port

for each connection, a protocol

instance is created





factory listens

on a port

for each connection, a protocol

instance is created




Twisted Uppercase ServerServer that returns the data, uppercased




Twisted Uppercase Server

upperserver.pySaturday, 18 June 2011




reactor.listenTCP(8000, factory)reactor.run()








A better client multiclient.py




A better client (output)




Questions so far?Exercise coming up!




Count connected clients

Announce number of connected clients when

connecting

HINT: Protocols have a “factory” instance attribute

Exercise 1




Counting uppercase server

upperserver_ex.pySaturday, 18 June 2011



An even better client

threadedclient.pySaturday, 18 June 2011



A brief recap

reactor runs forever

a factory instance is tied to a specific port

protocol instances are created for each client

implement specific methods to add functionality




Twisted Proxy Server (v1)

Client sends an URL followed by a newline

The server returns the contents of that URL

Connection is closed




“followed by a newline”




twisted.protocols

amp basic dict finger ftp

gps htb ident loopback

memcache mice pcppolicies portforward postfix

shoutcast sip socks

stateful telnet tls wire




twisted.protocols.basic

NetstringReceiver LineOnlyReceiver

LineReceiver IntNStringReceiverInt32StringReceiver Int16StringReceiver

Int8StringReceiver StatefulStringProtocol

FileSender




twisted.protocolsDon’t reinvent the wheel!





proxy1.pySaturday, 18 June 2011



Let’s time it!

timingclient.pySaturday, 18 June 2011



Let’s time it! (output)

Client

Server


http://google.com/

http://google.com/

http://apple.com/

http://apple.com/

http://amazon.com/

http://amazon.com/

http://orestis.gr/

http://orestis.gr/

http://google.com/

http://google.com/

http://apple.com/

http://apple.com/

http://amazon.com/

http://amazon.com/

http://orestis.gr/

http://orestis.gr/



Something’s wrong!

Individual requests

0.770982027054

0.566583156586

1.45738196373

1.01898193359

3.81392908096

Threaded client

3.81683182716




Eliminate Blocking?




But in this case...

Waiting!




The culprit




The culprit

You didn’t think it’d be that easy, right?




The callbacks must becooperative




The callbacks must be

cooperative

When accessing the network, return control backto the loop

The loop will call your code when the network is ready




Network programming

Server listens

on a port




Network programming

Server listens

on a port

Client connects

to port




Network programming

Server listens

on a port

Client connects

to port




High-level panini stall






Low-level panini stall




Low-level panini stall




Waste of time!

We are idling the CPU!

Nothing else can run!

How selfish of us!




Solution: Callbacks!




Callbacks, you know...






High-level panini stall




Callbacks can be messy

Add error handling?

Pass the result around?

Cancel the original request?

Consistent API?




Introducing Deferredtwisted.internet.defer


A D f d i



A Deferred is...

A promise of a result...

A result that will appear in the future...

A result you can pass around...

Something you can attach callbacks to.


D f d P i i



Deferred Panini




Deferreds are EverywhereGet used to them!




Twisted has re-implementations of most of

the stdlib. They had to do it - not a case of NIH!


S



So....


In context



In context...


Python reminder:



Python reminder:


Python reminder:



Python reminder:


A tidier way



A tidier way





y ( )

pr ox y

2 . p y


L t’ ti it! ( t t)

http://amazon.com/



Let’s time it! (output)

Client

Server


M h b tt !

http://amazon.com/

http://amazon.com/

http://amazon.com/

http://google.com/

http://google.com/

http://apple.com/

http://apple.com/

http://orestis.gr/

http://orestis.gr/

http://apple.com/

http://apple.com/

http://google.com/

http://google.com/

http://amazon.com/

http://amazon.com/

http://orestis.gr/

http://orestis.gr/

http://amazon.com/

http://amazon.com/

http://google.com/

http://google.com/

http://apple.com/

http://apple.com/

http://orestis.gr/

http://orestis.gr/

http://orestis.gr/



Much better!

Individual requests

0.486361026764

0.850247859955

0.998661994934

1.58235692978

3.917627811433

Threaded client

1.5850892067





cooperative

When accessing the network, return control backto the loop





never call blocking functionsreturn control to the loop


Exercise 2a



Implement a caching proxy server!

Save response data in plain dict

Lookup response data

QUESTION: Where should you store the dict?

Exercise 2a


Caching Proxy Server (v1)



pr ox y 2 _

e x 1 . p y


Cool Deferred Features:

http://web.client/

http://web.client/



Chaining


Cool Deferred Features:

http://www.google.com/


http://web.client/

http://web.client/



Deferring


Chaining and Deferring

http://web.client/

http://web.client/








http://web.client/

http://web.client/










Exercise 2b



Put those cool features into use!

One callback to get a page

One callback to store it to cache

One callback to write to transport

HINT: Use defer.succeed(data) to return a “ready”Deferred

Exercise 2b


Caching Proxy Server (v2)



pr ox y 2 _

e x 2 . p y




BREAKWrite questions on whiteboard




Writing clientsthe Twisted way


Remember this?



Remember this?


Remember this?



Remember this?


Remember this?



Remember this?




Return control to the loopIn clients, too







Connect to a host, then do



something


Connect to a host, then do



something




Let’s write a twisted clientRun the uppercase server


Twisted Simple Client (v1)




simpleclient.pySaturday, 18 June 2011

Simple Client v1 (output)





Observations



Observations

Data returns with random order

We cannot access the returned data

Loop never stops

Performance?




We need to gather resultshmm, what should we use?






simpleclient2.pySaturday, 18 June 2011






Observations



Observations



Loop never stops

Performance?




Stop the loop when

everything is finishedwait until all Deferreds have fired


Introducing DeferredList



Introducing DeferredList

A list of Deferreds! You create it with a list of Deferreds

When all the Deferreds have finished, its callback fires.


Introducing DefferedList



Introducing DefferedList



http://www.orestis.gr/

http://www.orestis.gr/



http://web.client/

http://web.client/










Observations



Observations



Loop never stops

Performance?











against the proxy server


http://orestis.gr/

http://orestis.gr/

http://apple.com/

http://apple.com/

http://amazon.com/

http://amazon.com/

http://google.com/

http://google.com/

http://orestis.gr/

http://orestis.gr/

http://apple.com/

http://apple.com/

http://amazon.com/

http://amazon.com/

http://google.com/

http://google.com/

http://apple.com/

http://apple.com/

http://google.com/

http://google.com/

http://amazon.com/

http://amazon.com/

http://orestis.gr/

http://orestis.gr/



Single threaded

performance! As good as threaded performance, without thecomplexity


Exercise 3



Write a memcached SET command-line script

Exercise 3

Send:

Receive:

Usage: must wait to store and return!

Test:


Memcached SET client



Memcached SET client


Exercise 4



Make this reusable!

Exercise 4


Exercise 4



e c se


Exercise 5



Write an HTTP GET command-line script


Exercise 5




A brief recap on writing

li t



clients

Figure out the protocol

Write the protocolCreate a factory, set instance variables

Access the variables from the protocol

Connect the factory to a host & socket


That’s tedious!



Sharing state between protocols can be usefulMany times we don’t need it

There’s another way to do this.


t.i.protocol.ClientCreator



p

Change your protocol to have __init__

Create it with a protocol class and argsConnect it to a host:port

Attach a callback

When the protocol is instantiated, callback is fired


ClientCreator




Exercise 6a



Rewrite the HTTP GET protocol to work with

t.i.p.ClientCreator


Exercise 6b



Rewrite the HTTP GET protocol to work with

t.i.p.ClientCreator

...and the plain factory way

HINT: Implement Factory.buildProtocol to customise

the way...




twisted.web“the early days”


twisted.web NOP




twisted.web

http://web.server/

http://web.server/

http://web.resource/





http://web.server/

http://web.server/





Let’s build something coolIt’s going to be easy


Website status dashboard



Monitor a list of websites

Shows their status and response time

Notify people via email when a site becomes slow

Notify people via email when a site goes offline and

comes back up




How should we go about it?Discuss


curl -N

http://localhost:8000/


http://web.server/

http://web.server/





monitor1.py


Timeouts!



Abort the attempt after 30 seconds

How should we specify that? An exception?


Introducing errbacks



Like callbacks, but for error condition

Called explicitly - d.errback(reason)

Called implicitly, when a callback function raises


Errback example (v1)



errback.py


Errback example (v2)



errback_wrong.pySaturday, 18 June 2011



defer.setDebugging(True)




There can only be one calleither one callback or one errback








Calling code




Calling code






Relax, it’s easier than it

looks


Calling code









addCallbacks?





Scenario: Download Google

Doodle Alt Text



Doodle Alt Text

getPage(“ ”)

On error, print “ERROR: Google down”

Try to find doodle text

On error, print “ERROR: No doodle found”

Finally, print doodle text


Scenario: Download Google

Doodle Alt Text





Doodle Alt Text


Scenario: Download GoogleDoodle Alt Text





Doodle Alt Text


getPage






Scenario: Download GoogleDoodle Alt Text



Doodle Alt Text


getPage






getPage




getPage




getPage




GET with timeout



httpget2.py

sudo python slowserver.pySaturday, 18 June 2011

curl -N





monitor2.pySaturday, 18 June 2011

Exercise 7



Differentiate between different errors:

twisted.names.dns.DomainError

twisted.error.TimeoutError

Everything else

Have a final callback that writes results to request

HINT: Use Failure.trap(error_class)


curl -N






getHostByName




curl -N






getHostByName






Let’s drink from the firehoseremove the site limit


Doesn’t scale!



You just DOSed your operating system

Twisted will happily open the connections

You need to ensure you don’t overload the system


Many issues



We overload the DNS

We saturate the network

We exhaust the open file limit

We do this for every request!


DeferredSemaphore




DeferredSemaphore



Example: You do not want to have more than 5simultaneous DNS queries and 10 GETs


Seems to work, but...



We are creating the workload while we are operating

on it

We seem to be blocking - everything is run inside a

giant callback chain





cooperativeWhen accessing the network, return control back

to the loop


Must do as little work as possible

Doesn’t eliminate CPU-bound delays!


Seems to work, but...



In our case, the limit.release is triggering a callback

which triggers a deferred which triggers a callbackimmediately

reactor doesn’t get a chance to breathe


twisted.internet.task




cooperate







DeferredQueue




Writing clients



timeouts

error handling


Using twisted from clientcode



show sendmail

show spawnProcess

show deferToThread/callInThread


twisted codebase



interfaces/adapters/components/plugins


More deferred goodness



cancellable deferreds


Perspective broker



just simple mention


manholes?




testing



Twisted Training

Documents