Python concurrency: libraries overview

Built-in solutions3-rd Party Solutions

Summary

Concurrency in PythonOverview of current solutions

Andrii V. Mishkovskyi

December 28, 2009

Andrii V. Mishkovskyi Concurrency in Python

Built-in solutions3-rd Party Solutions

Summary

Based on system threadsProcess-basedCoroutines

threading module

Kernel threadsSimilar to JVM’s threading moduleLots of synchronization primitives

Andrii V. Mishkovskyi Concurrency in Python

Built-in solutions3-rd Party Solutions

Summary

Based on system threadsProcess-basedCoroutines

Global Interpreter Lock

Simple locking mechanismDoesn’t allow execution of morethan 1 thread at a timeUnless it’s IO-bound threadCPU-bound threads don’t benefitfrom parallel execution

Andrii V. Mishkovskyi Concurrency in Python

Built-in solutions3-rd Party Solutions

Summary

Based on system threadsProcess-basedCoroutines

multiprocessing module

API similar to multithreading

Operates on process levelNice way of avoiding GIL issues

Andrii V. Mishkovskyi Concurrency in Python

Built-in solutions3-rd Party Solutions

Summary

Based on system threadsProcess-basedCoroutines

PEP 342

Simple support through “enhancedgenerators”yield expressionNo built-in library or framework

Andrii V. Mishkovskyi Concurrency in Python

Built-in solutions3-rd Party Solutions

Summary

TwistedStackless PythonKamaeliacogenOthers

Twisted

Emacs of Python concurrencyTons of unit testsYears of active developmentSupports almost every protocolthere is

Andrii V. Mishkovskyi Concurrency in Python

Twisted example

from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):

def dataReceived(self, data):self.transport.write(data)

factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()

Twisted example

from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):

def dataReceived(self, data):self.transport.write(data)

factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()

Twisted example

from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):

def dataReceived(self, data):self.transport.write(data)

factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()

Twisted example

from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):

def dataReceived(self, data):self.transport.write(data)

factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()

Twisted example

from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):

def dataReceived(self, data):self.transport.write(data)

factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()

Built-in solutions3-rd Party Solutions

Summary

TwistedStackless PythonKamaeliacogenOthers

Stackless Python

Patch to CPython to not use C stackAbility to monkeypatch existingcodebase with “stackless sockets”100% compatible with CPython

Andrii V. Mishkovskyi Concurrency in Python

Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):

def __init__(self, conn):self.serversocket = \

socket.socket(socket.AF_INET,socket.SOCK_STREAM)

self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)

self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)()

Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):

def __init__(self, conn):self.serversocket = \

socket.socket(socket.AF_INET,socket.SOCK_STREAM)

self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)

self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)()

Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):

def __init__(self, conn):self.serversocket = \

socket.socket(socket.AF_INET,socket.SOCK_STREAM)

self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)

self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)()

Stackless Example, part II

def accept(self):while self.serversocket.accept:

(clientsocket, address) = \self.serversocket.accept()

stackless.tasklet(self.manage)(clientsocket,

address)stackless.schedule()

def manage(self, clientsocket, address):clientsocket.send(

cliensocket.recv(4096))s = Server((’0.0.0.0’, 8000))stackless.run()

Stackless Example, part II

def accept(self):while self.serversocket.accept:

(clientsocket, address) = \self.serversocket.accept()

stackless.tasklet(self.manage)(clientsocket,

address)stackless.schedule()

def manage(self, clientsocket, address):clientsocket.send(

cliensocket.recv(4096))s = Server((’0.0.0.0’, 8000))stackless.run()

Stackless Example, part II

def accept(self):while self.serversocket.accept:

(clientsocket, address) = \self.serversocket.accept()

stackless.tasklet(self.manage)(clientsocket,

address)stackless.schedule()

def manage(self, clientsocket, address):clientsocket.send(

cliensocket.recv(4096))s = Server((’0.0.0.0’, 8000))stackless.run()

Built-in solutions3-rd Party Solutions

Summary

TwistedStackless PythonKamaeliacogenOthers

Kamaelia

Developed by BBC ResearchFollows UNIX principlesComponents all the way round

Andrii V. Mishkovskyi Concurrency in Python

Kamaelia Example

import Axonfrom Kamaelia.Chassis.ConnectedServer\

import SimpleServerclass Echo(Axon.Component.component):

def main(self):while 1:

while self.dataReady("inbox"):self.send(self.recv("inbox"),

"outbox")yield 1

SimpleServer(protocol=Echo, port=1500).run()

Kamaelia Example

import Axonfrom Kamaelia.Chassis.ConnectedServer\

import SimpleServerclass Echo(Axon.Component.component):

def main(self):while 1:

while self.dataReady("inbox"):self.send(self.recv("inbox"),

"outbox")yield 1

SimpleServer(protocol=Echo, port=1500).run()

Kamaelia Example

import Axonfrom Kamaelia.Chassis.ConnectedServer\

import SimpleServerclass Echo(Axon.Component.component):

def main(self):while 1:

while self.dataReady("inbox"):self.send(self.recv("inbox"),

"outbox")yield 1

SimpleServer(protocol=Echo, port=1500).run()

Kamaelia Example

import Axonfrom Kamaelia.Chassis.ConnectedServer\

import SimpleServerclass Echo(Axon.Component.component):

def main(self):while 1:

while self.dataReady("inbox"):self.send(self.recv("inbox"),

"outbox")yield 1

SimpleServer(protocol=Echo, port=1500).run()

Built-in solutions3-rd Party Solutions

Summary

TwistedStackless PythonKamaeliacogenOthers

cogen

Coroutine-basedExtensible schedulerQuite low-level (if compared toKamaelia)

Andrii V. Mishkovskyi Concurrency in Python

cogen example, part I

from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():

srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:

conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))

cogen example, part I

from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():

srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:

conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))

cogen example, part I

from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():

srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:

conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))

cogen example, part I

from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():

srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:

conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))

cogen example, part II

@coroutinedef handler(sock, addr):

data = yield sock.read()yield sock.write(data)sock.close()return

m = schedulers.Scheduler()m.add(server)m.run()

cogen example, part II

@coroutinedef handler(sock, addr):

data = yield sock.read()yield sock.write(data)sock.close()return

m = schedulers.Scheduler()m.add(server)m.run()

cogen example, part II

@coroutinedef handler(sock, addr):

data = yield sock.read()yield sock.write(data)sock.close()return

m = schedulers.Scheduler()m.add(server)m.run()

cogen example, part II

@coroutinedef handler(sock, addr):

data = yield sock.read()yield sock.write(data)sock.close()return

m = schedulers.Scheduler()m.add(server)m.run()

Built-in solutions3-rd Party Solutions

Summary

TwistedStackless PythonKamaeliacogenOthers

OthersConcurrence – nice asynchrounousframework with async MySQL drivereventlet – based on coroutines,developed and used by “SecondLife”py.execnet – distribute tasks overnetwork

Andrii V. Mishkovskyi Concurrency in Python

Built-in solutions3-rd Party Solutions

Summary

GIL doesn’t harm 90% of theexisting codeBut Twisted, Kamaelia and othersare often cleanerPython is not Erlang, it’s better ;)

Andrii V. Mishkovskyi Concurrency in Python