Built-in solutions3-rd Party Solutions
Summary
Concurrency in PythonOverview of current solutions
Andrii V. Mishkovskyi
December 28, 2009
Andrii V. Mishkovskyi Concurrency in Python
Built-in solutions3-rd Party Solutions
Summary
Based on system threadsProcess-basedCoroutines
threading module
Kernel threadsSimilar to JVM’s threading moduleLots of synchronization primitives
Andrii V. Mishkovskyi Concurrency in Python
Built-in solutions3-rd Party Solutions
Summary
Based on system threadsProcess-basedCoroutines
Global Interpreter Lock
Simple locking mechanismDoesn’t allow execution of morethan 1 thread at a timeUnless it’s IO-bound threadCPU-bound threads don’t benefitfrom parallel execution
Andrii V. Mishkovskyi Concurrency in Python
Built-in solutions3-rd Party Solutions
Summary
Based on system threadsProcess-basedCoroutines
multiprocessing module
API similar to multithreading
Operates on process levelNice way of avoiding GIL issues
Andrii V. Mishkovskyi Concurrency in Python
Built-in solutions3-rd Party Solutions
Summary
Based on system threadsProcess-basedCoroutines
PEP 342
Simple support through “enhancedgenerators”yield expressionNo built-in library or framework
Andrii V. Mishkovskyi Concurrency in Python
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
Twisted
Emacs of Python concurrencyTons of unit testsYears of active developmentSupports almost every protocolthere is
Andrii V. Mishkovskyi Concurrency in Python
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
Stackless Python
Patch to CPython to not use C stackAbility to monkeypatch existingcodebase with “stackless sockets”100% compatible with CPython
Andrii V. Mishkovskyi Concurrency in Python
Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):
def __init__(self, conn):self.serversocket = \
socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)()
Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):
def __init__(self, conn):self.serversocket = \
socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)()
Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):
def __init__(self, conn):self.serversocket = \
socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)()
Stackless Example, part II
def accept(self):while self.serversocket.accept:
(clientsocket, address) = \self.serversocket.accept()
stackless.tasklet(self.manage)(clientsocket,
address)stackless.schedule()
def manage(self, clientsocket, address):clientsocket.send(
cliensocket.recv(4096))s = Server((’0.0.0.0’, 8000))stackless.run()
Stackless Example, part II
def accept(self):while self.serversocket.accept:
(clientsocket, address) = \self.serversocket.accept()
stackless.tasklet(self.manage)(clientsocket,
address)stackless.schedule()
def manage(self, clientsocket, address):clientsocket.send(
cliensocket.recv(4096))s = Server((’0.0.0.0’, 8000))stackless.run()
Stackless Example, part II
def accept(self):while self.serversocket.accept:
(clientsocket, address) = \self.serversocket.accept()
stackless.tasklet(self.manage)(clientsocket,
address)stackless.schedule()
def manage(self, clientsocket, address):clientsocket.send(
cliensocket.recv(4096))s = Server((’0.0.0.0’, 8000))stackless.run()
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
Kamaelia
Developed by BBC ResearchFollows UNIX principlesComponents all the way round
Andrii V. Mishkovskyi Concurrency in Python
Kamaelia Example
import Axonfrom Kamaelia.Chassis.ConnectedServer\
import SimpleServerclass Echo(Axon.Component.component):
def main(self):while 1:
while self.dataReady("inbox"):self.send(self.recv("inbox"),
"outbox")yield 1
SimpleServer(protocol=Echo, port=1500).run()
Kamaelia Example
import Axonfrom Kamaelia.Chassis.ConnectedServer\
import SimpleServerclass Echo(Axon.Component.component):
def main(self):while 1:
while self.dataReady("inbox"):self.send(self.recv("inbox"),
"outbox")yield 1
SimpleServer(protocol=Echo, port=1500).run()
Kamaelia Example
import Axonfrom Kamaelia.Chassis.ConnectedServer\
import SimpleServerclass Echo(Axon.Component.component):
def main(self):while 1:
while self.dataReady("inbox"):self.send(self.recv("inbox"),
"outbox")yield 1
SimpleServer(protocol=Echo, port=1500).run()
Kamaelia Example
import Axonfrom Kamaelia.Chassis.ConnectedServer\
import SimpleServerclass Echo(Axon.Component.component):
def main(self):while 1:
while self.dataReady("inbox"):self.send(self.recv("inbox"),
"outbox")yield 1
SimpleServer(protocol=Echo, port=1500).run()
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
cogen
Coroutine-basedExtensible schedulerQuite low-level (if compared toKamaelia)
Andrii V. Mishkovskyi Concurrency in Python
cogen example, part I
from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():
srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:
conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))
cogen example, part I
from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():
srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:
conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))
cogen example, part I
from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():
srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:
conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))
cogen example, part I
from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():
srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:
conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))
cogen example, part II
@coroutinedef handler(sock, addr):
data = yield sock.read()yield sock.write(data)sock.close()return
m = schedulers.Scheduler()m.add(server)m.run()
cogen example, part II
@coroutinedef handler(sock, addr):
data = yield sock.read()yield sock.write(data)sock.close()return
m = schedulers.Scheduler()m.add(server)m.run()
cogen example, part II
@coroutinedef handler(sock, addr):
data = yield sock.read()yield sock.write(data)sock.close()return
m = schedulers.Scheduler()m.add(server)m.run()
cogen example, part II
@coroutinedef handler(sock, addr):
data = yield sock.read()yield sock.write(data)sock.close()return
m = schedulers.Scheduler()m.add(server)m.run()
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
OthersConcurrence – nice asynchrounousframework with async MySQL drivereventlet – based on coroutines,developed and used by “SecondLife”py.execnet – distribute tasks overnetwork
Andrii V. Mishkovskyi Concurrency in Python
Built-in solutions3-rd Party Solutions
Summary
GIL doesn’t harm 90% of theexisting codeBut Twisted, Kamaelia and othersare often cleanerPython is not Erlang, it’s better ;)
Andrii V. Mishkovskyi Concurrency in Python
