Top Banner

Click here to load reader

Python concurrency: libraries overview

May 06, 2015

ReportDownload

Technology

  • 1.Built-in solutions3-rd Party SolutionsSummary Concurrency in PythonOverview of current solutions Andrii V. MishkovskyiDecember 28, 2009 Andrii V. Mishkovskyi Concurrency in Python

2. Built-in solutions Based on system threads 3-rd Party Solutions Process-based SummaryCoroutines threading module Kernel threadsSimilar to JVMs threading moduleLots of synchronization primitives Andrii V. Mishkovskyi Concurrency in Python 3. Built-in solutions Based on system threads3-rd Party Solutions Process-basedSummaryCoroutines Global Interpreter Lock Simple locking mechanism Doesnt allow execution of more than 1 thread at a time Unless its IO-bound thread CPU-bound threads dont benet from parallel execution Andrii V. Mishkovskyi Concurrency in Python 4. Built-in solutions Based on system threads 3-rd Party Solutions Process-based SummaryCoroutines multiprocessing module API similar to multithreadingOperates on process levelNice way of avoiding GIL issues Andrii V. Mishkovskyi Concurrency in Python 5. Built-in solutions Based on system threads 3-rd Party Solutions Process-based SummaryCoroutines PEP 342 Simple support through enhancedgeneratorsyield expressionNo built-in library or frameworkAndrii V. Mishkovskyi Concurrency in Python 6. TwistedBuilt-in solutions Stackless Python3-rd Party Solutions KamaeliaSummarycogen Others TwistedEmacs of Python concurrency Tons of unit tests Years of active development Supports almost every protocol there isAndrii V. Mishkovskyi Concurrency in Python 7. Twisted example from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):def dataReceived(self, data):self.transport.write(data)factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run() 8. Twisted example from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):def dataReceived(self, data):self.transport.write(data)factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run() 9. Twisted example from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):def dataReceived(self, data):self.transport.write(data)factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run() 10. Twisted example from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):def dataReceived(self, data):self.transport.write(data)factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run() 11. Twisted example from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):def dataReceived(self, data):self.transport.write(data)factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run() 12. TwistedBuilt-in solutions Stackless Python3-rd Party Solutions KamaeliaSummarycogen Others Stackless PythonPatch to CPython to not use C stack Ability to monkeypatch existing codebase with stackless sockets 100% compatible with CPython Andrii V. Mishkovskyi Concurrency in Python 13. Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):def __init__(self, conn):self.serversocket =socket.socket(socket.AF_INET,socket.SOCK_STREAM)self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)() 14. Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):def __init__(self, conn):self.serversocket =socket.socket(socket.AF_INET,socket.SOCK_STREAM)self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)() 15. Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):def __init__(self, conn):self.serversocket =socket.socket(socket.AF_INET,socket.SOCK_STREAM)self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)() 16. Stackless Example, part IIdef accept(self):while self.serversocket.accept:(clientsocket, address) = self.serversocket.accept()stackless.tasklet(self.manage)(clientsocket, address)stackless.schedule()def manage(self, clientsocket, address):clientsocket.send(cliensocket.recv(4096))s = Server((0.0.0.0, 8000))stackless.run() 17. Stackless Example, part IIdef accept(self):while self.serversocket.accept:(clientsocket, address) = self.serversocket.accept()stackless.tasklet(self.manage)(clientsocket, address)stackless.schedule()def manage(self, clientsocket, address):clientsocket.send(cliensocket.recv(4096))s = Server((0.0.0.0, 8000))stackless.run() 18. Stackless Example, part IIdef accept(self):while self.serversocket.accept:(clientsocket, address) = self.serversocket.accept()stackless.tasklet(self.manage)(clientsocket, address)stackless.schedule()def manage(self, clientsocket, address):clientsocket.send(cliensocket.recv(4096))s = Server((0.0.0.0, 8000))stackless.run() 19. TwistedBuilt-in solutions Stackless Python3-rd Party Solutions KamaeliaSummarycogen Others KamaeliaDeveloped by BBC Research Follows UNIX principles Components all the way roundAndrii V. Mishkovskyi Concurrency in Python 20. Kamaelia Example import Axonfrom Kamaelia.Chassis.ConnectedServerimport SimpleServerclass Echo(Axon.Component.component):def main(self):while 1:while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox")yield 1SimpleServer(protocol=Echo, port=1500).run() 21. Kamaelia Example import Axonfrom Kamaelia.Chassis.ConnectedServerimport SimpleServerclass Echo(Axon.Component.component):def main(self):while 1:while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox")yield 1SimpleServer(protocol=Echo, port=1500).run() 22. Kamaelia Example import Axonfrom Kamaelia.Chassis.ConnectedServerimport SimpleServerclass Echo(Axon.Component.component):def main(self):while 1:while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox")yield 1SimpleServer(protocol=Echo, port=1500).run() 23. Kamaelia Example import Axonfrom Kamaelia.Chassis.ConnectedServerimport SimpleServerclass Echo(Axon.Component.component):def main(self):while 1:while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox")yield 1SimpleServer(protocol=Echo, port=1500).run() 24. TwistedBuilt-in solutions Stackless Python3-rd Party Solutions KamaeliaSummarycogen Others cogen Coroutine-basedExtensible schedulerQuite low-level (if compared toKamaelia) Andrii V. Mishkovskyi Concurrency in Python 25. cogen example, part I from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import [email protected] server():srv = sockets.Socket()srv.bind((0.0.0.0, 8000))srv.listen(10)while True:conn, addr = yield srv.accept()m.add(handler, args=(conn, addr)) 26. cogen example, part I from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import [email protected] server():srv = sockets.Socket()srv.bind((0.0.0.0, 8000))srv.listen(10)while True:conn, addr = yield srv.accept()m.add(handler, args=(conn, addr)) 27. cogen example, part I from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import [email protected] server():srv = sockets.Socket()srv.bind((0.0.0.0, 8000))srv.listen(10)while True:conn, addr = yield srv.accept()m.add(handler, args=(conn, addr)) 28. cogen example, part I from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import [email protected] server():srv = sockets.Socket()srv.bind((0.0.0.0, 8000))srv.listen(10)while True:conn, addr = yield srv.accept()m.add(handler, args=(conn, addr)) 29. cogen example, part II @coroutinedef handler(sock, addr):data = yield sock.read()yield sock.write(data)sock.close()returnm = schedulers.Scheduler()m.add(server)m.run() 30. cogen example, part II @coroutinedef handler(sock, addr):data = yield sock.read()yield sock.write(data)sock.close()returnm = schedulers.Scheduler()m.add(server)m.run() 31. cogen example, part II @coroutinedef handler(sock, addr):data = yield sock.read()yield sock.write(data)sock.close()returnm = schedulers.Scheduler()m.add(server)m.run() 32. cogen example, part II @coroutinedef handler(sock, addr):data = yield sock.read()yield sock.write(data)sock.close()returnm = schedulers.Scheduler()m.add(server)m.run() 33. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia SummarycogenOthers Others Concurrence nice asynchrounous framework with async MySQL driver eventlet based on coroutines, developed and used by Second Life py.execnet distribute tasks over networkAndrii V. Mishkovskyi Concurrency in Python 34. Built-in solutions 3-rd Party Solutions Summary GIL doesnt harm 90% of the existing code But Twisted, Kamaelia and others are often cleaner Python is not Erlang, its better ;)Andrii V. Mishkovskyi Concurrency in Python