Introducing to Asynchronous Programming

Introducing to asynchronous programming

Oleksandr Fedorov• Software Engineer.• More than 5 years in Python.• 1.5 years of QA Automation.

E-mail: [email protected] ID: a.g.fedorof

Agenda1. The evolution of a loop.2. We will call you back.3. The Future is here.4. Await asynchronously.

BDD with Python

Part 1 - The evolution of a loop

The evolution of a loop

from anykey import get_key

def loop(): while True: key = get_key() if key is not None: break

if __name__ == '__main__': loop()

The evolution of a loop

Tasks: 183 total, 2 running, 181 sleeping, 0 stopped, 0 zombie

PID USER PR NI SHR S %CPU %MEM TIME+ COMMAND 1949 alex 20 0 4408 R 99,7 0,1 0:29.94 python 1602 alex 20 0 28484 S 3,7 1,2 6:20.05 Viber 29713 alex 20 0 79952 S 2,0 1,9 1:41.21 skype 640 root 20 0 23980 S 1,3 0,4 2:29.21 Xorg

The evolution of a loop

import time

from anykey import get_key

def loop(): while True: key = get_key() if key is not None: break time.sleep(0.5)

if __name__ == '__main__': loop()

The evolution of a loop – `select` example.

# twisted/internet/selectreactor.py# class SelectReactor... Kinda of.

def doSelect(self, timeout): r, w, ignored = _select(self._reads, self._writes, [], timeout)

for selectables, method in ((r, "doRead"), (w, "doWrite")): for selectable in selectables: self._doReadOrWrite(selectable, method)

The evolution of a loop. I/O loops.

# Our exampleloop() # Twisted

from twisted.internet import reactorreactor.run()

# Tornadoimport tornado.iolooptornado.ioloop.IOLoop.current().start()

# pygamewhile 1: for event in pygame.event.get(): if event.type == pygame.QUIT: sys.exit() pygame.display.flip()

# TkinterApp().mainloop()

# Asyncio import asyncio asyncio.get_event_loop().run_forever()

# JavaScript /* Nothing to do, it already runs */

Part 2 – We will call you back

We will call you back

def loop(): while True: key = get_key()

if key == 'q': break

if __name__ == '__main__': loop()

We will call you back

def loop(): while True: key = get_key()

if key == 'a': go_left() elif key == 'b': ... elif key == 'c': ... elif key == 'q': break

if __name__ == '__main__': loop()

def loop(callbacks): while True: key = get_key()

if key in callbacks: callbacks[key]() elif key == 'q': break

if __name__ == '__main__':

loop({ 'a': go_left, 'b': ..., 'c': ..., })

We will call you back - jQuery

$.ajax({ url: "page.html", success: function(){ alert("done"); }});

$.ajax({ statusCode: { 404: function() { alert( "page not found" ); } }});

We will call you back - Tkinter

import tkinter as tk

class Application(tk.Frame): def __init__(self, master=None): ... self.createWidgets()

def createWidgets(self): self.hi_there = tk.Button(self) self.hi_there["text"] = "click me" self.hi_there["command"] = self.say_hi

self.QUIT = tk.Button(self, text="QUIT", fg="red", command=root.destroy)

def say_hi(self): print("hi there, everyone!")

root = tk.Tk()app = Application(master=root)app.mainloop()

Part 3 – The Future is here

The Future is here

Future (aka Deferred, aka Promise)

• Composite of callbacks.• Callbacks can be added in both "pending" and "done" states.• The result passed to a future is propagated to its callbacks.• It's possible to propagate errors (exceptions).

The Future is here – Simple example

class Future: _nope = object()

def __init__(self): self._callbacks = [] self._result = self._nope

def add_callback(self, function, *args, **kwargs): future = Future() self._callbacks.append( (function, args, kwargs, future))

if self._result != self._nope: self.done(self._result) return future

def done(self, result=None): self._result = result

while self._callbacks: function, args, kwargs, future = \ self._callbacks.pop()

func_result = function( result, *args, **kwargs) future.done(func_result)

def callback1(result): print('Callback 1 gets:', result) return 'callback1_result'

def callback2(result): print('Callback 2 gets:', result) return 'callback2_result'

def callback_n(result, n=1): print('Callback {} gets:'.format(n), result) return 'callback_n_result'

future = Future()new_future = future.add_callback(callback1)\ .add_callback(callback2)

future.done('Initial data')new_future.add_callback(callback_n, n=3)

Callback 1 gets: Initial dataCallback 2 gets: callback1_resultCallback 3 gets: callback2_result

The Future is here – Promise in JavaScript

/* A function that returns a promise */function readFile(filename, enc){ return new Promise(function (fulfill, reject){ fs.readFile(filename, enc, function (err, res){ if (err) reject(err); else fulfill(res); }); });}

/* Adding callbacks */readFile(filename, 'utf8').then( function (res){ alert('Done: ' + res); }, function (reason){ alert('Error: ' + reason); });

readFile(filename, 'utf8').then(...).then(...).then(...);

The Future is here – Deferred in Twisted

from twisted.internet import defer

def get_data(): d = defer.Deferred() d.callback(3) return d

def print_result(res): print res raise AssertionError('Catch me!')

def handle_exception(failure): failure.trap(AssertionError) sys.stderr.write(str(failure))

d = get_data()d.addCallback(print_data)d.addErrback(handle_exception)

The Future is here – Future in asyncio(not to be confused with concurrent.futures.Future)

import asyncio

def get_data(): future = asyncio.Future() future.set_result('Future is done!') return future

def print_result(future): print(future.result()) loop.stop()

loop = asyncio.get_event_loop()future = asyncio.ensure_future(get_data())future.add_done_callback(print_result)

try: loop.run_forever()finally: loop.close()

The Future is here

Problems

• Spaghetti code.• The execution flow is not clear.• Errors handling is harder.

https://xkcd.com/338/

Part 4 – Await asynchronously.

Await asynchronously – General coroutine

def coroutine(data):

print('Passed 1:', data) res1 = yield data + 1 print('Passed 2:', res1)

res2 = yield res1 + 1 print('Passed 3:', res2)

Passed 1: 1Got 1: 2Passed 2: 20Got 2: 21

cor = coroutine(1)

val1 = next(cor)print('Got 1:', val1)

val2= cor.send(val1 * 10)

print('Got 2:', val2)

Await asynchronously – General coroutine

def coroutine(data):

print('Passed 1:', data) res1 = yield data + 1 print('Passed 2:', res1)

yield res1 + 1

class Coroutine: def __init__(self, data): self._data = data self._state = 0

def __next__(self): self.send(None)

def send(self, res):

if self._state == 0:

if res is not None:

raise TypeError(

"can't send non-None value to "

"a just-started generator")

print('Passed 1:', self._data)

return self._data + 1

elif self._state == 1:

print('Passed 2:', res)

return res + 1

else:

raise StopIteration

self._state += 1

Await asynchronously – General asyncio coroutine

@coroutinedef my_coroutine(*args, **kwargs): future1 = run_some_coroutine() res1 = yield future1 print("First result:", res1)

res2 = yield run_some_coroutine(res1) print("Second result:", res2) return res2 + 1

res_future = my_coroutine()loop = asyncio.get_event_loop()loop.run_until_complete(res_future)print('Total result', res_future.result())

First result: 10Second result: 20Total result 21

Await asynchronously – General asyncio coroutine

@coroutinedef my_coroutine(*args): res1 = yield future1 res2 = yield future2 return res2

res_future = my_coroutine()print(res_future.result())

def coroutine(func): def outer(*args, **kwargs): cor = func(*args, **kwargs) future1 = next(cor)

res_future = asyncio.Future()

future1.add_done_callback( partial(rewind_future_callback, cor=cor, res_future=res_future))

return res_future return outer

12

3

3

4

Await asynchronously – General asyncio coroutine

@coroutinedef my_coroutine(*args): res1 = yield future1 res2 = yield future2 return res2

res_future = my_coroutine()print(res_future.result())

def rewind_future_callback( future1, cor, res_future):

res1 = future1.result()

try: coroutine_result = cor.send(res1) except StopIteration as ex: res_future.set_result(ex.value) else: coroutine_result.add_done_callback( partial(rewind_future_callback, cor=cor, res_future=res_future))

1

2

3

4

Await asynchronously – General asyncio coroutine

@coroutinedef my_coroutine(*args): res1 = yield from future1 res2 = yield from future2 return res2

async def my_coroutine(*args): res1 = await future1 res2 = await future2 return res2

Await asynchronously – Some rules

async def my_coroutine(*args): res = await get_data()

1. Use async def to create a coroutine.2. Use await to get real data from a future, coroutine or a task.

async def do_stuff(): try: return await moar_stuff() except SomeException: return None

3. Handle exceptions as usual.

class ClassWithCoroutines: def __init__(self, loop): self._loop = loop

4. Keep your loop around.

async with asyncfile() as file: async for line in file: print(line)

5. Use async with and async for. They are awesome.

Thank you!