Elegant Solutions For Everyday Python Problems - PyCon Canada 2017
Post on 21-Jan-2018
1617 Views
Preview:
Transcript
Elegant Solutions For Everyday Python ProblemsNina Zakharenko@nnjabit.ly/elegant-python-caℹ There are links in these slides. Follow along ^
This talk is for you if:— You're an intermediate python programmer— You're coming to python from another language— You want to learn about fancy features like: magic
methods, iterators, decorators, and context managers
slides: bit.ly/elegant-python-ca
@nnja
How do we make code elegant?We pick the right tool for the job!
Resources for converting from Python 2 -> 3
Beauty isin the eye ofthe beholder
You're used to implementing __str__ and __repr__ --but there's a whole other world of powerful magic methods!
By implementing a few straightforward methods,you can make your objects behave like built-ins such as:
— numbers— lists— dictionaries— and more...
@nnja
class Money:
currency_rates = { '$': 1, '€': 0.88, }
def __init__(self, symbol, amount): self.symbol = symbol self.amount = amount
def __repr__(self): return '%s%.2f' % (self.symbol, self.amount)
@nnja
class Money:
currency_rates = { '$': 1, '€': 0.88, }
def __init__(self, symbol, amount): self.symbol = symbol self.amount = amount
def __repr__(self): return '%s%.2f' % (self.symbol, self.amount)
@nnja
class Money:
# defined currency_rates, __init__, and repr above...
def convert(self, other): """Convert other amount to our currency""" new_amount = ( other.amount / self.currency_rates[other.symbol] * self.currency_rates[self.symbol]) return Money(self.symbol, new_amount)
@nnja
__repr__ in action
>>> soda_cost = Money('$', 5.25)
>>> soda_cost $5.25
>>> pizza_cost = Money('€', 7.99)
>>> pizza_cost €7.99
@nnja
class Money:
def __add__(self, other): """ Add 2 Money instances using '+' """ new_amount = self.amount + self.convert(other).amount return Money(self.symbol, new_amount)
@nnja
>>> soda_cost = Money('$', 5.25)
>>> pizza_cost = Money('€', 7.99)
>>> soda_cost + pizza_cost $14.33
More on Magic Methods: Dive into Python3 - Special Method Names
>>> soda_cost = Money('$', 5.25)
>>> pizza_cost = Money('€', 7.99)
>>> soda_cost + pizza_cost $14.33
>>> pizza_cost + soda_cost €12.61
More on Magic Methods: Dive into Python3 - Special Method Names
@nnja
some magic methods map to built-in functions
class Alphabet: letters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
def __len__(self): return len(self.letters)
>>> my_alphabet = Alphabet()>>> len(my_alphabet) 26
@nnja
Making classes iterable— In order to be iterable, a class needs to implement
__iter__()— __iter__() must return an iterator— In order to be an iterator a class needs to implement
__next__() which must raise StopIteration when there are no more items to return
or next() in python2
^ can be confusing at first, but remember these guidelines for making classes Great explanation of iterable vs. iterator vs. generator
class IterableServer:
services = [ {'active': False, 'protocol': 'ftp', 'port': 21}, {'active': True, 'protocol': 'ssh', 'port': 22}, {'active': True, 'protocol': 'http', 'port': 21}, ] def __init__(self): self.current_pos = 0 def __iter__(self): # can return self, because __next__ implemented return self def __next__(self): while self.current_pos < len(self.services): service = self.services[self.current_pos] self.current_pos += 1 if service['active']: return service['protocol'], service['port'] raise StopIteration next = __next__ # optional python2 compatibility
@nnja
>>> for protocol, port in IterableServer(): print('service %s is running on port %d' % (protocol, port))
service ssh is running on port 22service http is running on port 21
... not bad@nnja
tip: use a generatorwhen your iterator doesn't need to maintain a lot of state
@nnja
class Server:
services = [ {'active': False, 'protocol': 'ftp', 'port': 21}, {'active': True, 'protocol': 'ssh', 'port': 22}, {'active': True, 'protocol': 'http', 'port': 21}, ]
def __iter__(self): for service in self.services: if service['active']: yield service['protocol'], service['port']
@nnja
class Server:
services = [ {'active': False, 'protocol': 'ftp', 'port': 21}, {'active': True, 'protocol': 'ssh', 'port': 22}, {'active': True, 'protocol': 'http', 'port': 21}, ]
def __iter__(self): for service in self.services: if service['active']: yield service['protocol'], service['port']
@nnja
Why does this work?use single parenthesis ( ) to create a generator comprehension^ technically, a generator expression but I like this term better, and so does Ned Batchelder
>>> my_gen = (num for num in range(1))>>> my_gen <generator object <genexpr> at 0x107581bf8>
@nnja
An iterator must implement __next__()
>>> next(my_gen) # remember __len__() mapped to built-in len() 0
and raise StopIteration when there are no more elements
>>> next(my_gen)... StopIteration Traceback (most recent call last)
For more tools for working with iterators, check out itertools
alias methods
class Word:
def __init__(self, word): self.word = word
def __repr__(self): return self.word
def __add__(self, other_word): return Word('%s %s' % (self.word, other_word))
# Add an alias from method __add__ to the method concat concat = __add__
@nnja
When we add an alias from __add__ to concat because methods are just objects
>>> # remember, concat = __add__>>> first_name = Word('Max')>>> last_name = Word('Smith')
>>> first_name + last_name Max Smith
>>> first_name.concat(last_name) Max Smith
>>> Word.__add__ == Word.concat True@nnja
Dog class
>>> class Dog: sound = 'Bark' def speak(self): print(self.sound + '!', self.sound + '!')
>>> my_dog = Dog()>>> my_dog.speak() Bark! Bark!
read the docs
getattr(object, name, default)
>>> class Dog: sound = 'Bark' def speak(self): print(self.sound + '!', self.sound + '!')
>>> my_dog = Dog()>>> my_dog.speak() Bark! Bark!
>>> getattr(my_dog, 'speak') <bound method Dog.speak of <__main__.Dog object at 0x10b145f28>>
>>> speak_method = getattr(my_dog, 'speak')>>> speak_method() Bark! Bark!
read the docs
getattr(object, name, default)
>>> class Dog: sound = 'Bark' def speak(self): print(self.sound + '!', self.sound + '!')
>>> my_dog = Dog()>>> my_dog.speak() Bark! Bark!
>>> getattr(my_dog, 'speak') <bound method Dog.speak of <__main__.Dog object at 0x10b145f28>>
>>> speak_method = getattr(my_dog, 'speak')>>> speak_method() Bark! Bark!
read the docs
Example: command line tool with dynamic commands
class Operations: def say_hi(self, name): print('Hello,', name)
def say_bye(self, name): print ('Goodbye,', name)
def default(self, arg): print ('This operation is not supported.')
if __name__ == '__main__': operations = Operations() # let's assume error handling command, argument = input('> ').split() getattr(operations, command, operations.default)(argument)
read the docs
Output
$ python getattr.py
> say_hi Nina
Hello, Nina
> blah blah
This operation is not supported.✨
additional reading - inverse of getattr() is setattr()
functool.partial(func, *args, **kwargs)
— Return a new partial object which behaves like func called with args & kwargs
— if more args are passed in, they are appended to args— if more keyword arguments are passed in, they extend
and override kwargs
read the docs on partials
functool.partial(func, *args, **kwargs)
>>> from functools import partial>>> basetwo = partial(int, base=2)>>> basetwo functools.partial(<class 'int'>, base=2)>>> basetwo('10010') 18
read the docs on partials
library I !
: github.com/jpaugh/agithub
agithub is a (badly named) REST API client with transparent syntax which facilitates rapid prototyping — on any REST API!
— Implemented in 400 lines.— Add support for any REST API in ~30 lines of code.— agithub knows everything it needs to about protocol
(REST, HTTP, TCP), but assumes nothing about your upstream API.
@nnja
define endpoint url & other connection properties
class GitHub(API):
def __init__(self, token=None, *args, **kwargs):
props = ConnectionProperties(
api_url = kwargs.pop('api_url', 'api.github.com'))
self.setClient(Client(*args, **kwargs))
self.setConnectionProperties(props)
then, start using the API!
>>> gh = GitHub('token')
>>> status, data = gh.user.repos.get(visibility='public', sort='created')
>>> # ^ Maps to GET /user/repos
>>> data
... ['tweeter', 'snipey', '...']
github.com/jpaugh/agithub
class API: def __getattr__(self, key): return IncompleteRequest(self.client).__getattr__(key) __getitem__ = __getattr__
class IncompleteRequest: def __getattr__(self, key): if key in self.client.http_methods: htmlMethod = getattr(self.client, key) return partial(htmlMethod, url=self.url) else: self.url += '/' + str(key) return self __getitem__ = __getattr__
class Client: http_methods = ('get') # ...
def get(self, url, headers={}, **params): return self.request('GET', url, None, headers)
github.com/jpaugh/agithub source: base.py
class API: def __getattr__(self, key): return IncompleteRequest(self.client).__getattr__(key) __getitem__ = __getattr__
class IncompleteRequest: def __getattr__(self, key): if key in self.client.http_methods: htmlMethod = getattr(self.client, key) return partial(htmlMethod, url=self.url) else: self.url += '/' + str(key) return self __getitem__ = __getattr__
class Client: http_methods = ('get') # ...
def get(self, url, headers={}, **params): return self.request('GET', url, None, headers)
github.com/jpaugh/agithub source: base.py
class API: def __getattr__(self, key): return IncompleteRequest(self.client).__getattr__(key) __getitem__ = __getattr__
class IncompleteRequest: def __getattr__(self, key): if key in self.client.http_methods: htmlMethod = getattr(self.client, key) return partial(htmlMethod, url=self.url) else: self.url += '/' + str(key) return self __getitem__ = __getattr__
class Client: http_methods = ('get') # ...
def get(self, url, headers={}, **params): return self.request('GET', url, None, headers)
github.com/jpaugh/agithub source: base.py
given a non-existant path:
>>> status, data = this.path.doesnt.exist.get()>>> status... 404
& because __getitem__ is aliased to __getattr__:
>>> owner, repo = 'nnja', 'tweeter'>>> status, data = gh.repos[owner][repo].pulls.get()>>> # ^ Maps to GET /repos/nnja/tweeter/pulls>>> data.... # {....}
github.com/jpaugh/agithub
Context Managers& new in python 3: async context managers
When should I use one?Need to perform an action before and/or after an operation.
Common scenarios:
— Closing a resource after you're done with it (file, network connection)
— Perform cleanup before/after a function call
@nnja
Example Problem: Feature Flags
Turn features of your application on and off easily.
Uses of feature flags:
— A/B Testing— Rolling Releases — Show Beta version to users opted-in to Beta Testing
Program
More on Feature Flags
Example - FeatureFlags Class
class FeatureFlags: """ Example class which stores Feature Flags and their state. """
SHOW_BETA = 'Show Beta version of Home Page'
flags = { SHOW_BETA: True }
@classmethod def is_on(cls, name): return cls.flags[name]
@classmethod def toggle(cls, name, on): cls.flags[name] = on
feature_flags = FeatureFlags()
@nnja
How do we temporarily turn features on and off when testing flags?
Want:
with feature_flag(FeatureFlags.SHOW_BETA): assert '/beta' == get_homepage_url()
@nnja
Using Magic Methods __enter__ and __exit__
class feature_flag: """ Implementing a Context Manager using Magic Methods """
def __init__(self, name, on=True): self.name = name self.on = on self.old_value = feature_flags.is_on(name)
def __enter__(self): feature_flags.toggle(self.name, self.on)
def __exit__(self, *args): feature_flags.toggle(self.name, self.old_value)
See: contextlib.contextmanager
The be!er way: using the contextmanager decorator
from contextlib import contextmanager
@contextmanagerdef feature_flag(name, on=True): old_value = feature_flags.is_on(name) feature_flags.toggle(name, on) yield feature_flags.toggle(name, old_value)
See: contextlib.contextmanager
The be!er way: using the contextmanager decorator
from contextlib import contextmanager
@contextmanagerdef feature_flag(name, on=True): """ The easier way to create Context Managers """ old_value = feature_flags.is_on(name) feature_flags.toggle(name, on) # behavior of __enter__() yield feature_flags.toggle(name, old_value) # behavior of __exit__()
See: contextlib.contextmanager
Note: yield?
from contextlib import contextmanager
@contextmanagerdef feature_flag(name, on=True): """ The easier way to create Context Managers """ old_value = feature_flags.is_on(name) feature_flags.toggle(name, on) # behavior of __enter__() yield feature_flags.toggle(name, old_value) # behavior of __exit__()
See: contextlib.contextmanager
either implementation
def get_homepage_url(): """ Method that returns the path of the home page we want to display. """ if feature_flags.is_on(FeatureFlags.SHOW_BETA): return '/beta' else: return '/homepage'
def test_homepage_url_with_context_manager():
with feature_flag(FeatureFlags.SHOW_BETA): # saw the beta homepage... assert get_homepage_url() == '/beta'
with feature_flag(FeatureFlags.SHOW_BETA, on=False): # saw the standard homepage... assert get_homepage_url() == '/homepage'
@nnja
either implementation
def get_homepage_url(): """ Method that returns the path of the home page we want to display. """ if feature_flags.is_on(FeatureFlags.SHOW_BETA): return '/beta' else: return '/homepage'
def test_homepage_url_with_context_manager():
with feature_flag(FeatureFlags.SHOW_BETA): assert get_homepage_url() == '/beta' print('seeing the beta homepage...')
with feature_flag(FeatureFlags.SHOW_BETA, on=False): assert get_homepage_url() == '/homepage' print('seeing the standard homepage...')
@nnja
DecoratorsThe simple explanation:
Syntactic sugar that allows modification of an underlying function.
@nnja
Recap!Decorators:— Wrap a function in another function.— Do something:
— before the call— after the call— with provided arguments— modify the return value or arguments
@nnja
def say_after(hello_function):
def say_nice_to_meet_you(name):
hello_function(name)
print('It was nice to meet you!')
return say_nice_to_meet_you
def hello(name):
print('Hello', name)
>>> hello('Nina')
Hello Nina
>>> say_after(hello)('Nina')
Hello Nina It was nice to meet you!
— say_after(hello) returns the function say_nice_to_meet_you
— then we call say_nice_to_meet_you('Nina')@nnja
def say_after(hello_function):
def say_nice_to_meet_you(name):
hello_function(name)
print('It was nice to meet you!')
return say_nice_to_meet_you
@say_after
def hello(name):
print('Hello', name)
>>> hello('Nina')
Hello Nina It was nice to meet you!
— calling the decorated function hello(name)— is the same as calling an undecorated hello with
say_after(hello)('Nina')
@nnja
closure example
def multiply_by(num): def do_multiplication(x): return x * num return do_multiplication
multiply_by_five = multiply_by(5)
>>> multiply_by_five(4) 20
@nnja
decorators that take arguments
def greeting(argument): def greeting_decorator(greet_function): def greet(name): greet_function(name) print('It was %s to meet you!' % argument) return greet return greeting_decorator
@greeting('bad')def aloha(name): print ('Aloha', name)
@nnja
decorators that take arguments
def say_this_after(argument):
def say_after(hello_function):
def say_after_meeting(name):
hello_function(name)
print('It was %s to meet you' % argument)
return say_after_meeting
return say_after
@say_this_after('bad')
def hello(name):
print('Hello', name)
Is the same as calling this on an undecorated function:
say_after_bad = say_this_after('bad')(hello)say_after_bad('Nina')
@nnja
losing context with a decorator !
def say_bye(func): def wrapper(name): func() print('Bye', name) return wrapper
@say_byedef my_name():""" Say my name""" print('Nina')
>>> my_name.__name__ 'wrapper'>>>> my_name.__doc__ # ... empty
@nnja
solution: use wraps, or wrapt library! !
from contextlib import wrapsdef say_adios(func): @wraps(func) # pass in which function to wrap def wrapper(): func() print('Adios!') return wrapper
@say_adiosdef say_max():""" Says the name Max""" print('Max')
>>> say_max.__name__ 'say_max'>>> say_max.__doc__ ' Says the name Max'
@nnja
Decorators: Common uses— logging— timing— validation— rate limiting— mocking/patching
@nnja
As of python 3.2 ContextDecorators are in the standard library. They're the best of both worlds!
— By using ContextDecorator you can easily write classes that can be used both as decorators with @ and context managers with the with statement.
— ContextDecorator is used by contextmanager(), so you get this functionality
✨
automatically
✨
.
— Alternatively, you can write a class that extends from ContextDecorator or uses ContextDecorator as a mixin, and implements __enter__, __exit__ and __call__
— If you use python2, a backport package is available here: contextlib2
@nnja
Remember @contextmanager from earlier?
from contextlib import contextmanager
@contextmanagerdef feature_flag(name, on=True): old_value = feature_flags.is_on(name) feature_flags.toggle(name, on) yield feature_flags.toggle(name, old_value)
@nnja
use it as a context manager
def get_homepage_url(): beta_flag_on = feature_flags.is_on(FeatureFlags.SHOW_BETA) return '/beta' if beta_flag_on else '/homepage'
with feature_flag(FeatureFlags.SHOW_BETA): assert get_homepage_url() == '/beta'
or use as a decorator
@feature_flag(FeatureFlags.SHOW_BETA, on=False)def get_profile_page(): beta_flag_on = feature_flags.is_on(FeatureFlags.SHOW_BETA) return 'beta.html' if beta_flag_on else 'profile.html'
assert get_profile_page() == 'profile.html'
@nnja
library I !
: freezegun lets your python tests ❇ travel through time! ❇
from freezegun import freeze_time
# use it as a Context Managerdef test(): with freeze_time("2012-01-14"): assert datetime.datetime.now() == datetime.datetime(2012, 1, 14) assert datetime.datetime.now() != datetime.datetime(2012, 1, 14)
# or a decorator@freeze_time("2012-01-14")def test(): assert datetime.datetime.now() == datetime.datetime(2012, 1, 14)
read the source sometime, it's mind-bending!
@nnja
NamedTupleUseful when you need lightweight representations of data.
Create tuple subclasses with named fields.
@nnja
Example
from collections import namedtuple
CacheInfo = namedtuple( "CacheInfo", ["hits", "misses", "max_size", "curr_size"])
@nnja
Giving NamedTuples default values
RoutingRule = namedtuple( 'RoutingRule', ['prefix', 'queue_name', 'wait_time'])
(1) By specifying defaults
RoutingRule.__new__.__defaults__ = (None, None, 20)
(2) or with _replace to customize a prototype instance
default_rule = RoutingRule(None, None, 20)user_rule = default_rule._replace(prefix='user', queue_name='user-queue')
@nnja
NamedTuples can be subclassed and extended
class Person(namedtuple('Person', ['first_name', 'last_name'])): """ Stores first and last name of a Person""" __slots__ = ()
def __str__(self): return '%s %s' % (self.first_name, self.last_name)
>>> me = Person('nina', 'zakharenko')
>>> str(me) 'nina zakharenko'
>>> me Person(first_name='nina', last_name='zakharenko')
@nnja
Tip
Use __slots__ = () in your NamedTuples!
— It prevents the creation of instance dictionaries.— It lowers memory consumption.— Allows for faster access
@nnja
"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take
away." — Antoine de Saint-Exupery
@nnja
New Tools— Magic Methods
— make your objects behave like builtins (numbers, list, dict, etc)
— Method ❇Magic❇— alias methods— * getattr— functool.partial
@nnja
— ContextManagers— Close resources
— Decorators— do something before/after call, modify return value
or validate arguments— ContextDecorators
— ContextManagers + Decorators combined!
@nnja
— Iterators & Generators— Loop over your objects— yield
— NamedTuple— Lightweight classes
@nnja
Thanks!
@nnja
nina@nnja.io
bit.ly/elegant-python-ca
@nnja
top related