Script to PyPi to Github Script to PyPi to Github @__mharrison__ @__mharrison__ http://hairysun.com http://hairysun.com
Aug 31, 2014
Script to PyPi to GithubScript to PyPi to Github
@__mharrison__@__mharrison__http://hairysun.comhttp://hairysun.com
About MeAbout Me
● 12 years Python12 years Python● Worked in HA, Search, Open Source, BI Worked in HA, Search, Open Source, BI
and Storageand Storage● Author of multiple Python BooksAuthor of multiple Python Books
ContinuumsContinuums
More like perl-TMTOWTDIMore like perl-TMTOWTDI
AgendaAgenda
● Project DevelopmentProject Development
– VersioningVersioning
– ConfigurationConfiguration
– LoggingLogging
– File inputFile input
– Shell invocationShell invocation
● Environment Layout * virtualenv * pipEnvironment Layout * virtualenv * pip
● Project layoutProject layout
Agenda (2)Agenda (2)● DocumentationDocumentation
● Automation/MakefileAutomation/Makefile
● PackagingPackaging
– setup.pysetup.py
– PyPiPyPi
● TestingTesting
● GithubGithub
● Travis CITravis CI
● PoachplatePoachplate
BeginBegin
WarningWarning
● Starting from basic Python knowledgeStarting from basic Python knowledge● Hands onHands on
– (short) lecture(short) lecture– (short) code(short) code– repeat until time is gonerepeat until time is gone
ProjectProject
Create pycat. Pythonic implementation of Create pycat. Pythonic implementation of catcat
ScriptingScripting
hello world hello world cat.pycat.py
importimport syssysforfor line line inin sys sys..stdin:stdin: printprint line, line,
hello world - Python 3hello world - Python 3
importimport syssysforfor line line inin sys sys..stdin:stdin: printprint(line, end(line, end==''''))
2 or 3?2 or 3?
2 or 3?2 or 3?
● Better legacy/library support for 2Better legacy/library support for 2● Possible to support bothPossible to support both
hello worldhello world
importimport syssysforfor line line inin sys sys..stdin:stdin: sys sys..stdoutstdout..write(line)write(line)
AssignmentAssignment
create create cat.pycat.py
Single File or Project?Single File or Project?
LayoutLayoutCan depend on distribution mechanism:Can depend on distribution mechanism:● Single fileSingle file● Zip fileZip file
– Python entry pointPython entry point– Splat/runSplat/run
● System packageSystem package● PyPi/Distribute/pip packagePyPi/Distribute/pip package
Single FileSingle File
● chmodchmod and place in and place in $PATH$PATH● add add #!/usr/bin/env python#!/usr/bin/env python
Single File (2)Single File (2)
● No reuseNo reuse
Zip FileZip File
● PYTHONPATH=cat.zip python -m PYTHONPATH=cat.zip python -m __main____main__
● tar -zxvf cat.zip; cd cat; tar -zxvf cat.zip; cd cat; python cat.pypython cat.py
Zip File (2)Zip File (2)
● No reuseNo reuse● No stacktraceNo stacktrace
System PackageSystem Package
● emerge -av qtileemerge -av qtile (rpm|apt|brew) (rpm|apt|brew)
System Package (2)System Package (2)
● Requires rootRequires root● At mercy of packager (maybe worse than At mercy of packager (maybe worse than
PyPi)PyPi)● ReuseReuse● Limited to single versionLimited to single version● python -m modulenamepython -m modulename
Pip PackagePip Package
““Best practice” is combo of:Best practice” is combo of:● DistributeDistribute● VirtualenvVirtualenv● PipPip
Pip Package (2)Pip Package (2)
$ virtualenv catenv$ virtualenv catenv$ source catenv/bin/activate$ source catenv/bin/activate$ pip install pycat$ pip install pycat
Pip Package (3)Pip Package (3)
● Multiple versionsMultiple versions● ReuseReuse● Effort to create Effort to create setup.pysetup.py
Minimal Example LayoutMinimal Example LayoutProject/Project/ README.txt README.txt project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py
Better Example LayoutBetter Example LayoutProject/Project/ .gitignore .gitignore doc/ doc/ Makefile Makefile index.rst index.rst README.txt README.txt Makefile Makefile bin/ bin/ runfoo.py runfoo.py project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py
AssignmentAssignment
create layout for create layout for cat.pycat.py
Semantic VersioningSemantic Versioning
VersioningVersioning
http://semver.orghttp://semver.org
Formal spec for versioning projectsFormal spec for versioning projects
Python VersioningPython Versioning
PEP 386PEP 386
N.N[.n]+[{a|b|c}N[.N]+][.postN][.devN]N.N[.n]+[{a|b|c}N[.N]+][.postN][.devN]
1.0a1 < 1.0a2 < 1.0b2.post3451.0a1 < 1.0a2 < 1.0b2.post345
setup.pysetup.py
fromfrom distutils.coredistutils.core importimport setup setup
setup(namesetup(name=='PyCat''PyCat',,........ version version=='1.0''1.0'))
Where to store version?Where to store version?
In In module.__version__module.__version__ (might cause (might cause importing issues)importing issues)
VersionVersion
If using Sphinx for docs, be sure to update:If using Sphinx for docs, be sure to update:
docs/docs/ conf.py conf.py
argparseargparse
ap ap == argparse argparse..ArgumentParser(versionArgumentParser(version=='1.0''1.0'))......apap..print_version()print_version()
AssignmentAssignment
Add version to projectAdd version to project
ConfigurationConfiguration
ConfigurationConfiguration● PythonPython
– Django (Django (settings.pysettings.py))
– Python (Python (site.pysite.py, , setup.pysetup.py))
● JSON/YAMLJSON/YAML
– Google App EngineGoogle App Engine
● Environment VariablesEnvironment Variables
– Python (Python (PYTHONPATHPYTHONPATH))
● .ini (.ini (ConfigParserConfigParser, , ConfigObjConfigObj))
– matplotlibmatplotlib
– ipythonipython
● Command line options (Command line options (argparseargparse))
● Sqlite blobSqlite blob
● Shelve/pickle blobShelve/pickle blob
Configuration (2)Configuration (2)
Are not mutually exclusiveAre not mutually exclusive
Configuration (3)Configuration (3)(Unix) Hierarchy:(Unix) Hierarchy:
● System rc (run control) (System rc (run control) (/etc/conf.d/etc/conf.d))
● User rc (User rc (~/.config/app/...~/.config/app/...))
● Environment variablesEnvironment variables
● Command line optionsCommand line options
http://www.faqs.org/docs/artu/ch10s02.htmlhttp://www.faqs.org/docs/artu/ch10s02.html
Filesystem Hierarchy Standard: Filesystem Hierarchy Standard: http://www.pathname.com/fhs/http://www.pathname.com/fhs/
Configuration (4)Configuration (4)
● Plain text config is easily approachablePlain text config is easily approachable● Careful with Python config on process Careful with Python config on process
run by rootrun by root
AssignmentAssignment
Add configuration Add configuration -n-n to to show line numbersshow line numbers
LoggingLogging
LoggingLogging
logginglogging module provides feature-rich module provides feature-rich logginglogging
Logging (2)Logging (2)
importimport logginglogging
logginglogging..basicConfig(levelbasicConfig(level==logginglogging..ERROR,ERROR, filename filename=='.log''.log'))......logginglogging..error(error('Error encountered in...''Error encountered in...'))
AssignmentAssignment
Add configuration Add configuration --verbose--verbose to log file to log file
being “catted”being “catted”
Dealing with File InputDealing with File Input
““Files”Files”
Dealing with?Dealing with?● FilenameFilename● file objectfile object● string datastring data
FilenameFilename
Somewhat analogous to an Somewhat analogous to an IterableIterable..● Can open/iterate many timesCan open/iterate many times● Implementation depends on fileImplementation depends on file● Need to manage closing fileNeed to manage closing file
File ObjectFile Object
Somewhat analogous to an Somewhat analogous to an IteratorIterator..● Can iterate once (unless Can iterate once (unless seekseeked)ed)
● Can accept Can accept filefile, , StringIOStringIO, , socketsocket, , generator, etcgenerator, etc
● Memory friendly - scalableMemory friendly - scalable
String DataString Data
No iterator analogy.No iterator analogy.● Memory hog - less scalableMemory hog - less scalable
Stdlib ExamplesStdlib ExamplesModule String Data File Filename
json loads load
pickle loads load
xml.etree.ElementTree
fromstring parse parse
xml.dom.minidom parseString parse parse
ConfigParser cp.readfp cp.read(filenames)
csv reader DictReader
pyyaml 3rd party load, safe_load load, safe_load
Stdlib Take-awaysStdlib Take-aways
● mostly functionsmostly functions● file interface is required, others optionalfile interface is required, others optional● parseparse or or loadload
ExampleExample>>>>>> importimport syssys
>>>>>> defdef parseparse(fin):(fin):...... forfor line line inin upper(fin): upper(fin):...... sys sys..stdoutstdout..write(line)write(line)
>>>>>> defdef upperupper(iterable):(iterable):...... forfor item item inin iterable: iterable:...... yieldyield strstr(item)(item)..upper()upper()
Create file to parseCreate file to parse
>>> >>> withwith openopen(('/tmp/data''/tmp/data', , 'w''w') ) asas fout: fout:... ... fout fout..write(write('line1'line1\n\n''))... ... fout fout..write(write('line2'line2\n\n''))
Filename to Filename to filefile
>>> >>> filename filename == '/tmp/data''/tmp/data'>>> >>> withwith openopen(filename) (filename) asas fin: fin:... ... parse(fin) parse(fin)LINE1LINE1LINE2LINE2
String data to String data to filefile>>> >>> data data == "string"string\n\ndatadata\n\n"">>> >>> importimport StringIOStringIO>>> >>> parse(StringIOparse(StringIO..StringIO(data))StringIO(data))STRINGSTRINGDATADATA
Parse IterableParse Iterable
>>> >>> data data == [ ['foo'foo\n\n'', , 'bar'bar\n\n'']]>>> >>> parse(data)parse(data)FOOFOOBARBAR
More More filefile benefits benefits
● Combine with generators to filter, tweakCombine with generators to filter, tweak● Easier to testEasier to test
AssignmentAssignment
Add a Add a parseparse function function
Invoking Shell Invoking Shell CommandsCommands
Reading outputReading output
>>> >>> importimport subprocesssubprocess>>> >>> p p == subprocess subprocess..Popen(Popen('id -u''id -u', shell, shell==TrueTrue, , stdoutstdout==subprocesssubprocess..PIPE, stderrPIPE, stderr==subprocesssubprocess..PIPE)PIPE)>>> >>> pp..stdoutstdout..read()read()'1000\n''1000\n'>>> >>> pp..returncode returncode # None means not done# None means not done>>> >>> printprint p p..wait()wait()00
Feeding Feeding stdinstdinCan use Can use communicatecommunicate or or p2.stdin.writep2.stdin.write w/ w/ flush/closeflush/close..>>> >>> p2 p2 == subprocess subprocess..Popen(Popen('wc -l''wc -l', shell, shell==TrueTrue, , stdoutstdout==subprocesssubprocess..PIPE, stdinPIPE, stdin==subprocesssubprocess..PIPE, PIPE, stderrstderr==subprocesssubprocess..PIPE)PIPE)>>> >>> out, err out, err == p2 p2..communicate(communicate('foo'foo\n\nbarbar\n\n'') ) #p.stdin.flush()#p.stdin.flush()
>>> >>> outout'2\n''2\n'>>> >>> p2p2..returncodereturncode00
Chaining scriptsChaining scriptsChaining is pretty straightforward make sure to Chaining is pretty straightforward make sure to closeclose stdinstdin. . http://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subprhttp://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subprocess-pipelineocess-pipeline
>>> >>> p3 p3 == subprocess subprocess..Popen(Popen('sort''sort', shell, shell==TrueTrue,,... ... stdout stdout==subprocesssubprocess..PIPE,PIPE,... ... stdin stdin==subprocesssubprocess..PIPE)PIPE)>>> >>> p4 p4 == subprocess subprocess..Popen(Popen('uniq''uniq', shell, shell==TrueTrue,,... ... stdout stdout==subprocesssubprocess..PIPE,PIPE,... ... stdin stdin==p3p3..stdout,stdout,... ... close_fds close_fds==TrueTrue) ) # hangs w/o close_fds# hangs w/o close_fds
>>> >>> p3p3..stdinstdin..write(write('1'1\n\n22\n\n11\n\n''))>>> >>> p3p3..stdinstdin..flush(); p3flush(); p3..stdinstdin..close();close();>>> >>> p4p4..stdoutstdout..read()read()'1\n2\n''1\n2\n'
Chaining scripts and pythonChaining scripts and python
catcat 0-2, add 10 to them (in python) and 0-2, add 10 to them (in python) and wc wc -l-l results. results.>>> >>> importimport osos>>> >>> p5 p5 == subprocess subprocess..Popen(Popen('cat''cat', shell, shell==TrueTrue, , stdoutstdout==subprocesssubprocess..PIPE, stdinPIPE, stdin==subprocesssubprocess..PIPE, close_fdsPIPE, close_fds==TrueTrue))>>> >>> defdef p6p6((inputinput):):... ... ''' add 10 to line in input '''''' add 10 to line in input '''... ... forfor line line inin inputinput::... ... yieldyield ''%d%s%d%s'' %%((intint(line(line..strip())strip())+10+10, os, os..linesep)linesep)
Chaining scripts and python Chaining scripts and python (2)(2)
>>> >>> p7 p7 == subprocess subprocess..Popen(Popen('wc -l''wc -l', shell, shell==TrueTrue, , stdoutstdout==subprocesssubprocess..PIPE, stdinPIPE, stdin==subprocesssubprocess..PIPE, close_fdsPIPE, close_fds==TrueTrue))>>> >>> [p5[p5..stdinstdin..write(write(strstr(x)(x)++osos..linesep) linesep) forfor x x inin xrangexrange((33)])]>>> >>> p5p5..stdinstdin..close()close()>>> >>> [p7[p7..stdinstdin..write(x) write(x) forfor x x inin p6(p5 p6(p5..stdoutstdout..xreadlines())]xreadlines())]>>> >>> p7p7..stdinstdin..close()close()>>> >>> p7p7..stdoutstdout..read()read()'3\n''3\n'
EnvironmentEnvironment
Python EnvironmentPython Environment
● System Python or “Virtual” PythonSystem Python or “Virtual” Python● Installation methodInstallation method
Which PythonWhich Python
System PythonSystem Python● Requires rootRequires root● Only one version of libraryOnly one version of library● System packages may be out of dateSystem packages may be out of date
Which Python (2)Which Python (2)
““Virtual” PythonVirtual” Python● Runs as userRuns as user● Specify versionSpecify version● Sandboxed from systemSandboxed from system● Create multiple sandboxesCreate multiple sandboxes
Which Python (3)Which Python (3)
Install Install virtualenvvirtualenv
virtualenvvirtualenvInstallation:Installation:
● System package System package $ sudo apt-get install $ sudo apt-get install python-virtualenvpython-virtualenv
● With pip With pip $ pip install virtualenv$ pip install virtualenv● $ easy_install virtualenv$ easy_install virtualenv● $ wget $ wget https://raw.github.com/pypa/virtualenv/master/https://raw.github.com/pypa/virtualenv/master/virtualenv.py; python virtualenv.pyvirtualenv.py; python virtualenv.py
virtualenvvirtualenv (2) (2)
Create virtual environment:Create virtual environment:
$ virtualenv env_name$ virtualenv env_name
Directory StructureDirectory Structure
env_name/env_name/ bin/ bin/ activate activate python python pip pip lib/ lib/ python2.7/ python2.7/ site-packages/ site-packages/
virtualenvvirtualenv (3) (3)
Activate virtual environment:Activate virtual environment:
$ source env_name/bin/activate$ source env_name/bin/activate
virtualenvvirtualenv (4) (4)
Windows activate virtual environment:Windows activate virtual environment:
> \path\to\env\Scripts\activate> \path\to\env\Scripts\activate
May require May require PS C:\> PS C:\> Set-ExecutionPolicy AllSignedSet-ExecutionPolicy AllSigned
virtualenvvirtualenv (5) (5)
Comes with Comes with pippip to install packages: to install packages:
$ pip install sqlalchemy$ pip install sqlalchemy
virtualenvvirtualenv (6) (6)
$ which python # not /usr/bin/python$ which python # not /usr/bin/python/home/matt/work/courses/script-pypi-github/env/b/home/matt/work/courses/script-pypi-github/env/bin/pythonin/python
virtualenvvirtualenv (7) (7)
>>> sys.path>>> sys.path['', ..., ['', ..., '/home/matt/work/courses/script-py'/home/matt/work/courses/script-pypi-github/env/lib/python2.7/site-ppi-github/env/lib/python2.7/site-packages']ackages']
virtualenvvirtualenv (8) (8)
Use Use deactivatedeactivate shell function to reset shell function to reset PATHPATH::
$ deactivate$ deactivate
AssignmentAssignment
create virtualenv create virtualenv envenv
pippip
pippipRecursive Acronym - Recursive Acronym - PPip ip IInstalls nstalls PPackagesackages● installinstall● upgradeupgrade● uninstalluninstall● ““pin” versionspin” versions● requirements.txtrequirements.txt
pippip (2) (2)
Install:Install:
$ pip install sqlalchemy$ pip install sqlalchemy
pippip (3) (3)
Upgrade:Upgrade:
$ pip install --upgrade sqlalchemy$ pip install --upgrade sqlalchemy
pippip (4) (4)
Uninstall:Uninstall:
$ pip uninstall sqlalchemy$ pip uninstall sqlalchemy
pippip (5) (5)
““Pin” version:Pin” version:
$ pip install sqlalchemy==0.7$ pip install sqlalchemy==0.7
pippip (6) (6)
Requirements file:Requirements file:
$ pip install -r requirements.txt$ pip install -r requirements.txt
pippip (7) (7)
Requirements file Requirements file $ cat requirements.txt$ cat requirements.txt::
sqlalchemy==0.7sqlalchemy==0.7foobarfoobarbazlib>=1.0bazlib>=1.0
pippip (8) (8)
Create requirement file from env:Create requirement file from env:
$ pip freeze > req.txt$ pip freeze > req.txt
pippip (9) (9)
● pip docs say to install from virtualenvpip docs say to install from virtualenv● virtualenv docs say to install from pipvirtualenv docs say to install from pip
pippip (10) (10)Install from directory:Install from directory:
pip install --download packages -r pip install --download packages -r requirements.txtrequirements.txtpip install --no-index pip install --no-index --find-links=file://full/path/to/p--find-links=file://full/path/to/packages -r requirements.txtackages -r requirements.txt
distributedistribute, , setuptoolssetuptools, , distutilsdistutils
● pippip wraps wraps distributedistribute adds uninstall, adds uninstall, req.txtreq.txt
● distributedistribute fork of fork of setuptoolssetuptools● setuptoolssetuptools unmaintained, adds eggs, unmaintained, adds eggs,
dependencies, dependencies, easy_installeasy_install● distutilsdistutils stdlib packaging library stdlib packaging library
AssignmentAssignment
use pip to install use pip to install package from internetpackage from internet
More LayoutMore Layout
Better Example LayoutBetter Example LayoutProject/Project/ .gitignore .gitignore doc/ doc/ Makefile Makefile index.rst index.rst README.txt README.txt Makefile Makefile bin/ bin/ runfoo.py runfoo.py project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py
.gitignore.gitignore*.py[cod]*.py[cod]
# C extensions# C extensions*.so*.so
# Packages# Packages*.egg*.egg*.egg-info*.egg-infodistdistbuildbuildeggseggspartspartsbinbinvarvarsdistsdist
.gitignore.gitignoredevelop-eggsdevelop-eggs.installed.cfg.installed.cfglibliblib64lib64__pycache____pycache__
# Installer logs# Installer logspip-log.txtpip-log.txt
# Unit test / coverage reports# Unit test / coverage reports.coverage.coverage.tox.toxnosetests.xmlnosetests.xml
# Translations# Translations*.mo*.mo
https://github.com/github/gitignorehttps://github.com/github/gitignore
.gitignore.gitignore (2) (2)
See See blaze-coreblaze-core for example of C and for example of C and Docs. Docs. https://github.com/ContinuumIO/blaze-corhttps://github.com/ContinuumIO/blaze-coree
.gitignore.gitignore (3) (3)From From requestsrequests::
.coverage.coverageMANIFESTMANIFESTcoverage.xmlcoverage.xmlnosetests.xmlnosetests.xmljunit-report.xmljunit-report.xmlpylint.txtpylint.txttoy.pytoy.pyviolations.pyflakes.txtviolations.pyflakes.txtcover/cover/docs/_builddocs/_buildrequests.egg-info/requests.egg-info/*.pyc*.pyc*.swp*.swpenv/env/.workon.workont.pyt.pyt2.pyt2.py
https://github.com/kennethreitz/requestshttps://github.com/kennethreitz/requests
Other tipsOther tips● localsettings.pylocalsettings.py (for django) (for django)
● *~*~ (emacs) (emacs)
● Run Run git statusgit status and add outliers to and add outliers to .gitignore.gitignore
● Make settings global:Make settings global:git config --global core.excludesfile git config --global core.excludesfile Python.gitignorePython.gitignoregit config --global core.excludesfile git config --global core.excludesfile Python.gitignorePython.gitignore
AssignmentAssignment
add (premptive) add (premptive) .gitignore.gitignore
DocumentationDocumentation
DocumentationDocumentation
Two types:Two types:● Developer docs (README, INSTALL, Developer docs (README, INSTALL,
HACKING, etc)HACKING, etc)● End userEnd user
DeveloperDeveloper
READMEREADME - main entry point for project - main entry point for project● Brief introBrief intro● Links/ContactLinks/Contact● LicenseLicense
READMEREADME
For github integration name it For github integration name it README.rstREADME.rst or or README.mdREADME.md
LICENSELICENSE
Include text of license. Templates at Include text of license. Templates at http://opensource.org/licenses/index.htmlhttp://opensource.org/licenses/index.html
LicensingLicensing
Some include dunder meta in project docstring (Some include dunder meta in project docstring (requestsrequests __init__.py__init__.py):):
:copyright: (c) 2013 by Kenneth Reitz.:copyright: (c) 2013 by Kenneth Reitz.:license: Apache 2.0, see LICENSE for more :license: Apache 2.0, see LICENSE for more details.details.
(note IANAL)(note IANAL)
Licensing (2)Licensing (2)Some include dunder meta in project (Some include dunder meta in project (requestsrequests __init__.py__init__.py):):
__title__ = 'requests'__title__ = 'requests'__version__ = '1.1.0'__version__ = '1.1.0'__build__ = 0x010100__build__ = 0x010100__author__ = 'Kenneth Reitz'__author__ = 'Kenneth Reitz'__license__ = 'Apache 2.0'__license__ = 'Apache 2.0'__copyright__ = 'Copyright 2013 Kenneth Reitz'__copyright__ = 'Copyright 2013 Kenneth Reitz'
(note IANAL)(note IANAL)
Other filesOther files
● AUTHORSAUTHORS● HISTORY/CHANGELOGHISTORY/CHANGELOG● TODOTODO
AssignmentAssignment
create simple create simple READMEREADME
End User DocsEnd User Docs
Sphinx is a tool that makes it easy to create Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, intelligent and beautiful documentation, written by Georg Brandl and licensed written by Georg Brandl and licensed under the BSD license.under the BSD license.
http://sphinx-doc.orghttp://sphinx-doc.org
Suggested setupSuggested setup
Project/Project/ doc/ doc/ # sphinx stuff # sphinx stuff Makefile Makefile
Sphinx in 4 LinesSphinx in 4 Lines
$ cd docs$ cd docs$ sphinx-quickstart$ sphinx-quickstart$ $EDITOR index.rst$ $EDITOR index.rst$ make html$ make html
AssignmentAssignment
write docs at a later write docs at a later time :)time :)
MakefileMakefile
MotivationMotivationRunning commands often:Running commands often:
● nosetestsnosetests (plus options) (plus options)
● create sdistcreate sdist
● upload to PyPiupload to PyPi
● create virtualenvcreate virtualenv
● install dependenciesinstall dependencies
● cleanup cruftcleanup cruft
● create TAGScreate TAGS
● profileprofile
● sdistsdist
● PyPi - register and uploadPyPi - register and upload
● creating pngs from svgscreating pngs from svgs
● docsdocs
● Python 3 testingPython 3 testing
● etc...etc...
MakefileMakefile
● Knows about executing (build) Knows about executing (build) commandscommands
● Knows about dependenciesKnows about dependencies
ExampleExample
To test:To test:● Make virtualenvMake virtualenv● install code dependenciesinstall code dependencies● install nose (+coverage)install nose (+coverage)● run testsrun tests
Clean checkoutClean checkout$ make test$ make testvsvs
$ virtualenv env$ virtualenv env$ env/bin/activate$ env/bin/activate$ pip install -r deps.txt$ pip install -r deps.txt$ pip install nose coverage.py$ pip install nose coverage.py$ nosestests$ nosestests
Enough Enough makemake knowledge to be knowledge to be
dangerousdangerous
MakefileMakefile (1) (1)
Syntax of Syntax of MakefileMakefile::
file: dependentfilefile: dependentfile<TAB>Command1<TAB>Command1......<TAB>CommandN<TAB>CommandN
MakefileMakefile (2) (2)
Running (runs Running (runs MakefileMakefile by default): by default):
$ make file$ make file# will build dependentfile if # will build dependentfile if necessarynecessary# then build file# then build file
MakefileMakefile (3) (3)
Example:Example:
foo: foo.c foo.hfoo: foo.c foo.h<TAB>cc -c foo.c<TAB>cc -c foo.c<TAB>cc -o foo foo.o<TAB>cc -o foo foo.o
MakefileMakefile (4) (4)
Running (echoes commands by default Running (echoes commands by default -s-s for silent):for silent):
$ make$ makecc -c foo.ccc -c foo.ccc -o foo foo.occ -o foo foo.o
MakefileMakefile (5) (5)
Subsequent runs do nothing:Subsequent runs do nothing:
$ make$ makemake: `foo' is up to date.make: `foo' is up to date.
MakefileMakefile (6) (6)
Add a Add a cleanclean command: command:
.PHONY: clean.PHONY: clean
clean:clean:<TAB>rm foo.o foo<TAB>rm foo.o foo
MakefileMakefile (7) (7)
Since Since cleanclean isn't a file, need to use isn't a file, need to use .PHONY.PHONY to indicate that to to indicate that to makemake. (If you had a file . (If you had a file named named cleanclean it wouldn't try to build it). it wouldn't try to build it).
MakefileMakefile (8) (8)
(Simply Expanded) Variables (expanded when set):(Simply Expanded) Variables (expanded when set):
BIN := env/binBIN := env/binPY := $(BIN)/pythonPY := $(BIN)/pythonNOSE := $(BIN)/nosetestsNOSE := $(BIN)/nosetests
.PHONY: build.PHONY: buildbuild: envbuild: env<TAB>$(PY) setup.py sdist<TAB>$(PY) setup.py sdist
MakefileMakefile (9) (9)
(Recursively Expanded) Variables (expanded when used):(Recursively Expanded) Variables (expanded when used):
FILE = fooFILE = fooDATA = $(FILE)DATA = $(FILE)
# If DATA expanded would be foo# If DATA expanded would be foo
FILE = barFILE = bar# If DATA expanded would be bar# If DATA expanded would be bar
MakefileMakefile (10) (10)
Shell functions:Shell functions:
.PHONY: pwd.PHONY: pwdpwd:pwd:<TAB>pushd /etc<TAB>pushd /etc
MakefileMakefile (11) (11)Invoking:Invoking:
$ make pwd$ make pwdpushd /etcpushd /etcmake: pushd: Command not foundmake: pushd: Command not foundmake: *** [pwd] Error 127make: *** [pwd] Error 127
((pushdpushd is a bash function) is a bash function)
MakefileMakefile (12) (12)Shell functions:Shell functions:
SHELL := /bin/bashSHELL := /bin/bash
.PHONY: pwd.PHONY: pwdpwd:pwd:<TAB>pushd /etc<TAB>pushd /etc
MakefileMakefile (13) (13)
Multiple commands:Multiple commands:
SHELL := /bin/bashSHELL := /bin/bash
.PHONY: pwd.PHONY: pwdpwd:pwd:<TAB>pushd /etc<TAB>pushd /etc<TAB>pwd<TAB>pwd<TAB>popd<TAB>popd
MakefileMakefile (14) (14)
Multiple commands:Multiple commands:
$ make pwd$ make pwdpushd /etcpushd /etc/etc /tmp/foo/etc /tmp/foopwdpwd/tmp/foo/tmp/foopopdpopd/bin/bash: line 0: popd: directory stack empty/bin/bash: line 0: popd: directory stack empty
MakefileMakefile (15) (15)
Each tab indented command runs in its Each tab indented command runs in its own process. Use own process. Use ;; and put in one line or and put in one line or use use \\ for line continuation for line continuation
MakefileMakefile (16) (16)
Multiple commands (use line continuation Multiple commands (use line continuation \\):):
SHELL := /bin/bashSHELL := /bin/bash
.PHONY: pwd2.PHONY: pwd2pwd2:pwd2:<TAB>pushd /etc; \<TAB>pushd /etc; \<TAB>pwd; \<TAB>pwd; \<TAB>popd<TAB>popd
MakefileMakefile (17) (17)
Shell variables:Shell variables:
.PHONY: path.PHONY: pathpath:path:<TAB>echo $PATH<TAB>echo $PATH
MakefileMakefile (18) (18)
Make thinks they are make variables:Make thinks they are make variables:
$ make path$ make pathecho ATHecho ATHATHATH
MakefileMakefile (19) (19)
$$ needs to be escaped with needs to be escaped with $$::
.PHONY: path.PHONY: pathpath:path:<TAB>echo $$PATH<TAB>echo $$PATH
MakefileMakefile (18) (18)
Now it works:Now it works:
$ make path$ make pathecho $PATHecho $PATH/tmp/maketest/tmp/maketest
Makefiles for Python Makefiles for Python ProjectsProjects
Inspired by Rick Harding's Inspired by Rick Harding's TalkTalk
http://pyvideo.org/video/1354/starting-youhttp://pyvideo.org/video/1354/starting-your-project-right-setup-and-automationr-project-right-setup-and-automationhttps://github.com/mitechie/pyohio_2012https://github.com/mitechie/pyohio_2012
MakefileMakefile for Python for Python
Make virtualenv:Make virtualenv:
env:env:<TAB>virtualenv env<TAB>virtualenv env
MakefileMakefile for Python (2) for Python (2)
Make dependencies:Make dependencies:
.PHONY: deps.PHONY: depsdeps: envdeps: env<TAB>$(PIP) install -r <TAB>$(PIP) install -r requirements.txtrequirements.txt
MakefileMakefile for Python (3) for Python (3)
Testing with Testing with nosenose::
.PHONY: test.PHONY: testtest: nose depstest: nose deps<TAB>$(NOSE)<TAB>$(NOSE)
# nose depends on the nosetests binary# nose depends on the nosetests binarynose: $(NOSE)nose: $(NOSE)$(NOSE): env$(NOSE): env<TAB>$(PIP) install nose<TAB>$(PIP) install nose
Contrary OpinionsContrary Opinions
““Dynamic languages don't need anything Dynamic languages don't need anything like like makemake, unless they have some , unless they have some compile-time interface dependencies compile-time interface dependencies between modules”between modules”
http://stackoverflow.com/questions/758093http://stackoverflow.com/questions/7580939/why-are-there-no-makefiles-for-automati9/why-are-there-no-makefiles-for-automation-in-python-projectson-in-python-projects
Other optionsOther options
● paverpaver● fabricfabric● buildoutbuildout
PackagingPackaging
setup.pysetup.py overloaded overloaded
● create sdist (source distribution)create sdist (source distribution)● upload to pypiupload to pypi● install packageinstall package
setup.pysetup.py wart wart
requirerequire keyword of keyword of distutilsdistutils doesn't doesn't download reqs only download reqs only documentsdocuments them. Use them. Use requirements.txtrequirements.txt in combo with in combo with pippip..
setup.pysetup.py example exampleFrom From requestsrequests::
setup(setup( name='requests', name='requests', version=requests.__version__, version=requests.__version__, description='Python HTTP for Humans.', description='Python HTTP for Humans.', long_description=open('README.rst').read() + long_description=open('README.rst').read() + '\n\n' +'\n\n' + open('HISTORY.rst').read(), open('HISTORY.rst').read(), author='Kenneth Reitz', author='Kenneth Reitz', author_email='[email protected]', author_email='[email protected]', url='http://python-requests.org', url='http://python-requests.org',
setup.pysetup.py example (2) example (2)
packages=packages,packages=packages,package_data={'': ['LICENSE', 'NOTICE'], package_data={'': ['LICENSE', 'NOTICE'], 'requests': ['*.pem']},'requests': ['*.pem']},package_dir={'requests': 'requests'},package_dir={'requests': 'requests'},include_package_data=True,include_package_data=True,install_requires=requires,install_requires=requires,license=open('LICENSE').read(),license=open('LICENSE').read(),zip_safe=False,zip_safe=False,
setup.pysetup.py example (3) example (3) classifiers=(classifiers=( 'Development Status :: 5 - Production/Stable', 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'Intended Audience :: Developers', 'Natural Language :: English', 'Natural Language :: English', 'License :: OSI Approved :: Apache Software License', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python', 'Programming Language :: Python', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3', # 'Programming Language :: Python :: 3.0', # 'Programming Language :: Python :: 3.0', 'Programming Language :: Python :: 3.1', 'Programming Language :: Python :: 3.1', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.3', ), ),))
setup.pysetup.py modules modules
If project consists of a few modules this If project consists of a few modules this may be easiestmay be easiest
setup.pysetup.py packages packages
Need to explicitly list Need to explicitly list allall packages, not just packages, not just rootroot
setup.pysetup.py scripts scripts
Add executable Python files hereAdd executable Python files here
setup.pysetup.py non-Python files non-Python files
● add files to add files to MANIFEST.inMANIFEST.in (include in (include in package)package)
● add files to add files to package_datapackage_data in in setup.pysetup.py (include in install) Not recursive(include in install) Not recursive
MANIFEST.inMANIFEST.in language language
● include|exclude pat1 pat2 ...include|exclude pat1 pat2 ...● recursive-(include|exclude) dir pat1 pat2 ...recursive-(include|exclude) dir pat1 pat2 ...● global-(include|exclude) dir pat1 pat2 ...global-(include|exclude) dir pat1 pat2 ...● prune dirprune dir● graft dirgraft dir
http://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-chttp://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-cmdmd
setup.pysetup.py classifiers classifiers
Almost 600 different classifiers.Almost 600 different classifiers.
Not used by pip to enforce versions. For UI Not used by pip to enforce versions. For UI onlyonly
Create sdistCreate sdist
$ python setup.py sdist$ python setup.py sdist
PyPiPyPi
Validate Validate setup.pysetup.py::
$ python setup.py check$ python setup.py check
PyPi RegisterPyPi Register
Click on “Register” on right hand boxClick on “Register” on right hand box
https://pypi.python.org/pypi?%3Aaction=rehttps://pypi.python.org/pypi?%3Aaction=register_formgister_form
PyPi UploadPyPi Upload
$ python setup.py sdist register $ python setup.py sdist register uploadupload
PyPi Upload (2)PyPi Upload (2)$ python setup.py sdist register upload$ python setup.py sdist register upload......Creating tar archiveCreating tar archiveremoving 'rst2odp-0.2.4' (and everything under it)removing 'rst2odp-0.2.4' (and everything under it)running registerrunning registerrunning checkrunning checkWe need to know who you are, so please choose either:We need to know who you are, so please choose either: 1. use your existing login, 1. use your existing login, 2. register as a new user, 2. register as a new user, 3. have the server generate a new password for you (and email it to you), 3. have the server generate a new password for you (and email it to you), oror 4. quit 4. quitYour selection [default 1]:Your selection [default 1]:11
PyPi Upload (3)PyPi Upload (3)
Username: mharrisonUsername: mharrisonPassword:Password:Registering rst2odp to http://pypi.python.org/pypiRegistering rst2odp to http://pypi.python.org/pypiServer response (200): OKServer response (200): OKI can store your PyPI login so future submissions will be faster.I can store your PyPI login so future submissions will be faster.(the login will be stored in /home/matt/.pypirc)(the login will be stored in /home/matt/.pypirc)Save your login (y/N)?ySave your login (y/N)?yrunning uploadrunning uploadSubmitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypiSubmitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypiServer response (200): OKServer response (200): OK
PyPi NotePyPi Note
Though PyPi packages are signed there is Though PyPi packages are signed there is no verification step during package no verification step during package installationinstallation
Non PyPi URLNon PyPi URL
$ pip install --no-index -f $ pip install --no-index -f http://dist.plone.org/thirdparty/ http://dist.plone.org/thirdparty/ -U PIL-U PIL
Personal PyPiPersonal PyPi
https://github.com/benliles/djangopypihttps://github.com/benliles/djangopypi
PyPiPyPiMakefileMakefile integration: integration:
# --------- PyPi ----------# --------- PyPi ----------.PHONY: build.PHONY: buildbuild: envbuild: env<TAB>$(PY) setup.py sdist<TAB>$(PY) setup.py sdist
.PHONY: upload.PHONY: uploadupload: envupload: env<TAB>$(PY) setup.py sdist register upload<TAB>$(PY) setup.py sdist register upload
TestingTesting
TestingTesting
Add tests to your projectAdd tests to your project
Testing (2)Testing (2)
● use use doctestdoctest or or unittestunittest
Testing (3)Testing (3)Use Use nosenose to run: to run:
$ env/bin/nosetests$ env/bin/nosetests....--------------------------Ran 2 tests in 0.007sRan 2 tests in 0.007s
OKOK
Testing (4)Testing (4)MakefileMakefile integration: integration:NOSE := env/bin/nosetestsNOSE := env/bin/nosetests# --------- Testing ----------# --------- Testing ----------.PHONY: test.PHONY: testtest: nose depstest: nose deps<TAB>$(NOSE)<TAB>$(NOSE)
# nose depends on the nosetests binary# nose depends on the nosetests binarynose: $(NOSE)nose: $(NOSE)$(NOSE): env$(NOSE): env<TAB>$(PIP) install nose<TAB>$(PIP) install nose
GithubGithub
GithubGithub
Just a satisfied userJust a satisfied user
Why Github?Why Github?
Not bazaar nor mercurialNot bazaar nor mercurial
Why Github? (2)Why Github? (2)
Code is a first class objectCode is a first class object
GithubGithub
Don't check in keys/passwords!Don't check in keys/passwords!
http://ejohn.org/blog/keeping-passwords-ihttp://ejohn.org/blog/keeping-passwords-in-source-control/#postcommentn-source-control/#postcomment
A Branching StrategyA Branching Strategy
http://github.com/nvie/gitflowhttp://github.com/nvie/gitflow
Travis CITravis CI
Travis CITravis CI
CI (continuous integration) for GithubCI (continuous integration) for Github
Travis CI (2)Travis CI (2)
Illustrates power of (web) hooksIllustrates power of (web) hooks
5 Steps5 Steps
● Sign-in with githubSign-in with github● Sync repos on Profile pageSync repos on Profile page● Enable repoEnable repo● Create a Create a .travis.yml.travis.yml file on github file on github● Push a commit on githubPush a commit on github
travis.ymltravis.ymllanguage: pythonlanguage: pythonpython:python: - "3.3" - "3.3" - "2.7" - "2.7" - "2.6" - "2.6" - "pypy" - "pypy"# command to run tests# command to run testsscript: make testscript: make test
More InfoMore Info
http://about.travis-ci.org/http://about.travis-ci.org/
PoachplatePoachplate
poachplatepoachplate
Example of most of what we have talked Example of most of what we have talked about todayabout today
https://github.com/mattharrison/poachplathttps://github.com/mattharrison/poachplatee
That's allThat's all
Questions? Tweet or email meQuestions? Tweet or email [email protected]@gmail.com
@__mharrison__@__mharrison__http://hairysun.comhttp://hairysun.com