Harnessing the Power of Python in ArcGIS using the Conda ...€¦ · 2016 Esri Developer Summit--Presentation, 2016 Esri Developer Summit, Harnessing the Power of Python in ArcGIS

HarnessingthePowerofPythoninArcGISUsingtheCondaDistributionShaunWalbridge

MarkJanikas

TingLee

https://github.com/scw/conda-devsummit-2016-talk

HandoutPDF

HighQualityPDF(2MB)

Conda

CondaBrandnew:thoughtitwasmoreimportanttoshowittoyouthantofocusontellingyouaboutitTimetodaytodiscussyourneedsandwhatwemightdotosolveyourproblems

WhyPython?Accessiblefornew-comers,andthe

Extensivepackagecollection(56thousandon ),broaduser-baseStronggluelanguageusedtobindtogethermanyenvironments,bothopensourceandcommercialOpensourcewithliberallicense—dowhatyouwant

mosttaughtfirstlanguageinUSuniversites

PyPI

PackageManagementforPython

Whynotpip,wheels,virtualenvs?Don’thandletheharderproblemofsystemdependencies,consideredoutofscopebyPythonpackagers—doesitendupinsite-packages?Packagedevs:OnOSXandLinux,‘easy’togetthedeps!Useasystempackagemanager(e.g.apt,brew,yum)andtheincludedcompiler(e.g.clang,gcc).It’sstillnoteasytomakereproduciblebuilds,andwhataboutWindows?

WhataboutWindows?WeareparticularlystuckonWindowswhichlacksbroadlyusedpackagemanagement

NuGetisgreat,butnotasystem-levelpackagemanagerIfmanagingapplications,try

OnlydevshaveaCcompilerontheirmachineTheessentialmodeliscompilersforfew,runtimesforall

Packagemanagementishard!(ExceptonJavaScript–universalcompilersarealeg-up)

Chocolatey

WhataboutWindows?WeareparticularlystuckonWindowswhichlacksbroadlyusedpackagemanagement

NuGetisgreat,butnotasystem-levelpackagemanagerIfmanagingapplications,try

OnlydevshaveaCcompilerontheirmachineTheessentialmodeliscompilersforfew,runtimesforall

Packagemanagementishard!(ExceptonJavaScript–universalcompilersarealeg-up)

EnterConda

Chocolatey

WhyConda?

ScientificPythoncommunityidentifiedthattherewasagapnotbeingaddressedbythecorePythoninfrastructure,limitingtheirabilitytogetpackagesintothehandsofusers

Industrystandardbuiltbypeoplewhocareaboutthisspace–ContinuumAnalytics

WhyConda?

Itsolvesahardproblem:

Handlesdependenciesformanylanguages(C,C++,RandofcoursePython)

BuiltforPythonfirst,butitreallysolvesamuchbroaderinfrastructuralissue.

Conda

CondaCross-platform:simplydeveloprecipesforbuildingandinstallingsoftwareonLinux,OSXandWindows.Allittakes:ameta.yaml,andabuildrecipe.Opensource(BSD):Esriisusingit,youcanuseitinyourownprojectsforothercontexts

CondaWhatcanitinstall?Notjustscientificpackages.Itcanhelpwith:

GUItoolkits(PyQt,TKinter)C++Libraries(Boost)IDEs(Spyder,Juptyer)

See foracomprehensivesetofbuildrecipes.EverythingfromapplicationstocompilerstoPythonmodules,hundredsofmaintainedrecipesacrossmanyproblemdomains.

conda-recipes

CondaEnvironments:CanisolateaPythonenvironment,flexiblymakechangeswithotaffectinginstalledsoftware.Requirements–includeexplicitstateinformation,notjustthepackagename.Namesaren’tenough!AlsohandlesplatformsandJupyternotebooks

HowDoesitWork?Condapackagescancomefromavarietyoflocations:

Ondisk(file://)PublicrepositorieshostedonAnacondaCloudPublicrepositoriesself-hostedPrivaterepositoriesPaidprivaterepositories

CondaBasicsCommandlineinterface

Willshowwhatwe’reworkingontomakethiseasier,especiallyfornon-developers

CondaCheatsheet

CondaBasicsTostart:

AcollectionofpackagesandPythoninstalliscalledanenvironmentorenv,thebuildingblockformanagingPythonwithCondaCanhavemultipleenvironmentsandseamlesslyswitchbetweenthem

conda --help

CondaBasicsActivatingenvironments,acoupleways:

UsetheshortcutsManuallyactivatetheenvironment:

cd /d C:\ArcGIS\bin\Python\Scripts activate arcgispro-py3

CondaBasicsOnceyou’reinanenvironmentgetdetailswithinfo:

Condainfoisthestartingpoint–ittellsyouthestateoftheenvironment.

conda info

CondaBasicsconda info

Current conda install:

platform : win-64 conda version : 4.0.4 conda-build version : not installed python version : 3.5.1.final.0 requests version : 2.9.1 root environment : C:\ArcGIS\bin\Python (writable) default environment : C:\ArcGIS\bin\Python\envs\arcgispro-py3 envs directories : C:\ArcGIS\bin\Python\envs package cache : C:\ArcGIS\bin\Python\pkgs channel URLs : https://conda.anaconda.org/esri/win-64/ https://conda.anaconda.org/esri/noarch/ https://repo.continuum.io/pkgs/free/win-64/ https://repo.continuum.io/pkgs/free/noarch/ config file : C:\ArcGIS\bin\Python\.condarc

CondaBasicsconda list

# packages in environment at C:\ArcGIS\bin\Python\envs\arcgispro-py3:#arcgispro 1.0 0 esrimatplotlib 1.4.3 np19py34_0 defaultsnose 1.3.7 py34_0 defaultsnumpy 1.9.3 py34_0e [arcgispro] esripandas 0.17.1 np19py34_0 esripip 8.0.3 py34_0 defaultspyparsing 2.0.3 py34_0 defaultspypdf2 1.25.1 py_0 esripython 3.4.4 2 defaultspython-dateutil 2.4.2 py34_0 defaultspytz 2015.7 py34_0 defaultsscipy 0.16.1 np19py34_0e [arcgispro] esrisetuptools 20.1.1 py34_0 defaultssix 1.10.0 py34_0 defaultssympy 0.7.6.1 py34_0 defaultswheel 0.29.0 py34_0 defaultsxlrd 0.9.4 py34_0 defaultsxlwt 1.0.0 py34_0 defaults

CondaBasicsCreatingnewenvironments:

Afewdifferentways.Canmanuallyspecifythedependencies:

Canalsouseafilewhichincludesallthedependencies:

Thesecancontainexplcitinformationaboutchannels,toensurethatthenewenvironmentpreciselymatchestherequirements.

conda create --name my_env python=3.4 numpy flask dask

conda create --name my_env --file my_sweet_depends.txt

Condavs…Name Means Will

Ship?

Conda Thecommanditself ✓Miniconda AminimumsetofPythonpackagesto

buildandrunConda.✓

Anaconda Adistribution200+packagesbuiltwithConda

AnacondaServer

Hostthefullinfrastructureinternally

CondaDemo

DeeperDive

CondaBehindFirewallHow’sitwork?

Lockitdown:Don’tusenetwork

Canvettheinstallation

Willworkoutoftheboxwithdefaultpackageswithoutanynetworkconnectivity

.condarcModifydefaultswithasimplesimpleYAMLfileforconfigurationCanbeupdatedwithconda config,justlikeusinggit configtoupdatethedefaultconfiguration

Adetailedexample.condarc

CreatingpackagesStraightforward:

Ametadatadocument(meta.yaml)specifyingthecontentsanddependenciesAbuildcommand(bld.bat,build.sh)specifyinghowtobuild

Creatingpackagesmeta.yaml:

package: name: pypdf2 version: "1.25.1"

source: fn: PyPDF2-1.25.1.tar.gz url: https://pypi.python.org/packages/source/P/PyPDF2/PyPDF2-1.25.1.tar.gz md5: ee5e5b01d00b120805e5049e56c6fd7c

requirements: run: - python

Creatingpackagesbld.bat:

"%PYTHON%" setup.py install

MultiplePythonsCurrently:

Platform Pythonversion

Desktop Python2.7.x(2.7.10)

Pro Python3.4.x(3.4.3)

MultiplePythonsUpgradecode?

Doitalready!Youcansupport2+3withoutthatmuchworkIfyouhitanissue,it’sprobablybecauseyoudon’tunderstandUnicodeyet– ,PragmaticUnicode,or,HowdoIstopthepain?

PythonmigrationforArcGISPro

WatchthisPyContalk

MultiplePythonsUpgradecode?

Doitalready!Youcansupport2+3withoutthatmuchworkIfyouhitanissue,it’sprobablybecauseyoudon’tunderstandUnicodeyet– ,PragmaticUnicode,or,HowdoIstopthepain?

But…thiscanbecostly.Formanyorganizations,asignificantburden,evenifthelanguagechangesarerelativelysmall.

PythonmigrationforArcGISPro

WatchthisPyContalk

MultiplePythonswithConda

WithConda,wecansupportmultipleplatforms:

Py2.7,3.4,3.5inPro1.3

Createanewenvironment,targetadifferentPython,userscannowusethatwiththePy2code

Stillneedtochangearcpy.mappingtoarcpy.mpwhenmovingfromDesktoptoPro,butnoPythonlanguagelevelchangesneeded.

ChallengesHavetomakesureyou’rerunningtherightPython(whathappens

whenyoutypepythonatthecommandline?)WewillmakethiseasyaspossibleIt’llbeeasytotellinappIsolatedinstallationfixesavarietyofissues

Requiressomeusereducationoverthe“onlyonePythononthebox”model

WhatDoIGetOutoftheBox?

CondacommandandaCondarootPythoninstallNewmodules(e.g.requests)CondaenvironmentwithalloftheArcGISProdependenciesasCondapackages

HowcanIusethis?WealreadyshipyoutheSciPystack–powerfulandoutofthebox,canusetoday(Proand10.4)Canstartusingcondatoday.Minicondaisfullystand-alone,won’taffectyourglobalPython(unlessyoutellitto)Packageyourwork:thisisanopportunitytodistributeit,possiblyincludingcommercialsideaswell.

WhereCanIRunThis?

ArcGISPro1.3(Release:2016UC)WillbethePythoninstall.UIforinteraction

Future:TakeadvantageofmorefeaturesIntegrationwithplatform

fromfutureimport*EffectivelymanagecomplexsoftwaredependencieswithConda.Thousandsofpackagesexisttoday,canintegrateitintoyour

organization’sneeds.

Resources

ResourcesCondaRecipes

Anaconda.org

CondaCheatsheet

Closing

ThanksEsriCondaTeam:

ContinuumAnalyticsforcreatingandopensourcingConda

RateThisSessioniOS,Android:Feedbackfromwithintheapp

WindowsPhone,don’tuseasmartphone?:Cuniformtabletsaccepted(sorry!limitation).

RateThisSessioniOS,Android:Feedbackfromwithintheapp

WindowsPhone,don’tuseasmartphone?:Cuniformtabletsaccepted(sorry!limitation).

WindowsPhone,ornosmartphone?Cuneiformtabletsaccepted.