Extending ArcGIS with Conda and R - 4326.us€¦ · 01/06/2016  · Extending ArcGIS ArcGIS is a system of record. ... from interactive environments like Spyder to Jupyter Notebooks.

ExtendingArcGISwithCondaandRShaunWalbridge

https://github.com/esrioceans/oceans-workshop-2016

DownloadPDF

ExtendingArcGIS

ExtendingArcGISArcGISisasystemofrecord.Combinedataandanalysisfrommanyfieldsandintoacommonenvironemnt.Whyextend?Can’tdoitall,wesupportover1000GPtools—enablingintegrationwithotherenvironmentstoextendtheplatform.DiscussedSciPyandwhat’s“inthebox”,nowdiscussthebroaderecosystemofopensource.

Today:RandCondaR-ArcGISBridge

ShortIntroductionDemo

Condaformanagingpackagesandenvironments

IntroductionDemoExercise

R

Why ?Powerfulcoredatastructuresandoperations

Dataframes,functionalprogrammingUnparalleledbreadthofstatisticalroutines

ThedefactolanguageofStatisticiansCRAN:6400packagesforsolvingproblemsVersatileandpowerfulplotting

RDataTypesyou’reusedtoseeing…

Numeric-Integer-Character-Logical-timestampDatatypes

RDataTypesyou’reusedtoseeing…

Numeric-Integer-Character-Logical-timestamp…butothersyouprobablyaren’t:

vector-matrix-data.frame-factor

Datatypes

DataFramesTreatstabular(andmulti-dimensional)dataasalabeled,indexedseriesofobservations.Soundssimple,butisagamechangerovertypicalsoftwarewhichisjustdoing2Dlayout(e.g.Excel)

DataTypes# Create a data frame out of an existing sourcedf.from.csv <- read.csv( "data/growth.csv", header=TRUE)

DataTypes# Create a data frame from scratchquarter <- c(2, 3, 1)person <- c("Goodchild", "Tobler", "Krige")

met.quota <- c(TRUE, FALSE, TRUE)df <- data.frame(person, met.quota, quarter)

DataTypesR> df person met.quota quarter1 Goodchild TRUE 22 Tobler FALSE 33 Krige TRUE 1

0D:SpatialPoints1D:SpatialLines2D:SpatialPolygons3D:Solid4D:Space-time

spTypesEntity+Attributemodel

StatisticalFormulas

DomainspecificlanguageforstatisticsSimilarpropertiesinotherpartsofthelanguage

formodelspecificationconsistency

fit.results <- lm(pollution ~ elevation + rainfall +

caret

R—ArcGISBridge

R—ArcGISBridge

ArcGISdeveloperscancreatetoolsandtoolboxesthatintegrateArcGISandRArcGISuserscanaccessRcodethroughgeoprocessingscriptsRuserscanaccessorganizationsGIS’data,managedintraditionalGISways

https://r-arcgis.github.io

R—ArcGISBridgeStoreyourdatainArcGIS,accessitquicklyinR,returnRobjectsback

toArcGISnativedatatypes(e.g.geodatabasefeatureclasses).

Knowshowtoconvertspatialdatatospobjects.

PackageDocumentation

AccessArcGISfromRStartbyloadingthelibrary,andinitializingconnectiontoArcGIS:

# load the ArcGIS-R bridge librarylibrary(arcgisbinding)# initialize the connection to ArcGIS. Only needed when running directly from R.arc.check_product()

AccessArcGISfromRFirst,selectadatasource(canbeafeatureclass,alayer,oratable):

Then,filterthedatatothesetyouwanttoworkwith(createsin-memorydataframe):

ThiscreatesanArcGISdataframe–lookslikeadataframe,butretainsreferencesbacktothegeometrydata.

input.fc <- arc.open('data.gdb/features')

filtered.df <- arc.select(input.fc, fields=c('fid', 'mean'), where_clause="mean < 100")

AccessArcGISfromRFinishedwithourworkinR,wanttogetthedatabacktoArcGIS.Writeourresultsbacktoanewfeatureclass,witharc.write:

arc.write('data.gdb/new_features', results.df)

BuildingRScriptTools

RArcGISBridgeDemo

DetailsofmodelbasedclusteringanalysisintheRSampleTools

HowToInstallInstallwiththeRbridgeinstallDetailedinstallationinstructions

WhereCanIRunThis?

PackageswithConda

Whypackagemanagement?

Softwareiscomposedofmanysmallercomponents,oftencalledpackagesorlibraries.It’softenbettertoreusecodethatsolvesaproblemwellratherthanrecreatingitBut,sharingcodeisahardproblem.Doyouhavethesamepackagesofthesameversionsasthedeveloperdid?

WhyConda?

ScientificPythoncommunityidentifiedthattherewasagapnotbeingaddressedbythecorePythoninfrastructure,limitingtheirabilitytogetpackagesintothehandsofusers

Industrystandardbuiltbypeoplewhocareaboutthisspace—ContinuumAnalytics

WhyConda?

Itsolvesthehardproblem:

Handlesdependenciesformanylanguages(C,C++,RandofcoursePython)BuiltforPythonfirst,butitreallysolvesamuchbroaderinfrastructuralissue.

Conda

Cross-platform:SimplydeveloprecipesforbuildingandinstallingsoftwareonLinux,OSXandWindows.Opensource:Esriisusingit,youcanuseitinyourownprojectsforothercontexts

Whatcanitinstall?Notjustscientificpackages,caninstalleverythingfrominteractiveenvironmentslike to .Spyder JupyterNotebooks

Environments—CanisolateaPythonenvironment,flexiblymakechangeswithotaffectinginstalledsoftware.Requirements—includeexplicitstateinformation,notjustthepackagename.AlsohandlesplatformsandJupyternotebooks

Wheredopackagescomefrom?

Condapackagescancomefromavarietyoflocations:

manythousandsofpackagesRepositories(e.g.AnacondaCloud,self-hosted)Ondisk

anaconda.org

CondaBasics

Condainfoisthestartingpoint—ittellsyouthestateoftheenvironment.

conda --help

conda info

CondaBasicsconda list

# packages in environment at C:\ArcGIS\bin\Python\envs\arcgispro-py3:#colorama 0.3.7 py35_0 defaultscycler 0.10.0 py35_0 defaultsfuture 0.15.2 py35_0 defaultsmatplotlib 1.5.3 np111py35_0e [arcgispro] esrimpmath 0.19 py35_1 defaultsnetcdf4 1.2.4 py35_0e [arcgispro] esrinose 1.3.7 py35_1 defaultsnumexpr 2.6.1 np111py35_0e [arcgispro] esrinumpy 1.11.2 py35_0e [arcgispro] esripandas 0.19.0 np111py35_0 defaultspip 8.1.2 py35_0 defaultspy 1.4.31 py35_0 defaultspyparsing 2.1.4 py35_0 defaultspypdf2 1.26.0 py_0 esripytest 2.9.2 py35_0 defaultspython 3.5.2 0 defaultspython-dateutil 2.5.3 py35_0 defaultspytz 2016.6.1 py35_0 defaultsrequests 2.11.1 py35_0 defaultsscipy 0.18.1 np111py35_0e [arcgispro] esrisetuptools 27.2.0 py35_1 defaultssympy 1.0 py35_0 defaultswheel 0.29.0 py35_0 defaults

CondaBasicsActivatingenvironments,acoupleways:

UsetheshortcutsManuallyactivatetheenvironment:

cd C:\ArcGIS\bin\Python\Scripts activate arcgispro-py3

CondaBasicsAcollectionofpackagesandPythoninstalliscalledanenvironmentorenv,thebuildingblockformanagingPythonwithCondaCanhavemultipleenvironmentsandseamlesslyswitchbetweenthem

Condavs…Name Means Included?

Conda Thecommanditself ✓Miniconda AminimumsetofPythonpackagesto

buildandrunConda.✓

Anaconda Adistribution200+packagesbuiltwithConda

AnacondaServer

Hostthefullinfrastructureinternally

DeeperDive

HowcanIusethis?WealreadyshipyoutheSciPystack—powerfulandoutoftheboxinallproductsCondacommandandaCondarootPythoninstallNewmodules(e.g.requests),environmentwithProCondaUI

HowcanIusethis?WealreadyshipyoutheSciPystack—powerfulandoutoftheboxinallproductsCondacommandandaCondarootPythoninstallNewmodules(e.g.requests),environmentwithProCondaUIGetpackages,expandyourpossibilityspacePackageyourwork:thisisanopportunitytodistributeit,possiblyincludingcommercialsideaswell.

CanIRunThis?

ArcGISPro1.3+IsthethePythoninstall.

ArcGISPro1.4UIforConda

CanIRunThis?Future:

IntegrationwithplatformHelpmakethingslikeMGETevenbetterShareresultswithorganizationsandothers

CondaHands-onActivity

Hands-onExercise