Status of the Open Data Cube - CEOSceos.org/document_management/Working_Groups/WGISS/Meetings/W… · Status of the Open Data Cube ... § The Open Data Cube (ODC) initiative was established

Status of the Open Data Cube

Brian KilloughCEOS Systems Engineering Office (SEO)

WGISS-44 MeetingSeptember 27, 2017

What are Data Cubes?

• DataCube=Time-seriesmulti-dimensional(space,time,datatype)stackofspatiallyalignedpixelsreadyforanalysis

• ProvenconceptbyAustraliawithplansforglobalimplementation

• AnalysisReadyData(ARD)...Dependentonpre-processedproductstoreducetheburdenonusers

• Opensourcesoftwareapproachallowsfreeaccess,promotesexpandedcontributions,andincreasesdatausage.

• Uniquefeatures:exploitstimeseries,increasesdatainteroperability,andsupportsmanynew applications. TIME

DataCubesareapopulartopic…“TheDataCubeManifesto”(PeterBaumann,EU)and“TheSixFacesoftheDataCube”(PeterStrobl,EC)

Benefits of Data Cubes

§ ExpandeduseofCEOSsatellitedata…expandeduserbase§ Reducedprocessingburden..dependencyonARD§ Enhancedinteroperability...improvedbyMRI§ Efficienttimeseriesanalyses§ Freeandopenaccess§ Flexibledeployment(localorcloud)§ Useofacommonarchitecture§ Communitydevelopmentandsharing…viaGitHub

OurgoalisNOT tosellaproductordistributeatool.OurgoalistoprovideaSOLUTION thathasVALUE andincreasestheIMPACT ofsatellitedata.

Open vs. CEOS Data Cubes

§ TheODCinitiativeislargerthanCEOS.§ TheOpenDataCube(ODC)initiativewasestablished

byCEOS,withagoaltocreateandfosteranopen“community”ofcontributors.

§ TheODC usesacommonarchitectureamongthevariousimplementationssothatalluserscansharetoolsandapplications.

§ TheCEOSDataCube(CDC)isone“implementation”oftheODC.Similarly,DigitalEarthAustralia(DEA)andUSGSLandChangeMonitoring,Assessment,andProjection(LCMAP)areimplementations.

§ TheCDC goalistofocusonbuildingglobalcapacitytoutilise satellitedataandcontributetoglobalinitiatives(e.g.UN-SDG,GFOI,GEOGLAM)throughtheuseofDataCubes.

CEOS Data Cube Vision

AsolutionsupportingCEOSobjectives…§BuildcapabilityofuserstoapplyCEOSsatellitedata§ SupportingpriorityCEOS/GEOagendasandSDGs

CEOSAgencieswantingtoparticipate …§ ThroughprovisionofCEOSAnalysisReadyData(ARD)products§Contributingtodevelopmentanduptakeofsolutions

Customerfocused…§ Trainingmaterialsandeasyinstallation/maintenance§Abrandthatpeopleknowandtrust§Anactivecommunityofusers

Scalablesolution…§OperationalDataCubesin20countriesby2022§Keypartners(e.g.GEO,WorldBank)supportingdatacubeprojects

The “Road to 20” Operational Data Cubes by 2020

Operational

Under Development

Under Review or Expressed Interest

3 operational, 4 under development, 22 under review = 29 total

The “Road to 20” Highlights

§ Colombia hasanoperational DataCubesinceDec2016withover25,000historicLandsatimages.Theycontinuetoexpandtheuserbase,applicationsanddatasets.TheColombiaDataCubewontheNationalEnvironmentalAwardofColombianSocietyofEngineersinMay2017andhasbeenapprovedbytheColombiaGovernmentinto2018.

§ Switzerland hasanoperational DataCubesinceJuly2017withover4,000historicLandsatimages.TheyhavereceivedSwissgovernmentapprovalanddevelopedanewwebsite(swissdatacube.org).Theirfutureplansincludeexpandeddatasets(Sentinel)andincreasedapplicationswithbothgovernmentanduniversityinvolvement.

§ Vietnam isslowlymakingprogressbyestablishingpilotcubesinseveralregionsusinganewhighperformancecomputingsystem.Theirfocusisonforests,rice,andwaterapplications.VNSCishostinganinternalDataCubeWorkshoponSept17.

§ Taiwan ismakingprogressonalocalHPCinstallationthroughsupportfromCSIRO.Theirfocusisforestsandwaterapplications.

§ Uganda hasreceivedsupportfromtheU.K.toinstallademocubefortheKaramojaregiononacloud(AWS).TheyhavemaderapidprogresswithlittleCEOSsupport.

§ Seethe“Roadto20”documentontheODCwebsiteformore!

Other Highlights

§ Plansareinplacetoleverage theexperienceofseveraloperationalimplementationstoexpandthepresenceofDataCubes…§ Switzerland>>>Georgia,Moldova§ U.K.>>>SolomonIslands,Vanuatu,Nauru§ Taiwan>>>Hondurus

§ WearemakingprogresswithWorldBanktosupportthedeploymentofaDataCubeinUruguaytosupportanagricultureandwaterqualityprojectwithdirectlinkstoDINAMA(UN-SDGstatisticalagency).

§ 4DataCubesideeventsareplannedforGEO-17 onOct23-24.Each1.5hoursegmentwillhaveadifferentDataCubetopic.

§ FutureoutreachopportunitiesatPecora-20 (USGS)andIGARSS-2018(JulyinValencia,Spain).

ODC Progress§ Wehaveestablishedan“ODCPartners”groupwhichincludesrepresentatives

fromtheNASA-SEO,GA,CSIRO,USGS,andUK-Catapult.§ Wehaveestablishedan“ODCSteering”groupwhichincludetechnical

representativesfromNASA-SEO,GA,CSIRO,andUSGS.§ Wehaveestablishedanewwebsitehttps://opendatacube.org§ Wehavedeveloped“whitepapers”fortheODCandCDCthatdescribethegoalsof

eachinitiativeandanODCgovernancedocumentforcodemanagement.§ WeconductedthefirstODCWorkshopattherecentIGARSSconferenceinFort

Worth,Texas,USAinJuly.§ Weareplanningthe2nd Annual

ODCTechnicalMeetinginCanberra,AustraliaonFeb14-16,2018.

Technical Progress

§ TheCDChasestablisheddetailedcontenttosupportDataCubedeployments§ Installation – systemrequirements,installationguide§ DataPreparation – ARDguidance,dataacquisitionguidance§ DataCubeCreation– ingestors forallpopulardatasets§ Applications – AWSdemo,Pythonnotebooks,growinglistof

algorithms§ Forum – discussiongroupsforusersupport

§ DataCubeingestionhasdemonstratedsignificantreductionindatastoragerequirementswhencomparingtheingestedDataCubestotheoriginaldata.o Landsat =3xto7xreduction(varieswithdataparameterselections).

Forexample,a1-degx1-degx1-yearLandsatDataCubeis~900MB.o Sentinel-1GRD=6xreduction(basedon30mgrid,VVandVHonly)

DataCubes§ 16cubeswith10+yearseach.§ Kenya,Cameroon(LakeChad),Togo(coastalAfrica),Ghana,Colombia,Tonga(PacificIsland),Vietnam,Australia(MenindeeLakes),Bangladesh.

UserInterfaceFeatures§ 9applications:cloudcoveragemaps,customcloud-freemosaics,fractionalcover,NDVIanomaly,waterdetection,waterquality,landslides,coastalchangeandurbanization.

§ OutputsinGeoTIFFandGIFanimation.§ NewfeaturesaddedinSept2017:datavisualizationtools,ingestion“ondemand”fornewcubesorsubsetting,indices,mosaics(medoid,geometricmedian)

http://tinyurl.com/datacubeuiFree and Open!

Thisisthefirst“hands-on”globaldemooftheDataCubetoshowitspotentialforrapidtimeseriesanalysisanddiverseapplications

Amazon (AWS) Demo Portal

Near-term Developments

§ DevelopanewQGIStoolpluginwithaweb-based(WCS)connectiontoaDataCubehostedonAWS(cloudstorage).ThiswillbereadybyNov2017.

§ DevelopandtestsampleiPython NotebooksonAWStodemonstrateinteractiveDataCubeapplicationsandprogrammingsimplicity

§ TestthePyCCD landchangedetectionalgorithmwithradardatasets

§ DevelopandtestanewWaterQualityalgorithmfromTonyVodacek (LandsatScienceTeam)basedonaLook-Up-TableapproachtoinferChlorophyll-A,CDOMandTSSconcentrations.

§ TestSentinel-1GRDandSLC cubeswiththeRandomForestlandclassificationclusteringalgorithm

Lessons Learned

Throughourinitialcountryinteractions,wehavelearnedanumberoflessons …

§ CountryusersshouldhavesomePythonprogrammingskills§ Itisimportanttoclearlyunderstandcountryneedsandtoguidethemtoward

theneededsatellitedataandapplicationtools§ Itisimportanttomaintainconsistentcustomercommunication(bothface-to-

faceandremote)tosustaindeploymentprogressandbuildtrust§ Itisimportanttoutilise relationshipswithinvestmentbanks(e.g.WorldBank)

andGEOtoincreaseaccesstocountrycontactsandfacilitatedeployment§ TheODCcommunityneedstocontinuetogrowandexpandtobuildconfidence

towardsdesiredoutcomesandtobuildthesupplyofopensourcetoolsandapplications