ESGF + DOCKER Luca Cinquini NASA/Jet Propulsion Laboratory + California Ins:tute of Technology ESGF F2F Workshop, Washington, DC, December 2016
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
ESGF + DOCKER
LucaCinquiniNASA/JetPropulsionLaboratory+CaliforniaIns:tuteofTechnology
ESGFF2FWorkshop,Washington,DC,December2016
DockerinaNutshell
• Dockeristheleading“containeriza:on”technology:runanapplica:onasa“blackbox”onanyDocker-enabledserver‣ Build:imagesarebuiltasbundlesthatincludetheapplica:onitself,allofitsdependencies,and“just-enough-opera:ng-system”‣ Ship:imagesarehostedononlinerepositoriessuchasDockerHub‣ Run:imagesarerunascontainersonanyhostthatincludesaDockerdaemon
•WhyusingDockerforESGF?‣ Promisestogreatlyimproveinstalla:onandmaintenanceofanESGFnode‣ PartofDREAMstrategyformodularizingtheESGFarchitectureandpor:ngittootherdomains
Demo:InstallandRunanESGFNode
• Pre-requisite:installDockerEngineonhost(Linux,MacOSX,Windows)• Instruc:ons:‣ gitcloneh`ps://github.com/ESGF/esgf-docker.git‣ cdesgf-docker‣ exportESGF_HOSTNAME=<hostFQDN>‣ exportESGF_CONFIG=<somedirectory>‣ ./esgf_node_init.sh‣ docker-composeup
h`ps://hub.docker.com/esggub/
h`ps://my-node.esgf.org/
CoG
TDS
Solr
ESGF/DockercompleteNodeArchitecture
• AllESGFservicesarerunasindependent,interac:ngDockercontainers• SpecificNodeconfigura:on(cer:ficates,passwords,XMLfiles)savedin$ESGF_CONFIG• SpecificNodedata(Postgresdb,Solrindexes,TDScatalogs,NetCDFfiles,CoGsitemedia)storedonDockervolumes
• Customnetworksisolateapplica:onsforaddi:onalsecurity(forexample,Postgresdb)
Threads Data Server+ OpenID Relying Party
(Tomcat)
ESGF DATA NODE
ESGF POSTGRES DB
TDS CATALOGS
ESGF CONFIG DATA
ESGF SOLR(Jetty)
SOLR DATA
ESGF CONFIGURATION(site specific)
esgf-search webapp(Tomcat)
ESGF INDEX NODE
ESGF COG
COG DATA
httpd + mod_wsgi
ESGF WEBUI NODE
ESGF IDP NODE
esgf-idp webapp(Tomcat)
ESGF/Docker complete Node Architecture
POSTGRES DATA
ESGF DATA
ESGF FTP
docker-compose-fdocker-compose.ymlup
DockerDetails
Dockerfile:“recipe”forbuildingaDockerimage
docker-compose.yml:configura:onfileforbundlingseveralimages
ESGF/DockerDataNodeArchitecture
docker-compose-fdocker-compose-data-node.ymlup
Threads Data Server+ OpenID Relying Party
(Tomcat)
ESGF DATA NODE
ESGF POSTGRES DB
TDS CATALOGS
ESGF CONFIG DATA
ESGF CONFIGURATION(site specific)
ESGF/Docker Data Node Architecture
POSTGRES DATA
ESGF DATA
ESGF FTP
ESGF/DockerIndexNodeArchitecture
docker-compose-fdocker-compose-index-node.ymlup
UsingstandardSolrreplica:on+distributedsearch
ESGF CONFIG DATA
ESGF SOLR(Jetty)SOLR DATA
ESGF CONFIGURATION(site specific)
esgf-search webapp(Tomcat)
ESGF INDEX NODE
httpd + mod_wsgi
ESGF WEBUI NODE
ESGF/Docker Index Node Architecture
Slave Shard Master Shard
ESGF/DockerIndexNodewithSolrCloud
docker-compose-fdocker-compose-solr-cloud.ymlup
ESGF CONFIG DATA
ESGF SOLR CLOUD(Jetty)SOLR DATA
ESGF CONFIGURATION(site specific)
esgf-search webapp(Tomcat)
ESGF INDEX NODE
httpd + mod_wsgi
ESGF WEBUI NODE
ESGF/Docker Solr Cloud Architecture
Shard 1 Shard 2 Shard 3ESGF SOLR CLOUD
(Jetty)
Shard 1 Shard 2 Shard 3ESGF SOLR CLOUD
(Jetty)
Shard 7 Shard 8 Shard 9
• Solr-Cloudadvantages:‣ Automa:cdistributedindexingandsearching(nocustomconfigura:on)‣ Loadbalancingandhighavailability‣ Faulttolerance‣ Scalability
AdvantagesofusingDockerforESGF
• Installa:on:‣ Installa:onscriptsaremuchmoremodularandmuchsmaller‣ Installa:onprocessismusheasier:simplydownloadtheimages,nocompila:oninvolved‣ Easierupgrades,possiblyonemoduleata:me,andreversible‣ Everybodyrunsexactlythesamesokware
• Architecture:‣ Candefineanddeploynewarchitecturesbysimplywri:ngnewconfigura:onfiles‣ Canintroducenewmodulesbysimplywri:ng&wiringnewimages(solr-cloud,nginx,…)
• Portability:ESGFnodewillrunonanyplanorm(Linux,MacOSx,Windows)includingCloud
• Scalability:modulescanbescaledarbitrarilybyrunningmorecontainers(e.g.TDS,WPS…)‣ Caveat:applica:onmustbewri`entoenabledistributedaccesstodata
• OtherDocker/Swarmadvancedfeatures:‣ Scalabilityontomul:plehostsclusters‣ Highavailability,faulttolerance,rollingupdates,…
DisadvantagesofusingDockerforESGF
•Dockerisanewparadigmforbuildingandrunningapplica:ons:‣ Newknowledgeforapplica:ondevelopers‣ Newtrainingfornodeadministrators‣ Excellentandup-to-datedocumenta:onisamust
•MustporttheremainingESGFmodulestoDocker:‣ ESGFPublisherClient‣ Globus‣MyProxyornewOAuthserver‣ Desktop+Dashboard‣ LAS
•Mustdevelopaprocesstomigrateallapplica:ondata:‣ ESGFPostgresdatabase(usersanddata)‣ TDScatalogs‣ Solrindexes‣ CoGPostgresdatabaseandsitedata
PossibleFutureRoadmap
•Establisha“whaleteam”ofexpertstoworkon:‣ Testthecurrentinfrastructure‣ Porttheremainingmodules‣Workonotheroutstanding:ckets‣ Developates:ngsuite‣ Developdatamigra:ontools‣ Reviseandexpandthedocumenta:on‣ Re-usecurrentesgf-iwtbiweeklymee:ngs?
•Migra:ngtoDockerwouldtake6-12months‣Mustsupportcurrentorupgradedinstalledinthemean:me‣MightwanttoswitchakerCMIP6-lookingintothelong-termlongevityofESGF…
Related Documents
![Docker 101 - techccu.csie.iotechccu.csie.io/2015/slides/frank.pdf · Docker Basics - CLI Docker client docker version docker info docker search [keyword] docker push/pull/commit docker](https://static.cupdf.com/doc/110x72/5f05ce717e708231d414cd40/docker-101-docker-basics-cli-docker-client-docker-version-docker-info-docker.jpg)
