Monitoring Docker - pepa.holla.czpepa.holla.cz/wp-content/uploads/2016/10/Monitoring-Docker.pdf · Zabbix Installing Zabbix Using containers Using vagrant Preparing our host machine

MonitoringDocker

TableofContents

MonitoringDocker

Credits

AbouttheAuthor

AbouttheReviewer

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmore

Whysubscribe?

FreeaccessforPacktaccountholders

Preface

Whatthisbookcovers

Whatyouneedforthisbook

Whothisbookisfor

Conventions

Readerfeedback

Customersupport

Downloadingtheexamplecode

Downloadingthecolorimagesofthisbook

Errata

Piracy

Questions

1.IntroductiontoDockerMonitoring

Pets,Cattle,Chickens,andSnowflakes

Pets

Cattle

Chickens

Snowflakes

Sowhatdoesthisallmean?

Docker

Launchingalocalenvironment

Cloningtheenvironment

Runningavirtualserver

Haltingthevirtualserver

Summary

2.UsingtheBuilt-inTools

Dockerstats

RunningDockerstats

Whatjusthappened?

Whataboutprocesses?

Dockertop

Dockerexec

Summary

3.AdvancedContainerResourceAnalysis

WhatiscAdvisor?

RunningcAdvisorusingacontainer

CompilingcAdvisorfromsource

Collectingmetrics

TheWebinterface

Overview

Processes

CPU

Memory

Network

Filesystem

Viewingcontainerstats

Subcontainers

Driverstatus

Images

Thisisallgreat,what’sthecatch?

Prometheus

LaunchingPrometheus

QueryingPrometheus

Dashboard

Thenextsteps

Alternatives?

Summary

4.ATraditionalApproachtoMonitoringContainers

Zabbix

InstallingZabbix

Usingcontainers

Usingvagrant

Preparingourhostmachine

TheZabbixwebinterface

Dockermetrics

Createcustomgraphs

Comparecontainerstoyourhostmachine

Triggers

Summary

5.QueryingwithSysdig

WhatisSysdig?

InstallingSysdig

UsingSysdig

Thebasics

Capturingdata

Containers

Furtherreading

UsingCsysdig

Summary

6.ExploringThirdPartyOptions

Awordaboutexternallyhostedservices

DeployingDockerinthecloud

WhyuseaSaaSservice?

SysdigCloud

Installingtheagent

Exploringyourcontainers

Summaryandfurtherreading

Datadog

Installingtheagent

Exploringthewebinterface


NewRelic

Installingtheagent

Exploringthewebinterface


Summary

7.CollectingApplicationLogsfromwithintheContainer

Viewingcontainerlogs

ELKStack

Startingthestack

Logspout

Reviewingthelogs

Whataboutproduction?

Lookingatthirdpartyoptions

Summary

8.WhatAretheNextSteps?

Somescenarios

Pets,Cattle,Chickens,andSnowflakes

Pets

Cattle

Chickens

Snowflakes

Scenarioone

Scenariotwo

Scenariothree

Alittlemoreaboutalerting

Chickens

CattleandPets

Sendingalerts

Keepingup

Summary

Index

MonitoringDocker

MonitoringDockerCopyright©2015PacktPublishing

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

Firstpublished:December2015

Productionreference:1041215

PublishedbyPacktPublishingLtd.

LiveryPlace

35LiveryStreet

BirminghamB32PB,UK.

ISBN978-1-78588-275-3

www.packtpub.com

http://www.packtpub.com

CreditsAuthor

RussMcKendrick

Reviewer

MarceloCorreiaPinheiro

CommissioningEditor

VeenaPagare

AcquisitionEditor

RahulNair

ContentDevelopmentEditor

AnishSukumaran

TechnicalEditor

SaurabhMalhotra

CopyEditor

TrishyaHajare

ProjectCoordinator

IzzatContractor

Proofreader

SafisEditing

Indexers

MariammalChettiyar

PriyaSane

ProductionCoordinator

ShantanuN.Zagade

CoverWork

ShantanuN.Zagade

AbouttheAuthorRussMcKendrickisanexperiencedsolutionsarchitectwhohasbeenworkinginITandIT-relatedindustriesforthebetterpartof23years.Duringhiscareer,hehashadvariedresponsibilitiesinanumberofindustries,rangingfromlookingafterentireITinfrastructurestoprovidingfirstline,secondline,andseniorsupportinclientfacing,andinternalteamsforcorporateorganizations.

HeworksalmostexclusivelywithLinux,usingopensourcesystemsandtoolsonvariousplatformsrangingfromdedicatedhardwareandvirtualmachinestopublicclouds.

AbouttheReviewerMarceloCorreiaPinheiroisaBraziliansoftwareengineerfromPortoAlegre.Hestartedtoworkasawebdesignerandprogrammerin2000withASPandPHP,naturallygettingintouchwiththeMicrosoft.NETframeworkandJavarunningrespectivedatabasesofchoiceforwebapplications.Since2003,hehasusedLinuxandUNIX-relatedoperationalsystems,fromSlackwaretoGoboLinux,Archlinux,CentOS,Debian,andtodayOSX,havingsomecontactwithBSDdistributionstoo.HehaslostsomenightscompilingandapplyingpatchestotheLinuxkerneltomakeitsdesktopwork.Sincethebeginning,hehasbeenactingasaproblemsolver,nomatterwhattheprogramminglanguage,database,orplatformis—opensourceenthusiast.

Afterafewyears,hedecidedtoliveinSãoPaulotoworkwithnewertechnologiessuchasNoSQL,cloudcomputing,andRuby,wherehestartedtoconducttechtalkswiththislanguageinLocaweb.HecreatedsometoolstostandardizedevelopmentusingtoolssuchasvagrantandRubygems—someoftheseintheirGitHub—inLocawebtoensurefastapplicationpackagingandreduceddeploymentrollbacks.In2013,hechangedhiscareertobeafull-stackdeveloperfollowingtheDevOpsmovement.Since2012,hehasattended,asaspeaker,someofthebiggestsoftwareconferencesinBrazil—RSonRails,QConSP,TheDeveloper’sConference,andRubyConfBrazil—talkingnotonlyaboutRuby,butalsoaboutsomeofthewell-knownDevOpstoolssuchasTerraform,Packer,Ansible,andDocker.Today,heworksasaDevOpsconsultantintheircompany.

Inhisfreetime,helovesplayingtheguitar,havingsomefunwithcats,traveling,anddrinkingbeer.Hecanbefoundonhisblog(http://salizzar.net),Twitter(https://twitter.com/salizzar),GitHub(https://github.com/salizzar)andLinkedin(https://www.linkedin.com/in/salizzar).

HehasworkedasareviewerforVagrantVirtualDevelopmentEnvironmentCookbook,aPacktPublishingbookwithusefulrecipesusingvagrantwithconfigurationmanagementtoolssuchasPuppet,Chef,Ansible,andSaltStack.

Iwanttothankallmyfriends,whobelievedinmypotentialsincethebeginningandwhostillfollowmedespitethedistance.Iwouldalsoliketothankmymentors,GleiconMoraes,RobertoGaiser,andRodrigoCampos,whogavemetheincentiveandtipstobeabettersoftwareengineerandperson.

http://salizzar.net

https://twitter.com/salizzar

https://github.com/salizzar

https://www.linkedin.com/in/salizzar

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmoreForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.

DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<[email protected]>formoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

https://www2.packtpub.com/books/subscription/packtlib

DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.

http://www.PacktPub.com


mailto:[email protected]


https://www2.packtpub.com/books/subscription/packtlib

Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser

FreeaccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.


PrefaceWiththeincreaseintheadoptionofDockercontainers,theneedtomonitorwhichcontainersarerunning,whatresourcestheyareconsuming,andhowitaffectstheoverallperformanceofthesystem,hasbecomeatime-relatedneed.MonitoringDockerwillteachyouhowmonitoringcontainersandkeepingakeeneyeontheworkingofapplicationshelptoimprovetheoverallperformanceoftheapplicationsthatrunonDocker.

ThisbookwillcovermonitoringcontainersusingDocker’snativemonitoringfunctions,variousplugins,andalsothird-partytoolsthathelpinmonitoring.Thebookwillfirstcoverhowtoobtaindetailedstatsfortheactivecontainers,resourcesconsumed,andcontainerbehavior.Thisbookwillalsoshowthereadershowtousethesestatstoimprovetheoverallperformanceofthesystem.

WhatthisbookcoversChapter1,IntroductiontoDockerMonitoring,discusseshowdifferentitistomonitorcontainerscomparedtomoretraditionalserverssuchasvirtualmachines,baremetalmachines,andcloudinstances(PetsversusCattleandChickensversusSnowflakes).Thischapteralsodetailstheoperatingsystemscoveredintheexampleslaterinthisbookandalsogivesalittleinformationonhowtogetalocaltestenvironmentupandrunningusingvagrant,sothatinstallationinstructionsandpracticalexamplescanbeeasilyfollowed.

Chapter2,UsingtheBuilt-inTools,helpsyoulearnaboutthebasicmetricsyoucangetoutofthevanillaDockerinstallationandhowyoucanusethem.Also,wewillunderstandhowtogetreal-timestatisticsonourrunningcontainers,howtousecommandsthatarefamiliartous,andhowtogetinformationontheprocessesthatarelaunchedaspartofeachcontainer.

Chapter3,AdvancedContainerResourceAnalysis,introducescAdvisorfromGoogle,whichaddsalotmoreprecisiontothebasictoolsprovidedbyDocker.YouwillalsolearnhowtoinstallcAdvisorandstartcollectingmetrics.

Chapter4,ATraditionalApproachtoMonitoringContainers,looksatatraditionaltoolformonitoringservices.Bytheendofthischapter,youshouldknowyourwayaroundZabbixandthevariouswaysyoucanmonitoryourcontainers.

Chapter5,QueryingwithSysdig,describesSysdigas“anopensource,system-levelexplorationtooltocapturesystemstateandactivityfromarunningLinuxinstance,thensave,filter,andanalyzeit.”Inthischapter,youwilllearnhowtouseSysdigtobothviewyourcontainers’performancemetricsinrealtimeandalsorecordsessionstoquerylater.

Chapter6,ExploringThirdPartyOptions,walksyouthroughafewoftheSoftwareasaService(SaaS)optionsthatareavailable,whyyouwouldusethem,andhowtoinstalltheirclientsonthehostserver.

Chapter7,CollectingApplicationLogsfromwithintheContainer,looksathowwecangetthecontentofthelogfilesfortheapplicationsrunningwithinourcontainerstoacentrallocationsothattheyareavailableevenifyouhavetodestroyandreplaceacontainer.

Chapter8,WhatAretheNextSteps?,looksatthenextstepsyoucantakeinmonitoringyourcontainersbytalkingaboutthebenefitsofaddingalertingtoyourmonitoring.Also,wewillcoversomedifferentscenariosandlookatwhichtypeofmonitoringisappropriateforeachofthem.

WhatyouneedforthisbookToensuretheexperienceisasconsistentaspossible,wewillbeinstallingvagrantandVirtualBoxtorunthevirtualmachinethatwillactasahosttorunourcontainers.VagrantisavailableforLinux,OSX,andWindows;fordetailsonhowtoinstallthis,seethevagrantwebsiteathttps://www.vagrantup.com/.ThedetailsofhowtodownloadandinstallVirtualBoxcanbefoundathttps://www.virtualbox.org/;again,VirtualBoxcanbeinstalledonLinux,OSX,andWindows.

https://www.vagrantup.com/

https://www.virtualbox.org/

WhothisbookisforThisbookisforDevOpsengineersandsystemadministratorswhowanttomanageDockercontainers,bettermanagethesecontainersusingexperttechniquesandmethods,andbettermaintainapplicationsbuiltonDocker.

ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.

Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“Wecanincludeothercontextsthroughtheuseoftheincludedirective.”

Ablockofcodeissetasfollows:

{

"fields":{

"@timestamp":[

1444567706641

]

},

"sort":[

1444567706641

]

}

Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

{

"fields":{

"@timestamp":[

1444567706641

]

},

"sort":[

1444567706641

]

}

Anycommand-lineinputoroutputiswrittenasfollows:

cd~/Documents/Projects/monitoring-docker/vagrant-ubuntu

vagrantup

Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:“ClickingtheNextbuttonmovesyoutothenextscreen.”

NoteWarningsorimportantnotesappearinaboxlikethis.

TipTipsandtricksappearlikethis.

ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.

Tosendusgeneralfeedback,simplye-mail<[email protected]>,andmentionthebook’stitleinthesubjectofyourmessage.

Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.


http://www.packtpub.com/authors

CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.


http://www.packtpub.com/support

DownloadingthecolorimagesofthisbookWealsoprovideyouwithaPDFfilethathascolorimagesofthescreenshots/diagramsusedinthisbook.Thecolorimageswillhelpyoubetterunderstandthechangesintheoutput.Youcandownloadthisfilefrom:http://www.packtpub.com/sites/default/files/downloads/Monitoring_Docker_ColorImages.pdf

http://www.packtpub.com/sites/default/files/downloads/Monitoring_Docker_ColorImages.pdf

ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.

Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.

http://www.packtpub.com/submit-errata

https://www.packtpub.com/books/content/support

PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

Pleasecontactusat<[email protected]>withalinktothesuspectedpiratedmaterial.

Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.


QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<[email protected]>,andwewilldoourbesttoaddresstheproblem.


Chapter1.IntroductiontoDockerMonitoringDockerhasbeenarecentbutveryimportantadditiontoaSysAdminstoolbox.

Dockerdescribesitselfasanopenplatformforbuilding,shipping,andrunningdistributedapplications.Thismeansthatdeveloperscanbundletheircodeandpassittotheiroperationsteam.Fromhere,theycandeploysafeintheknowledgethatitwillbedonesoinawaythatintroducesconsistencywiththeenvironmentinwhichthecodeisrunning.

Whenthisprocessisfollowed,itshouldmaketheage-olddevelopersversusoperationsargumentof“itworkedonmylocaldevelopmentserver”—athingofthepast.Sincebeforeits“productionready”1.0releasebackinJune2014,therehadbeenover10,000Dockerizedapplicationsavailable.Bytheendof2014,thatnumberhadrisentoover71,000.YoucanseehowDockergrewin2014bylookingattheinfographicthatwaspublishedbyDockerinearly2015,whichcanbefoundathttps://blog.docker.com/2015/01/docker-project-2014-a-whirlwind-year-in-review/.

Whilethedebateisstillragingabouthowproductionreadythetechnologyis,Dockerhasgainedanimpressivelistoftechnologypartners,includingRedHat,Canonical,HP,andevenMicrosoft.

CompaniessuchasGoogle,Spotify,Soundcloud,andCenturyLink,haveallopensourcedtoolsthatsupportDockerinsomeway,shape,orformandtherehasalsobeennumerousindependentdeveloperswhohavereleasedappsthatprovideadditionalfunctionalitytothecoreDockerproductset.Also,allthecompanieshavesprunguparoundtheDockerecosystem.

Thisbookassumesthatyouhavehadsomelevelofexperiencebuilding,running,andmanagingDockercontainers,andthatyouwouldnowliketostarttometricsfromyourrunningapplicationstofurthertunethem,orthatyouwouldliketoknowwhenaproblemoccurswithacontainersothatyoucandebuganyongoingissues.

IfyouhaveneverusedDockerbefore,youmaywanttotryoneoftheexcellentbooksthatserveandintroduceyoutoallthethingsthatDockerprovides,bookssuchasLearningDocker,PacktPublishing,orDocker’sownintroductiontocontainers,whichcanbefoundattheirdocumentationpages,asfollows:

LearningDocker:https://www.packtpub.com/virtualization-and-cloud/learning-dockerOfficialDockerdocs:https://docs.docker.com/

Now,wehaveabroughtourselvesuptospeedwithwhatDockeris;therestofthischapterwillcoverthefollowingtopics:

Howdifferentisittomonitorcontainersversusmoretraditionalserverssuchasvirtualmachines,baremetalmachine,andcloudinstances(Pets,Cattle,Chickens,andSnowflakes).

https://blog.docker.com/2015/01/docker-project-2014-a-whirlwind-year-in-review/

https://www.packtpub.com/virtualization-and-cloud/learning-docker

https://docs.docker.com/

WhataretheminimumversionsofDockeryoushouldberunning?HowtofollowinstructionsonbringingupanenvironmentlocallyusingVagrantinordertofollowthepracticalexercisesinthisbook

Pets,Cattle,Chickens,andSnowflakesBeforewestartdiscussingthevariouswaysinwhichyoucanmonitoryourcontainers,weshouldgetanunderstandingofwhataSysAdminsworldlookslikethesedaysandalsowherecontainersfitintoit.

AtypicalSysAdminwillprobablybelookingafteranestateofserversthatarehostedineitheranon-siteorthird-partydatacenter,somemayevenmanageinstanceshostedinapubliccloudsuchasAmazonWebServicesorMicrosoftAzure,andsomeSysAdminsmayjugglealltheirserverestatesacrossmultiplehostingenvironments.

Eachofthesedifferentenvironmentshasitsownwayofdoingthings,aswellasperformingbestpractices.BackinFebruary2012,RandyBiasgaveatalkatCloudscalingthatdiscussedarchitecturesforopenandscalableclouds.Towardstheendoftheslidedeck,RandyintroducedtheconceptofPetsversusCattle(whichheattributestoBillBaker,whowasthenanengineeratMicrosoft).

Youcanviewtheoriginalslidedeckathttp://www.slideshare.net/randybias/architectures-for-open-and-scalable-clouds.

PetsversusCattleisnowwidelyacceptedasagoodanalogytodescribemodernhostingpractices.

http://www.slideshare.net/randybias/architectures-for-open-and-scalable-clouds

PetsPetsareakintotraditionalphysicalserversorvirtualmachines,asfollows:

Eachpethasaname;forexample,myserver.domain.com.Whenthey’renotwell,youtakethemtothevettohelpthemgetbetter.YouemploySysAdminstolookafterthem.Youpaycloseattentiontothem,sometimesforyears.Youtakebackups,patchthem,andensurethattheyarefullydocumented.

CattleCattle,ontheotherhand,representmoremoderncloudcomputinginstances,asfollows:

You’vegottoomanytoname,soyougivethemnumbers;forexample,theURLcouldlooksomethinglikeip123123123123.eu.public-cloud.com.Whentheygetsick,youshootthemandifyourherdrequiresit,youreplaceanythingyou’vekilled:Aservercrashesorshowssignsthatitishavingproblems,youterminateitandyourconfigurationautomaticallyreplacesitwithanexactreplica.Youputtheminafieldandwatchthemfromfarandyoudon’texpectthemtolivelong.Ratherthanmonitoringtheindividualinstances,youmonitorthecluster.Whenmoreresourcesareneeded,youaddmoreinstancesandoncetheresourceisnolongerrequired,youterminatetheinstancestogetyoubacktoyourbaseconfiguration.

ChickensNextupisatermthatisagoodwayofdescribinghowcontainersfitintothePetsversusCattleworld;inablogposttitle“CloudComputing:Pets,Cattleand…Chickens?”onActiveState,BernardGoldendescribescontainersasChickens:

They’remoreefficientthancattlewhenitcomestoresourceuse.Acontainercanbootinsecondswhereainstanceorservercantakeminutes;italsouseslessCPUpowerthanatypicalvirtualmachineorcloudinstance.Therearemanymorechickensthancattle.Youcanquitedenselypackcontainersontoyourinstancesorservers.Chickenstendtohaveashorterlifespanthancattleandpets.Containerslendthemselvestorunningmicros-services;thesecontainersmayonlybeactiveforafewminutes.

Theoriginalblogpostcanbefoundathttp://www.activestate.com/blog/2015/02/cloud-computing-pets-cattle-and-chickens.

http://www.activestate.com/blog/2015/02/cloud-computing-pets-cattle-and-chickens

SnowflakesThefinaltermisnotanimal-relatedanditdescribesatypeofserverthatyoudefiantlydon’twanttohaveinyourserverestate,aSnowflake.ThistermwaspennedbyMartinFowlerinablogposttitled“SnowflakeServer”.Snowflakesisatermappliedto“legacy”or“inherited”servers:

Snowflakesaredelicateandaretreatedwithkidgloves.Typically,theserverhasbeeninthedatacentersinceyoustarted.Nooneknowswhooriginallyconfigureditandthereisnodocumentationofit;allyouknowisthatitisimportant.Eachoneisuniqueandisimpossibletoexactlyreproduce.EventhemosthardenedSysAdminfearstorebootthemachineincaseitdoesn’tbootafterwards,asitisrunningend-of-lifesoftwarethatcannoteasilybereinstalled.

Martin’spostcanbefoundathttp://martinfowler.com/bliki/SnowflakeServer.html.

http://martinfowler.com/bliki/SnowflakeServer.html

Sowhatdoesthisallmean?Dependingonyourrequirementsandtheapplicationyouwanttodeploy,yourcontainerscanbelaunchedontoeitherpetorcattlestyleservers.Youcanalsocreateaclutchofchickensandhaveyourcontainersrunmicro-services.

Also,intheory,youcanreplaceyourfearedsnowflakeserverswithacontainer-basedapplicationthatmeetsalltheend-of-lifesoftwarerequirementswhileremainingdeployableonamodernsupportableplatform.

Eachofthedifferentstylesofserverhasdifferentmonitoringrequirements,inthefinalchapterwewilllookatPets,Cattle,Chickens,andSnowflakesagainanddiscussthetoolswehavecoveredinthecomingchapters.Wewillalsocoverbestpracticesyoushouldtakeintoconsiderationwhenplanningyourmonitoring.

DockerWhileDockerhititsversion1.0milestoneoverayearago,itisstillinit’sinfancy;witheachnewreleasecomesnewfeatures,bugfixes,andevensupportforsomeearlyfunctionalitythatisbeingdepreciated.

Dockeritselfisnowacollectionofseveralsmallerprojects;theseincludethefollowing:

DockerEngineDockerMachineDockerComposeDockerSwarmDockerHubDockerRegistryKitmatic

Inthisbook,wewillbeusingDockerEngine,DockerCompose,andtheDockerHub.

DockerEngineisthecorecomponentoftheDockerprojectanditprovidesthemainbulkoftheDockerfunctionality.WheneverDockerorthedockercommandismentionedinthisbook,IwillbereferringtoDockerEngine.

ThebookassumesyouhaveDockerEngineversion1.71orlaterinstalled;olderversionsofDockerEnginemaynotcontainthenecessaryfunctionalityrequiredtorunthecommandsandsoftwarecoveredintheupcomingchapters.

DockerComposestarteditslifeasathird-partyorchestrationtoolcalledFigbeforebeingpurchasedbyDockerin2014.Itisdescribedasawayofdefiningamulti-containerapplicationusingYAML(http://yaml.org).Simplyput,thismeansthatyouquicklydeploycomplexapplicationsusingasinglecommandthatcallsahumanreadableconfigurationfile.

WeassumethatyouhaveDockerCompose1.3.3orlaterinstalled;thedocker-compose.ymlfilesmentionedinthisbookhavebeenwrittenwiththisversioninmind.

Finally,themajorityoftheimageswewillbedeployingduringthisbookwillbesourcedfromtheDockerHub(https://hub.docker.com/),whichnotonlyhousesapublicregistrycontainingover40,000publicimagesbutalso100officialimages.ThefollowingscreenshotshowstheofficialrepositorieslistingontheDockerHubwebsite:

http://yaml.org

https://hub.docker.com/

YoucanalsosignupandusetheDockerHubtohostyourownpublicandprivateimages.

LaunchingalocalenvironmentWhereverpossible,Iwilltrytoensurethatthepracticalexercisesinthisbookwillbeabletoberunonalocalmachinesuchasyourdesktoporlaptop.Forthepurposesofthisbook,IwillassumethatyourlocalmachineisrunningeitherarecentversionOSXoranup-to-dateLinuxdistributionandhasahighenoughspecificationtorunthesoftwarementionedinthischapter.

ThetwotoolswewillbeusingtolaunchourDockerinstanceswillalsorunonWindows;therefore,itshouldbepossibletofollowtheinstructionswithinthis,althoughyoumayhavetorefertheusageguidesforanychangestothesyntax.

DuetothewayinwhichDockerisarchitected,alotofthecontentofthisbookwillhaveyourunningcommandsandinteractingwiththecommandlineonthevirtualserverthatisactingasthehostmachine,ratherthanthecontainersthemselves.Becauseofthis,wewillnotbeusingeitherDockerMachineorKitematic.

BothofthesearetoolsprovidedbyDockertoquicklybootstrapaDocker-enabledvirtualserveronyourlocalmachine,asunfortunatelythehostmachinesdeployedbythesetoolscontainastrippeddownoperatingsystemthatisoptimizedforrunningDockerwiththesmallestfootprintaspossible.

Aswewillbeinstallingadditionalpackagesonthehostmachines,astrippeddown“Dockeronly”operatingsystemmaynothavethecomponentsavailabletomeettheprerequisitesofthesoftwarethatwewillberunninginthelaterchapters;therefore,toensurethattherearenoproblemsfurtheron,weberunningafulloperatingsystem.

Personally,IpreferaRPM-basedoperatingsystemsuchasRedHatEnterpriseLinux,Fedora,orCentOS,asIhavebeenusingthemprettymuchsincethedayIfirstloggedintoaLinuxserver.

However,asalotofreaderswillbefamiliarwiththeDebian-basedUbuntu,Iwillbeprovidingpracticalexamplesforbothoperatingsystems.

Toensuretheexperienceisasconsistentaspossible,wewillbeinstallingVagrantandVirtualBoxtorunthevirtualmachinethatwillactasahosttorunourcontainers.

Vagrant,writtenbyMitchellHashimoto,isacommandlinetoolforcreatingandconfiguringreproducibleandportablevirtualmachineenvironments.TherehavebeennumerousblogpostsandarticlesthatactuallypitchDockeragainstVagrant;however,inourcase,thetwotechnologiesworkquitewelltogetherinprovidingarepeatableandconsistentenvironment.

VagrantisavailableforLinux,OSX,andWindows.Fordetailsonhowtoinstall,gototheVagrantwebsiteathttps://www.vagrantup.com/.

VirtualBoxisagreatallroundopensourcevirtualizationplatformoriginallydevelopedbySunandnowmaintainedbyOracle.Itallowsyoutorunboth32-bitand64-bitguestoperatingsystemsonyourlocalmachine.DetailsonhowtodownloadandinstallVirtualBoxcanbefoundathttps://www.virtualbox.org/;again,VirtualBoxcanbeinstalled

https://www.vagrantup.com/

https://www.virtualbox.org/

onLinux,OSX,andWindows.

CloningtheenvironmentThesourcefortheenvironmentalongwiththepracticalexamplescanbefoundonGitHubintheMonitoringDockerrepositoryathttps://github.com/russmckendrick/monitoring-docker.

Toclonetherepositoryonaterminalonyourlocalmachine,runthefollowingcommands(replacingthefilepathasneeded):

mkdir~/Documents/Projects

cd~/Documents/Projects/

gitclonehttps://github.com/russmckendrick/monitoring-docker.git

Oncecloned,youshouldseeadirectorycalledmonitoring-dockerandthenenterthatdirectory,asfollows:

cd~/Documents/Projects/monitoring-docker

https://github.com/russmckendrick/monitoring-docker

RunningavirtualserverIntherepository,youwillfindtwofolderscontainingthenecessaryVagrantfiletolauncheitheraCentOS7oraUbuntu14.04virtualserver.

IfyouwouldliketousetheCentOS7vagrantbox,changethedirectorytovagrant-centos:

cdvagrant-centos

Onceyouareinthevagrant-centosdirectory,youwillseethatthereisaVagrantfile;thisfileisallyouneedtolaunchaCentOS7virtualserver.Afterthevirtualserverhasbeenbooted,thelatestversionofdockeranddocker-composewillbeinstalledandthemonitoring-dockerdirectorywillalsobemountedinsidethevirtualmachineusingthemountpoint/monitoring-docker.

Tolaunchthevirtualserver,simplytypethefollowingcommand:

vagrantup

Thiswilldownloadthelatestversionofthevagrantboxfromhttps://atlas.hashicorp.com/russmckendrick/boxes/centos71andthenbootthevirtualserver;it’sa450MBdownloadsoitmaytakeseveralminutestodownload;itonlyhastodothisonce.

Ifallgoeswell,youshouldseesomethingsimilartothefollowingoutput:

https://atlas.hashicorp.com/russmckendrick/boxes/centos71

Nowthatyouhavebootedthevirtualserver,youcanconnecttoitusingthefollowingcommand:

vagrantssh

Onceloggedin,youshouldverifythatdockeranddocker-composearebothavailable:

Finally,youcantryrunningthehello-worldcontainerusingthefollowingcommand:

dockerrunhello-world

Ifeverythinggoesasexpected,youshouldseethefollowingoutput:

Totrysomethingmoreambitious,youcanrunanUbuntucontainerwiththefollowingcommand:

dockerrun-itubuntubash

BeforewelaunchandentertheUbuntucontainer,letsconfirmthatwearerunningtheCentOShostmachinebycheckingthereleasefilethatcanbefoundin/etc:

Now,wecanlaunchtheUbuntucontainer.Usingthesamecommand,wecanconfirmthatweareinsidetheUbuntucontainerbyviewingitsreleasefile:

Toexitthecontainerjusttypeinexit.Thiswillstopthecontainerfromrunning,asithasterminatedtheonlyrunningprocesswithinthecontainer,whichwasbash,andreturnedyoutothehostCentOSmachine.

AsyoucanseeherefromourCentOS7host,wehavelaunchedandremovedanUbuntucontainer.

BoththeCentOS7andUbuntuVagrantfileswillconfigureastaticIPaddressonyourvirtualmachine.Itis192.168.33.10;also,thereisaDNSrecordforthisIPaddress

availableatdocker.media-glass.es.Thesewillallowyoutoaccessanycontainersthatexposethemselvestoabrowserateitherhttp://192.168.33.10/orhttp://docker.media-glass.es/.

TipTheURLhttp://docker.media-glass.es/willonlyworkwhilethevagrantboxisup,andyouhaveacontainerrunningwhichservesWebpages.

Youcanseethisinactionbyrunningthefollowingcommand:

dockerrun-d-p80:80russmckendrick/nginx-php

TipDownloadingtheexamplecode

Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

ThiswilldownloadandlaunchacontainerrunningNGINX.Youcanthengotohttp://192.168.33.10/orhttp://docker.media-glass.es/inyourbrowser;youshouldseeaforbiddenpage.ThisisbecausewehavenotyetgivenNGINXanycontenttoserve(moreonthiswillbecoveredlaterinthebook):

Formoreexamplesandideas,gotothewebsiteathttp://docs.docker.com/userguide/.

http://docker.media-glass.es

http://docker.media-glass.es/



http://www.packtpub.com/support


http://docs.docker.com/userguide/

HaltingthevirtualserverTologoutofthevirtualserverandreturntoyourlocalmachine,youtypeexit.

Youshouldnowseeyourlocalmachine’sterminalprompt;however,thevirtualserveryoubootedwillstillberunninginthebackgroundhappily,usingresources,untilyoueitherpoweritdownusingthefollowingcommand:

vagranthalt

Terminatethevirtualserveraltogetherusingvagrantdestroy:

vagrantdestroy

Tocheckthecurrentstatusofthevirtualserver,youcanrunthefollowingcommand:

vagrantstatus

Theresultoftheprecedingcommandisgiveninthefollowingoutput:

Eitherpoweringthevirtualserverbackonorcreatingitfromscratchagain,canbeachievedbyissuingthevagrantupcommandagain.

TheprecedingdetailsshowhowtousetheCentOS7vagrantbox.IfyouwouldprefertolaunchanUbuntu14.04virtualserver,youcandownloadandinstallthevagrantboxbygoingintothevagrant-ubuntudirectoryusingthefollowingcommand:

cd~/Documents/Projects/monitoring-docker/vagrant-ubuntu

vagrantup

Fromhere,youwillbeablerunvagrantupandfollowthesameinstructionsusedtobootandinteractwiththeCentOS7virtualserver.

SummaryInthischapter,wetalkedaboutdifferenttypesofserverandalsodiscussedhowyourcontainerizedapplicationscanfitintoeachofthecategories.WehavealsoinstalledVirtualBoxandusedVagranttolauncheitheraCentOS7orUbuntu14.04virtualserver,withdockeranddocker-composeinstalled.

Ournewvirtualserverenvironmentwillbeusedthroughouttheupcomingchapterstotestthevariousdifferenttypesofmonitoring.Inthenextchapter,wewillstartourjourneybyusingDocker’sin-builtfunctionalitytoexploremetricsaboutourrunningcontainers.

Chapter2.UsingtheBuilt-inToolsInthelaterchaptersofthisbook,wewillexplorethemonitoringpartsofthelargeeco-systemthathasstartedtoflourisharoundDockeroverthelast24months.However,beforewepressaheadwiththat,weshouldtakealookatwhatispossiblewithavanillainstallationofDocker.Inthischapter,wewillcoverthefollowingtopics:

UsingthetoolsbuiltintoDockertogetreal-timemetricsoncontainerperformanceUsingstandardoperatingsystemcommandstogetmetricsonwhatDockerisdoingGeneratingatestloadsoyoucanviewthemetricschanging

DockerstatsSinceversion1.5,therehasbeenabasicstatisticcommandbuiltintoDocker:

dockerstats--help

Usage:dockerstats[OPTIONS]CONTAINER[CONTAINER…]

Displayalivestreamofoneormorecontainers'resourceusagestatistics

--help=falsePrintusage

--no-stream=falseDisablestreamingstatsandonlypullthefirst

result

Thiscommandwillstreamdetailsoftheresourceutilizationofyourcontainersinrealtime.Thebestwaytofindoutaboutthecommandistoseeitinaction.

RunningDockerstatsLet’slaunchacontainerusingthevagrantenvironment,whichwecoveredinthelastchapter:

[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-centos/

[russ@mac~]$vagrantup

Bringingmachine'default'upwith'virtualbox'provider…

==>default:Importingbasebox'russmckendrick/centos71'...

==>default:MatchingMACaddressforNATnetworking…

==>default:Checkingifbox'russmckendrick/centos71'isuptodate…

.....

==>default:=>Installingdocker-engine…

==>default:=>Configuringvagrantuser…

==>default:=>Startingdocker-engine…

==>default:=>Installingdocker-compose…

==>default:=>FinishedinstallationofDocker

[russ@mac~]$vagrantssh

Nowthatyouareconnectedtothevagrantserver,launchthecontainerusingtheDockercomposefilein/monitoring_docker/Chapter01/01-basic/:

[vagrant@centos7~]$cd/monitoring_docker/Chapter01/01-basic/

[vagrant@centos701-basic]$docker-composeup-d

Creating01basic_web_1…

Youhavenowpulleddownandlaunchedacontainerinthebackground.Thecontaineriscalled01basic_web_1anditrunsNGINXandPHPservingasinglePHPinformationpage(http://php.net/manual/en/function.phpinfo.php).

Tocheckwhethereverythinghasbeenlaunchedasexpected,rundocker-composeps.YoushouldseeyoursinglecontainerwithStateofUp:

[vagrant@centos701-basic]$docker-composeps

NameCommandStatePorts

---------------------------------------------------------------

01basic_web_1/usr/local/bin/runUp0.0.0.0:80->80/tcp

Finally,youshouldbeabletoseethepagecontainingtheoutputofthePHPinformationathttp://192.168.33.10/(thisIPaddressishardcodedintothevagrantconfiguration),ifyouputitinyourlocalbrowser:

http://php.net/manual/en/function.phpinfo.php

Now,youhaveacontainerupandrunning;let’slookatsomeofthebasicstats.Weknowfromtheoutputofdocker-composethatourcontaineriscalled01basic_web_1,soenterthefollowingcommandtostartstreamingstatisticsinyourterminal:

dockerstats01basic_web_1

Itwilltakeasecondtoinitiate;afterthisisdone,youshouldseeyourcontainerlistedalongwiththestatisticsforthefollowing:

CPU%:ThisshowsyouhowmuchoftheavailableCPUresourcethecontaineriscurrentlyusing.MEMUSEAGE/LIMIT:ThistellsyouhowmuchRAMthecontainerisutilizing;italsodisplayshowmuchallowancethecontainerhas.Ifyouhaven’texplicitlysetalimit,itwillshowthetotalamountofRAMonthehostmachine.MEM%:ThisshowsyouwhatpercentageoftheRAMallowancethecontainerisusing.

NETI/O:Thisgivesarunningtotalofhowmuchbandwidthhasbeentransferredinandoutofthecontainer.

Ifyougobacktoyourbrowserwindowandstarttorefreshhttp://192.168.33.10/,youwillseethatthevaluesineachofthecolumnsstarttochange.Tostopstreamingthestatistics,pressCtrl+c.

Ratherthankeepingonhittingrefreshoverandoveragain,let’sgeneratealotoftrafficto01basic_web_1,whichshouldputthecontainerunderaheavyload.

Here,wewilllaunchacontainerthatwillsend10,000requeststo01basic_web_1usingApacheBench(https://httpd.apache.org/docs/2.2/programs/ab.html).Althoughitwilltakeaminuteortwotoexecute,weshouldrundockerstatsassoonaspossible:

dockerrun-d--name=01basic_load--link=01basic_web_1russmckendrick/abab

-k-n10000-c5http://01basic_web_1/&&dockerstats01basic_web_1

01basic_load

AftertheApacheBenchimagehasbeendownloadedandthecontainerthatwillbecalled01basic_loadstarts,youshouldseethestatisticsforboth01basic_web_1and01basic_loadbegintostreaminyourterminal:

CONTAINERCPU%MEMUSAGE/LIMITMEM%NETI/O

01basic_load18.11%12.71MB/1.905GB0.67%335.2MB/5.27MB

01basic_web_1139.62%96.49MB/1.905GB5.07%5.27MB/335.2MB

Afterawhile,youwillnoticethatmostofthestatisticsfor01basic_loadwilldropofftozero;thismeansthatthetesthascompletedandthatthecontainerrunningthetesthasexited.Thedockerstatscommandcanonlystreamstatisticsfortherunningcontainers;onesthathaveexitedarenolongerrunningand,therefore,donotproduceoutputwhenrunningdockerstats.

ExitfromdockerstatsusingCtrl+c;toseetheresultsoftheApacheBenchcommand,youcantypedockerlogs01basic_load;youshouldseesomethinglikethefollowingscreenshot:

https://httpd.apache.org/docs/2.2/programs/ab.html

Youshouldn’tworryifyouseeanyfailureslikeintheprecedingoutput.ThisexercisewaspurelytodemonstratehowtoviewthestatisticsoftherunningcontainersandnottotuneawebservertohandletheamountoftrafficwesenttoitusingApacheBench.

Toremovethecontainersthatwelaunched,runthefollowingcommands:

[vagrant@centos701-basic]$docker-composestop

Stopping01basic_web_1…

[vagrant@centos701-basic]$docker-composerm

Goingtoremove01basic_web_1

Areyousure?[yN]y

Removing01basic_web_1…

[vagrant@centos701-basic]$dockerrm01basic_load

01basic_load

Tocheckwhethereverythinghasbeenremovedsuccessfully,rundockerps-aandyoushouldnotbeabletoseeanyrunningorexitedcontainersthathave01basic_intheirnames.

Whatjusthappened?WhilerunningtheApacheBenchtest,youmayhavenoticedthattheCPUutilizationonthecontainerrunningNGINXandPHPwashigh;intheexampleintheprevioussection,itwasusing139.62percentoftheavailableCPUresource.

Aswedidnotattachanyresourcelimitstothecontainerswelaunched,itwaseasyforourtesttousealloftheavailableresourcesonthehostVirtualMachine(VM).IfthisVMwasbeingusedbyseveralusers,allrunningtheirowncontainers,theymayhavestartedtonoticethattheirapplicationshadstartedtoslowdownor,evenworse,theapplicationshadstartedshowingerrors.

Ifyoueverfindyourselfinthissituation,youcanusedockerstatstohelptrackdowntheculprit.

Runningdockerstats$(dockerps-q)willstreamthestatisticsforallthecurrentlyrunningcontainers:

CONTAINERCPU%MEMUSAGE/LIMITMEM%NETI/O

361040b7b33e0.07%86.98MB/1.905GB4.57%2.514kB/738B

56b459ae9092120.06%87.05MB/1.905GB4.57%2.772kB/738B

a3de616f84ba0.04%87.03MB/1.905GB4.57%2.244kB/828B

abdbee7b52070.08%86.61MB/1.905GB4.55%3.69kB/738B

b85c49cf740c0.07%86.15MB/1.905GB4.52%2.952kB/738B

Asyoumayhavenoticed,thisdisplaysthecontainerIDratherthanthename;thisinformationshould,however,beenoughtospottheresourcehogsothatyoucanquicklystopit:

[vagrant@centos701-basic]$dockerstop56b459ae9092

56b459ae9092

Oncestopped,youcanthengetthenameoftheroguecontainerbyrunningthefollowingcommand:

[vagrant@centos701-basic]$dockerps-a|grep56b459ae9092

56b459ae9092russmckendrick/nginx-php"/usr/local/bin/run"9

minutesagoExited(0)26secondsagomy_bad_container

Alternatively,formoredetailedinformation,youcanrundockerinspect56b459ae9092,whichwillgiveyoualltheinformationyouneedonthecontainer.

Whataboutprocesses?OneofthegreatthingsaboutDockeristhatitisn’treallyvirtualization;asmentionedinthepreviouschapter,itisagreatwayofisolatingprocessesratherthanrunninganentireoperatingsystem.

Thiscangetconfusingwhenrunningtoolssuchastoporps.Togetanideajusthowconfusingthiscanget,letslaunchseveralcontainersusingdocker-composeandseeforourselves:

[vagrant@centos7~]$cd/monitoring_docker/Chapter01/02-multiple

[vagrant@centos702-multiple]$docker-composeup-d

Creating02multiple_web_1…

[vagrant@centos702-multiple]$docker-composescaleweb=5





Starting02multiple_web_2…




Now,wehavefivewebserversthathaveallbeenlaunchedfromthesameimageusingthesameconfiguration.OneofthefirstthingsIdowhenloggingintoaservertotroubleshootaproblemisrunps-aux;thiswillshowalltherunningprocesses.Asyoucansee,whenrunningthecommand,therearealotprocesseslisted.

EvenjusttryingtolookattheprocessesforNGINXisconfusing,asthereisnothingtodifferentiatetheprocessesfromonecontainertoanother,asshowninthefollowingoutput:

So,howcanyouknowwhichcontainerownswhichprocesses?

DockertopThiscommandlistsalltheprocessesthatarerunningwithinacontainer;thinkofitasawayoffilteringtheoutputofthepsauxcommandweranonthehostmachine:

Asdockertopisanimplementationofthestandardpscommand,anyflagsyouwouldnormallypasstopsshouldworkasfollows:

[vagrant@centos702-multiple]$dockertop02multiple_web_3–aux

[vagrant@centos702-multiple]$dockertop02multiple_web_3-faux

DockerexecAnotherwaytoviewwhatisgoingonwithinacontaineristoenterit.Toenableyoutodothis,Dockerintroducedthedockerexeccommand.Thisallowsyoutospawnanadditionalprocesswithinanalreadyrunningcontainerandthenattachtotheprocess;so,ifwewantedtolookatwhatiscurrentlyrunningon02multiple_web_3,weshouldusethefollowingcommandspawnabashshellwithinanalreadyrunningcontainer:

dockerexec-t-i02multiple_web_3bash

Onceyouhaveanactiveshellonthecontainer,youwillnoticethatyourprompthaschangedtothecontainer’sID.Yoursessionisnowisolatedtothecontainer’senvironment,meaningthatyouwillonlybeabletointeractwiththeprocessesbelongingtothecontaineryouentered.

Fromhere,youcanrunthepsauxortopcommandasyouwoulddoonthehostmachine,andonlyseetheprocessesassociatedwiththecontaineryouareinterestedin:

Toleavethecontainer,typeinexit,youshouldseeyourpromptchangebackinyourhostmachine.

Finally,youcanstopandremovethecontainersbyrunningdocker-composestopanddocker-composekill.

SummaryInthischapter,wesawhowwecangetreal-timestatisticsonourrunningcontainersandhowwecanusecommandsthatarefamiliartous,togetinformationontheprocessesthatarelaunchedaspartofeachcontainer.

Onthefaceofit,dockerstatsseemslikeareallybasicpieceoffunctionalitythatisn’treallyanythingmorethanatooltohelpyouidentifywhichcontainerisusingalltheresourceswhileaproblemisoccurring.However,theDockercommandisactuallypullingtheinformationfromaquitepowerfulAPI.

ThisAPIformsthebasisforalotofthemonitoringtoolswewillbelookingatinthenextfewchapters.

Chapter3.AdvancedContainerResourceAnalysisInthelastchapter,welookedathowyoucanusetheAPIbuiltintoDockertogainaninsighttowhatresourcesyourcontainersarerunning.Now,wearetoseehowwecantakeittothenextlevelbyusingcAdvisorfromGoogle.Inthischapter,youwillcoverthefollowingtopics:

HowtoinstallcAdvisorandstartcollectingmetricsLearnallaboutthewebinterfaceandreal-timemonitoringWhatyouroptionsareforshippingmetricstoaremotePrometheusdatabaseforlong-termstorageandtrendanalysis

WhatiscAdvisor?GoogledescribescAdvisorasfollows:

“cAdvisor(ContainerAdvisor)providescontainerusersanunderstandingoftheresourceusageandperformancecharacteristicsoftheirrunningcontainers.Itisarunningdaemonthatcollects,aggregates,processes,andexportsinformationaboutrunningcontainers.Specifically,foreachcontainer,itkeepsresourceisolationparameters,historicalresourceusage,histogramsofcompletehistoricalresourceusage,andnetworkstatistics.Thisdataisexportedbyacontainerandismachine-wide.”

TheprojectstartedofflifeasaninternaltoolatGoogleforgaininganinsightintocontainersthathadbeenlaunchedusingtheirowncontainerstack.

NoteGoogle’sowncontainerstackwascalled“LetMeContainThatForYou”orlmctfyforshort.TheworkonlmctfyhasbeeninstalledasaGoogleportfunctionalityovertolibcontainerthatispartoftheOpenContainerInitiative.Furtherdetailsonlmctfycanbefoundathttps://github.com/google/lmctfy/.

cAdvisoriswritteninGo(https://golang.org);youcaneithercompileyourownbinaryoryoucanusethepre-compiledbinarythataresuppliedviaacontainer,whichisavailablefromGoogle’sownDockerHubaccount.Youcanfindthisathttp://hub.docker.com/u/google/.

Onceinstalled,cAdvisorwillsitinthebackgroundandcapturemetricsthataresimilartothatofthedockerstatscommand.Wewillgothroughthesestatsandunderstandwhattheymeanlaterinthischapter.

cAdvisortakesthesemetricsalongwiththoseforthehostmachineandexposesthemviaasimpleandeasy-to-usebuilt-inwebinterface.

https://github.com/google/lmctfy/

https://golang.org

http://hub.docker.com/u/google/

RunningcAdvisorusingacontainerThereareanumberofwaystoinstallcAdvisor;theeasiestwaytogetstartedistodownloadandrunthecontainerimagethatcontainsacopyofaprecompiledcAdvisorbinary.

BeforerunningcAdvisor,let’slaunchafreshvagranthost:







.....







TipUsingabackslash

Aswehavealotoptionstopasstothedockerruncommand,weareusing\tosplitthecommandovermultiplelinessoit’seasiertofollowwhatisgoingon.

Onceyouhaveaccesstothehostmachine,runthefollowingcommand:

dockerrun\

--detach=true\

--volume=/:/rootfs:ro\

--volume=/var/run:/var/run:rw\

--volume=/sys:/sys:ro\

--volume=/var/lib/docker/:/var/lib/docker:ro\

--publish=8080:8080\

--privileged=true\

--name=cadvisor\

google/cadvisor:latest

YoushouldnowhaveacAdvisorcontainerupandrunningonyourhostmachine.Beforewestart,let’slookatcAdvisorinmoredetailbydiscussingwhywehavepassedalltheoptionstothecontainer.

ThecAdvisorbinaryisdesignedtorunonthehostmachinealongsidetheDockerbinary,sobylaunchingcAdvisorinacontainer,weareactuallyisolatingthebinaryinitsdownenvironment.TogivecAdvisoraccesstotheresourcesitrequiresonthehostmachine,wehavetomountseveralpartitionsandalsogivethecontainerprivilegedaccesstoletthecAdvisorbinarythinkitisbeingexecutedonthehostmachine.

NoteWhenacontainerislaunchedwith--privileged,Dockerwillenablefullaccesstodevicesonthehostmachine;also,DockerwillconfigurebothAppArmororSELinuxtoallowyourcontainerthesameaccesstothehostmachineasaprocessrunningoutsidethecontainerwillhave.Forinformationonthe--privilegedflag,seethispostontheDockerblogathttp://blog.docker.com/2013/09/docker-can-now-run-within-docker/.

http://blog.docker.com/2013/09/docker-can-now-run-within-docker/

CompilingcAdvisorfromsourceAsmentionedintheprevioussection,cAdvisorreallyoughttobeexecutedonthehostmachine;thismeans,youmayhavetouseacasetocompileyourowncAdvisorbinaryandrunitdirectlyonthehost.

TocompilecAdvisor,youwillneedtoperformthefollowingsteps:

1. InstallGoandMercurialonthehostmachine—version1.3orhigherofGoisneededtocompilecAdvisor.

2. SetthepathforGotoworkfrom.3. GrabthesourcecodeforcAdvisorandgodep.4. SetthepathforyourGobinaries.5. BuildthecAdvisorbinaryusinggodeptosourcethedependenciesforus.6. Copythebinaryto/usr/local/bin/.7. DownloadeitheranUpstartorSystemdscriptandlaunchtheprocess.

Ifyoufollowedtheinstructionsintheprevioussection,youwillalreadyhaveacAdvisorprocessrunning.Beforecompilingfromsource,youshouldstartwithacleanhost;let’slogoutofthehostandlaunchafreshcopy:

[vagrant@centos7~]$exit

logout

Connectionto127.0.0.1closed.

[russ@mac~]$vagrantdestroy

default:Areyousureyouwanttodestroythe'default'VM?[y/N]y

==>default:ForcingshutdownofVM…

==>default:DestroyingVMandassociateddrives…

==>default:Runningcleanuptasksfor'shell'provisioner…






.....







TobuildcAdvisorontheCentOS7host,runthefollowingcommand:

sudoyuminstall-ygolanggitmercurial

exportGOPATH=$HOME/go

goget-dgithub.com/google/cadvisor

gogetgithub.com/tools/godep

exportPATH=$PATH:$GOPATH/bin

cd$GOPATH/src/github.com/google/cadvisor

godepgobuild.

sudocpcadvisor/usr/local/bin/

sudowgethttps://gist.githubusercontent.com/russmckendrick/f647b2faad5d92c96

771/raw/86b01a044006f85eebbe395d3857de1185ce4701/cadvisor.service-O

/lib/systemd/system/cadvisor.service

sudosystemctlenablecadvisor.service

sudosystemctlstartcadvisor

OntheUbuntu14.04LTShost,runthefollowingcommand:

sudoapt-get-yinstallsoftware-properties-common

sudoadd-apt-repositoryppa:evarlast/golang1.4

sudoapt-getupdate

sudoapt-get-yinstallgolangmercurial

exportGOPATH=$HOME/go

goget-dgithub.com/google/cadvisor

gogetgithub.com/tools/godep

exportPATH=$PATH:$GOPATH/bin

cd$GOPATH/src/github.com/google/cadvisor

godepgobuild.

sudocpcadvisor/usr/local/bin/

sudowgethttps://gist.githubusercontent.com/russmckendrick/f647b2faad5d92c96

771/raw/e12c100d220d30c1637bedd0ce1c18fb84beff77/cadvisor.conf-O

/etc/init/cadvisor.conf

sudostartcadvisor

YoushouldnowhavearunningcAdvisorprocess.Youcancheckthisbyrunningpsaux|grepcadvisorandyoushouldseeaprocesswithapathof/usr/local/bin/cadvisorrunning.

CollectingmetricsNow,youhavecAdvisorrunning;whatdoyouneedtodotoconfiguretheserviceinordertostartcollectingmetrics?Theshortansweris,nothingatall.WhenyoustartedthecAdvisorprocess,itinstantlystartedpollingyourhostmachinetofindoutwhatcontainersarerunningandgatheredinformationonboththerunningcontainersandyourhostmachine.

TheWebinterfacecAdvisorshouldberunningonthe8080port;ifyouopenhttp://192.168.33.10:8080/,youshouldbegreetedwiththecAdvisorlogoandanoverviewofyourhostmachine:

Thisinitialpagestreamslivestatsaboutthehostmachine,thougheachsectionisrepeatedwhenyoustarttodrilldownandviewthecontainers.Tostartwith,let’slookateachsectionusingthehostinformation.

OverviewThisoverviewsectiongivesyouabird’s-eyeviewofyoursystem;itusesgaugessoyoucanquicklygetanideaofwhichresourcesarereachingtheirlimits.Inthefollowingscreenshot,thereisverylittleinthewayofCPUutilizationandthefilesystemusageisrelativelylow;however,weareusing64%oftheavailableRAM:

ProcessesThefollowingscreenshotdisplaysacombinedviewoftheoutputofthepsaux,dockerpsandtopcommandsweusedinthepreviouschapter:

Hereiswhateachcolumnheadingmeans:

User:ThisshowswhichuserisrunningtheprocessPID:ThisistheuniqueprocessIDPPID:ThisisthePIDoftheparentprocessStartTime:ThisshowswhattimetheprocessstartedCPU%:ThisisthepercentageoftheCPUtheprocessiscurrentlyconsumingMEM%:ThisisthepercentageoftheRAMtheprocessiscurrentlyconsumingRSS:ThisshowshowmuchofthemainmemorytheprocessisusingVirtualSize:ThisshowshowmuchofthevirtualmemorytheprocessisusingStatus:Thisshowsthecurrentstatusoftheprocess;thisarethestandardLinuxprocessstatecodesRunningTime:ThisshowshowlongtheprocesshasbeenrunningCommand:ThisshowswhichcommandtheprocessisrunningContainer:Thisshowswhichcontainertheprocessisattachedto;thecontainerlistedas/isthehostmachine

Astherecouldbeseveralhundredprocessesactive,thissectionissplitintopages;youcannavigatetothesewiththebuttonsonthebottom-left.Also,youcansorttheprocessesbyclickingonanyoftheheadings.

CPUThefollowinggraphshowstheCPUutilizationoverthelastminute:

Hereiswhateachtermmeans:

TotalUsage:ThisshowsanaggregateusageacrossallcoresUsageperCore:ThisgraphbreaksdowntheusagepercoreUsageBreakdown(notshownintheprecedingscreenshot):Thisshowsaggregateusageacrossallcores,butbreaksitdowntowhatisbeingusedbythekernelandwhatisbeingusedbytheuser-ownedprocesses

MemoryTheMemorysectionissplitintotwoparts.Thegraphtellsyouthetotalamountofmemoryusedbyalltheprocessesforthehostorcontainer;thisisthetotalofthehotandcoldmemory.TheHotmemoryisthecurrentworkingset:pagesthathavebeentouchedbythekernelrecently.TheColdmemoryisthepagethathasn’tbeentouchedforawhileandcouldbereclaimedifneeded.

TheUsageBreakdowngivesavisualrepresentationofthetotalmemoryinthehostmachine,orallowanceinthecontainer,alongsidethetotalandhotusage:

NetworkThissectionshowstheincomingandoutgoingtrafficoverthelastminute.Youcanchangetheinterfaceusingthedrop-downboxonthetop-left.Thereisalsoagraphthatshowsanynetworkingerrors.Typically,thisgraphshouldbeflat.Ifitisn’t,thenyouwillbeseeingperformanceissueswithyourhostmachineorcontainer:

FilesystemThefinalsectiongivesabreakdownofthefilesystemusage.Inthefollowingscreenshot,/dev/sda1isthebootpartition,/dev/sda3isthemainfilesystem,and/dev/mapper/docker-8…isanaggregateofthewritefilesystemsofyourrunningcontainers:

ViewingcontainerstatsAtthetopofthepage,thereisalinkofyourrunningcontainers;youcaneitherclickonthelinkorgodirectlytohttp://192.168.33.10:8080/docker/.Oncethepageloads,youshouldseealistofallyourrunningcontainers,andalsoadetailedoverviewofyourDockerprocess,andfinallyalistoftheimagesyouhavedownloaded.

SubcontainersSubcontainersshowsalistofyourcontainers;eachentryisaclickablelinkthatwilltakeyoutoapagethatwillgiveyouthefollowingdetails:

Isolation:

CPU:ThisshowsyoutheCPUallowancesofthecontainer;ifyouhavenotsetanyresourcelimits,youwillseethehost’sCPUinformationMemory:Thisshowsyouthememoryallowancesofthecontainer;ifyouhavenotsetanyresourcelimits,yourcontainerwillshowanunlimitedallowance

Usage:

Overview:ThisshowsgaugessoyoucanquicklyseehowclosetoanyresourcelimitsyouareProcesses:ThisshowstheprocessesforjustyourselectedcontainerCPU:ThisshowstheCPUutilizationgraphsisolatedtojustyourcontainerMemory:Thisshowsthememoryutilizationofyourcontainer

DriverstatusThedrivergivesthebasicstatsonyourmainDockerprocess,alongwiththeinformationonthehostmachine’skernel,hostname,andalsotheunderlyingoperatingsystem.

Italsogivesinformationonthetotalnumberofcontainersandimages.Youmaynoticethatthetotalnumberofimagesisamuchlargerfigurethanyouexpectedtosee;thisisbecauseitiscountingeachfilesystemasanindividualimage.

NoteFormoredetailsonDockerimages,seetheDockeruserguideathttps://docs.docker.com/userguide/dockerimages/.

Italsogivesyouadetailedbreakdownofyourstorageconfiguration.

https://docs.docker.com/userguide/dockerimages/

ImagesFinally,yougetalistoftheDockerimageswhichareavailableonthehostmachine.ItliststheRepository,Tag,Size,andwhentheimagewascreated,alongwiththeimages’uniqueID.Thisletsyouknowwheretheimageoriginatedfrom(Repository),whichversionoftheimageyouhavedownloaded(Tag)andhowbigtheimageis(Size).

Thisisallgreat,what’sthecatch?Soyouaremaybethinkingtoyourselfthatallofthisinformationavailableinyourbrowserisreallyuseful;beingabletoseereal-timeperformancemetricsinaneasilyreadableformatisareallyplus.

ThebiggestdrawbackofusingthewebinterfaceforcAdvisor,asyoumayhavenoticed,isthatitonlyshowsyouoneminute’sworthofmetrics;youcanquiteliterallyseetheinformationdisappearinginrealtime.

Asapaneofglassgivesareal-timeviewintoyourcontainers,cAdvisorisabrillianttool;ifyouwanttoreviewanymetricsthatareolderthanoneminute,youareoutofluck.

Thatis,unlessyouconfiguresomewheretostoreallofyourdata;thisiswherePrometheuscomesin.

PrometheusSowhat’sPrometheus?Itsdevelopersdescribeitasfollows:

Prometheusisanopen-sourcesystem’smonitoringandalertingtoolkitbuiltatSoundCloud.Sinceitsinceptionin2012,ithasbecomethestandardforinstrumentingnewservicesatSoundCloudandisseeinggrowingexternalusageandcontributions.

OK,butwhatdoesthathavetodowithcAdvisor?Well,Prometheushasquiteapowerfuldatabasebackendthatstoresthedataitimportsasatimeseriesofevents.

Wikipediadescribesatimeseriesasfollows:

“Atimeseriesisasequenceofdatapoints,typicallyconsistingofsuccessivemeasurementsmadeoveratimeinterval.Examplesoftimeseriesareoceantides,countsofsunspots,andthedailyclosingvalueoftheDowJonesIndustrialAverage.Timeseriesareveryfrequentlyplottedvialinecharts.”

https://en.wikipedia.org/wiki/Time_series

OneofthethingscAdvisordoes,bydefault,isexposeallthemetricsitiscapturingonasinglepageat/metrics;youcanseethisathttp://192.168.33.10:8080/metricsonourcAdvisorinstallation.Themetricsareupdatedeachtimethepageisloaded:

Asyoucanseeintheprecedingscreenshot,thisisjustasinglelongpageofrawtext.ThewayPrometheusworksisthatyouconfigureittoscrapethe/metricsURLatauser-definedinterval,let’ssayeveryfiveseconds;thetextisinaformatthatPrometheusunderstandsanditisingestedintothePrometheus’stimeseriesdatabase.

Whatthismeansisthat,usingPrometheus’spowerfulbuilt-inquerylanguage,youcanstarttodrilldownintoyourdata.Let’slookatgettingPrometheusupandrunning.

https://en.wikipedia.org/wiki/Time_series

LaunchingPrometheusLikecAdvisorthereareseveralwaysyoucanlaunchPrometheus.Tostartwith,wewilllaunchacontainerandinjectourownconfigurationfilesothatPrometheusknowswhereourcAdvisorendpointis:

dockerrun\

--detach=true\

--

volume=/monitoring_docker/Chapter03/prometheus.yml:/etc/prometheus/promethe

us.yml\

--publish=9090:9090\

--name=prometheus\

prom/prometheus:latest

Onceyouhavelaunchedthecontainer,PrometheuswillbeaccessibleonthefollowingURL:http://192.168.33.10:9090.WhenyoufirstloadtheURL,youwillbetakentoastatuspage;thisgivessomebasicinformationonthePrometheusinstallation.Theimportantpartofthispageisthelistoftargets.ThisliststheURLthatPrometheuswillbescrappingtocapturemetrics;youshouldseeyourcAdvisorURLlistedwithastateofHEALTHY,asshowninthefollowingscreenshot:

Anotherinformationpagecontainsthefollowing:

Runtimeinformation:ThisdisplayshowlongPrometheushasbeenupandpollingdata,ifyouhaveconfiguredanendpointBuildinformation:ThiscontainsthedetailsoftheversionofPrometheusthatyouhavebeenrunningConfiguration:ThisisacopyoftheconfigurationfileweinjectedintothecontainerwhenitwaslaunchedRules:Thisisacopyofanyrulesweinjected;thesewillbeusedforalertingStartupflags:Thisshowsalltheruntimevariablesandtheirvalues

QueryingPrometheusAsweonlyhaveafewcontainersupandrunningatthemoment,let’slaunchonethatrunsRedissowecanstarttolookatthequerylanguagebuiltintoPrometheus.

WewillusetheofficialRedisimageforthisandasweareonlygoingtousethisasanexamplewewon’tneedtopassitanyuservariables:

dockerrun--namemy-redis-server-dredis

Wenowhaveacontainercalledmy-redis-serverrunning.cAdvisorshouldalreadybeexposingmetricsaboutthecontainertoPrometheus;let’sgoaheadandsee.InthePrometheuswebinterface,gototheGraphlinkinthemenuatthetopofthepage.Here,youwillbepresentedwithatextboxintowhichyoucanenteryourquery.Tostartwith,let’slookattheCPUusageoftheRediscontainer.

Inthebox,enterthefollowing:

container_cpu_usage_seconds_total{job="cadvisor",name="my-redis-server"}

Then,afterclickingonExecute,youshouldhavetworesultsreturned,listedintheConsoletabofthepage.Ifyouremember,cAdvisorrecordstheCPUusageofeachoftheCPUcoresthatthecontainerhasaccessto,whichiswhywehavetwovaluesreturned,onefor“cpu00”andonefor“cpu01”.ClickingontheGraphlinkwillshowyouresultsoveraperiodoftime:

Asyoucanseeintheprecedingscreenshot,wenowhaveaccesstotheusagegraphsforthelast25minutes,whichisabouthowlongagoIlaunchedtheRedisinstancebefore

generatingthegraph.

DashboardAlso,whencreatingoneofthegraphsusingthequerytoolinthemainapplication,youcaninstallaseparateDashboardapplication.ThisrunsinasecondcontainerthatconnectstoyourmainPrometheuscontainerusingtheAPIasadatasource.

BeforewestarttheDashboardcontainer,weshouldinitializeaSQLite3databasetostoreourconfiguration.Toensurethatthedatabaseispersistent,wewillstorethisonthehostmachinein/tmp/prom/file.sqlite3:

dockerrun\

--volume=/tmp/prom:/tmp/prom\

-eDATABASE_URL=sqlite3:/tmp/prom/file.sqlite3\

prom/promdash./bin/rakedb:migrate

Oncewehaveinitializedthedatabase,wecanlaunchtheDashboardapplicationproperly:

dockerrun\

--detach=true\

--volume=/tmp/prom:/tmp/prom\

-eDATABASE_URL=sqlite3:/tmp/prom/file.sqlite3\

--publish=3000:3000\

--name=promdash\

prom/promdash

Theapplicationshouldnowbeaccessibleathttp://192.168.33.10:3000/.Thefirstthingweneedtodoissetupthedatasource.Todothis,clickontheServerslinkatthetopofthescreenandthenclickonNewServer.Here,youwillbeaskedtoprovidethedetailsofyourPrometheusserver.NametheserverandenterthefollowingURL:

Name:cAdvisorURL:http://192.168.33.10:9090ServerType:Prometheus

OnceyouclickonCreateServer,youshouldreceiveamessagesayingServerwassuccessfullycreated.Nextup,youneedtocreateadirectory;thisiswhereyourdashboardswillbestored.

ClickontheDashboardslinkinthetopmenuandthenclickonNewdirectoryandcreateonecalledTestdirectory.Now,youarereadytostartcreatingDashboards.ClickonNewDashboard,callitMyDashboard,placeitinTestdirectory.OnceyouclickonCreateDashboard,youwillbetakentothepreviewscreen.

Fromhere,youcanbuildupdashboardsusingthecontrolinthetopright-handsideofeachsection.Toadddata,yousimplyenterthequeryyouwouldliketoseeinthedashboardsection:

NoteFordetailedinformationonhowtocreateDashboards,seethePROMDASHsectionofthePrometheusdocumentationathttp://prometheus.io/docs/visualization/promdash/.

http://prometheus.io/docs/visualization/promdash/

ThenextstepsAtthemoment,wearerunningPrometheusinasinglecontaineranditsdataisbeingstoredwithinthatsamecontainer.Thismeans,ifforanyreasonthecontaineristerminated,ourdataislost;italsomeansthatwecan’tupgradewithoutloosingoutdata.Togetaroundthisproblem,wecancreateadatavolumecontainer.

NoteAdatavolumecontainerisaspecialtypeofcontainerthatonlyexistsasstorageforothercontainers.Formoredetails,seetheDockeruserguideathttps://docs.docker.com/userguide/dockervolumes/#creating-and-mounting-a-data-volume-container.

Firstofall,let’smakesurewehaveremovedalltherunningPrometheuscontainers:

dockerstopprometheus&&dockerrmPrometheus

Nextup,let’screateadatacontainercalledpromdata:

dockercreate\

--volume=/promdata\

--name=promdata\

prom/prometheus/bin/true

Finally,launchPrometheusagain,thistime,usingthedatacontainer:

dockerrun\

--detach=true\

--volumes-frompromdata\

--

volume=/monitoring_docker/Chapter03/prometheus.yml:/etc/prometheus/promethe

us.yml\

--publish=9090:9090\

--name=prometheus\

prom/prometheus

Thiswillensurethat,ifyouhavetoupgradeorrelaunchyourcontainer,themetricsyouhavebeencapturingaresafeandsound.

WehaveonlytouchedonthebasicsofusingPrometheusinthissectionofthebook;forfurtherinformationontheapplication,Irecommendthefollowinglinksasagoodstartingpoint:

Documentation:http://prometheus.io/docs/introduction/overview/Twitter:https://twitter.com/PrometheusIOProjectpage:https://github.com/prometheus/prometheusGooglegroups:https://groups.google.com/forum/#!forum/prometheus-developers

https://docs.docker.com/userguide/dockervolumes/#creating-and-mounting-a-data-volume-container

http://prometheus.io/docs/introduction/overview/

https://twitter.com/PrometheusIO

https://github.com/prometheus/prometheus

https://groups.google.com/forum/#!forum/prometheus-developers

Alternatives?TherearesomealternativestoPrometheus.OnesuchalternativeisInfluxDBthatdescribesitselfasfollows:

Anopen-sourcedistributedtimeseriesdatabasewithnoexternaldependencies.

However,atthetimeofwriting,cAdvisorisnotcurrentlycompatiblewiththelatestversionofInfluxDB.TherearepatchesinthecodebaseforcAdvisor;however,theseareyettomakeitthroughtotheGoogle-maintainedDockerImage.

FormoredetailsonInfluxDBandit’snewvisualizationcomplainapplicationChronograf,seetheprojectwebsiteathttps://influxdb.com/andformoredetailsonhowtoexportcAdvisorstatisticstoInfluxDB,seethesupportingdocumentationforcAdvisorathttps://github.com/google/cadvisor/tree/master/docs.

https://influxdb.com/

https://github.com/google/cadvisor/tree/master/docs

SummaryInthischapter,welearnedhowtotaketheviewingreal-timestatisticsofourcontainersoffthecommandlineandintothewebbrowser.WeexploredsomedifferentmethodstoinstallGoogle’scAdvisorapplicationandalsohowtouseitswebinterfacetokeepaneyeonourrunningcontainers.WealsolearnedhowtocapturemetricsfromcAdvisorandstorethemusingPrometheus,amoderntimeseriesdatabase.

Thetwomaintechnologieswehavecoveredinthischapterhaveonlybeenpublicallyavailableforlessthantwelvemonths.Inthenextchapter,wewilllookatusingamonitoringtoolthathasbeeninaSysAdminstoolboxforover10years—Zabbix.

Chapter4.ATraditionalApproachtoMonitoringContainersSofar,wehavelookedatonlyafewtechnologiestomonitorourcontainers,sointhischapter,wewillbelookingmoreatatraditionaltoolformonitoringservices.Bytheendofthischapter,youshouldknowyourwayaroundZabbixandthevariouswaysyoucanmonitoryourcontainers.Wewillcoverthefollowingtopicsinthischapter:

HowtorunaZabbixServerusingcontainersHowtolaunchaZabbixServeronavagrantmachineHowtoprepareourhostsystemformonitoringcontainersusingtheZabbixagentHowtofindyourwayaroundtheZabbixwebinterface

ZabbixFirstthingsfirst,whatisZabbixandwhyuseit?

Ihavepersonallybeenusingitsinceversion1.2;theZabbixsitedescribesitasfollows:

“WithZabbix,itispossibletogathervirtuallylimitlesstypesofdatafromthenetwork.High-performancereal-timemonitoringmeansthattensofthousandsofservers,virtualmachines,andnetworkdevicescanbemonitoredsimultaneously.Alongwithstoringthedata,visualizationfeaturesareavailable(overviews,maps,graphs,screens,andsoon),aswellasveryflexiblewaysofanalyzingthedataforthepurposeofalerting.

Zabbixoffersgreatperformancefordatagatheringandcanbescaledtoverylargeenvironments.DistributedmonitoringoptionsareavailablewiththeuseofZabbixproxies.Zabbixcomeswithaweb-basedinterface,secureuserauthentication,andaflexibleuserpermissionschema.Pollingandtrappingissupported,withnativehigh-performanceagentsgatheringdatafromvirtuallyanypopularoperatingsystem;agent-lessmonitoringmethodsareavailableaswell.”

AtthetimeIstartedusingZabbix,theonlyrealviableoptionswereasfollows:

Nagios:https://www.nagios.org/Zabbix:http://www.zabbix.com/Zenoss:http://www.zenoss.org/

Outofthethesethreeoptions,Zabbixseemedtobethemoststraightforwardoneatthetime.ItwasdoingenoughworktomanagetheseveralhundredserversIwasgoingtomonitorwithouthavingtohavetheextraworkoflearningthecomplexitiesofsettingupNagiosorZenoss;afterall,giventhetaskthesoftwarehad,IneededtobeabletotrustthatIhadsetitupcorrectly.

Inthischapter,whileIamgoingtogointosomedetailaboutthesetupandthebasicsofusingZabbix,wewillonlybetouchingonsomeofthefunctionalities,whichcandoalotmorethanjustmonitoryourcontainers.Formoreinformation,Iwouldrecommendthefollowingasagoodstartingpoint:

Zabbixblog:http://blog.zabbix.comZabbix2.4manual:https://www.zabbix.com/documentation/2.4/manualFurtherreading:https://www.packtpub.com/all/?search=zabbix

https://www.nagios.org/

http://www.zabbix.com/

http://www.zenoss.org/

http://blog.zabbix.com

https://www.zabbix.com/documentation/2.4/manual

https://www.packtpub.com/all/?search=zabbix

InstallingZabbixAsyoumayhavenoticedfromthelinksintheprevioussection,therearealotofmovingpartsinZabbix.Itleveragesseveralopensourcetechnologies,andaproduction-readyinstallationneedsalittlemoreplanningthanwecangointointhischapter.BecauseofthiswearegoingtolookattwowaysofinstallingZabbixquicklyrathergointotoomuchdetail.

UsingcontainersAtthetimeofwriting,thereareoverahundredDockerimagesavailableontheDockerHub(https://hub.docker.com)thatmentionsZabbix.Theserangefromfullserverinstallationstojustthevariousparts,suchastheZabbixagentorproxyservices.

Outoftheoneslisted,thereisonethatisrecommendbyZabbixitself.So,wewilllookatthisone;itcanbefoundatthefollowingURLs:

DockerHub:https://hub.docker.com/u/zabbix/Projectpage:https://github.com/zabbix/zabbix-community-docker

TogettheZabbixServercontainerupandrunning,wemustfirstlaunchadatabasecontainer.Let’sstartafreshwithourvagrantinstancebyrunningthefollowingcommand:


[russ@mac~]$vagrantdestroy

default:Areyousureyouwanttodestroythe'default'VM?[y/N]y

==>default:ForcingshutdownofVM…

==>default:DestroyingVMandassociateddrives…

==>default:Runningcleanuptasksfor'shell'provisioner…






.....







Now,wehaveacleanenvironmentandit’stimetolaunchourdatabasecontainer,asfollows:

dockerrun\

--detach=true\

--publish=3306\

--env="MARIADB_USER=zabbix"\

--env="MARIADB_PASS=zabbix_password"\

--name=zabbix-db\

million12/mariadb

Thiswilldownloadthemillion12/mariadbimagefromhttps://hub.docker.com/r/million12/mariadb/andlaunchacontainercalledzabbix-db,runningMariaDB10(https://mariadb.org)withausercalledzabbixwhohasapasswordzabbix_password.WehavealsoopenedtheMariaDBport3306uponthecontainer,butaswewillbeconnectingtoitfromalinkedcontainer,thereisnoneedtoexposethatportonthehostmachine.

https://hub.docker.com

https://hub.docker.com/u/zabbix/

https://github.com/zabbix/zabbix-community-docker

https://hub.docker.com/r/million12/mariadb/

https://mariadb.org

Now,wehavethedatabasecontainerupandrunning,wenowneedtolaunchourZabbixServercontainer:

dockerrun\

--detach=true\

--publish=80:80\

--publish=10051:10051\

--link=zabbix-db:db\

--env="DB_ADDRESS=db"\

--env="DB_USER=zabbix"\

--env="DB_PASS=zabbix_password"\

--name=zabbix\

zabbix/zabbix-server-2.4

Thisdownloadstheimage,whichatthetimeofwritingisover1GBsothisprocesscouldtakeseveralminutesdependingonyourconnection,andlaunchesacontainercalledzabbix.Itmapsthewebserver(port80)andtheZabbixServerprocess(port10051)onthehosttothecontainer,createsalinktoourdatabasecontainer,setsupthealiasdb,andinjectsthedatabasecredentialsasenvironmentvariablessothatthescriptsthatlaunchwhenthecontainerbootscanpopulatethedatabase.

Youcanverifythateverythingworkedasexpectedbycheckingthelogsonthecontainer.Todothis,enterdockerlogszabbix.Thiswillprintdetailsofwhathappenedwhenthecontainerlaunchedonscreen:

Now,oncewehavethecontainerupandrunning,itistimetomovetothebrowserforourfirsttasteofthewebinterface.Gotohttp://192.168.33.10/inyourbrowserandyouwillbegreetedbyawelcomepage;beforewecanstartusingZabbix,weneedtocompletetheinstallation.

Onthewelcomepage,clickonNexttobetakentothefirststep.ThiswillverifythateverythingweneedtorunaZabbixServerisinstalled.Aswehavelauncheditinacontainer,youshouldseeOKnexttoalloftheprerequisites.ClickonNexttomoveontothenextstep.

Now,weneedtoconfigurethedatabaseconnectionforthewebinterface.Here,youshouldhavethesamedetailsasyoudidwhenyoulaunchedthecontainer,asillustratedinthefollowingscreenshot:

Onceyouhaveenteredthedetails,clickonTestconnectionandyoushouldreceiveanOKmessage;youwillnotbeabletoproceeduntilthistestcompletessuccessfully.OnceyouhaveenteredthedetailsandhaveanOKmessage,clickonNext.

Nextup,arethedetailsontheZabbixServerthatthewebinterfaceneedstoconnectto;clickonNexthere.Nextup,youwillreceiveasummaryoftheinstallation.Toproceed,clickonNextandyouwillbegetconfirmationthatthe/usr/local/src/zabbix/frontends/php/conf/zabbix.conf.phpfilehasbeencreated.ClickonFinishtobetakentotheloginpage.

UsingvagrantWhilewritingthischapter,IthoughtalotaboutprovidinganothersetofinstallationinstructionsfortheZabbixServerservice.WhilethebookisallaboutMonitoringDockercontainers,havingaserviceasresourceintensiveasZabbixrunninginsideacontainerfeelsalittlecounterintuitive.Becauseofthis,thereisavagrantmachinethatusesPuppettobootstrapaworkinginstallationofZabbixServer:

[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-zabbix/






.....

==>default:Debug:Receivedreporttoprocessfromzabbix.media-glass.es

==>default:Debug:Evictingcacheentryforenvironment'production'

==>default:Debug:Cachingenvironment'production'(ttl=0sec)

==>default:Debug:Processingreportfromzabbix.media-glass.eswith

processorPuppet::Reports::Store

Asyoumayhavenoticed,thereisalotofoutputstreamedtotheterminal,sowhatjusthappened?Firstofall,aCentOS7vagrantinstancewaslaunchedandthenaPuppetagentwasinstalled.Onceinstalled,theinstallationwashandedofftoPuppet.UsingtheZabbixPuppetmodulebyWernerDijkerman,ZabbixServerwasinstalled;formoredetailsonthemodule,seeitsPuppetForgepageathttps://forge.puppetlabs.com/wdijkerman/zabbix.

UnlikethecontainerizedversionofZabbixServer,thereisnoadditionalconfigurationrequired,soyoushouldbeabletoaccesstheZabbixloginpageathttp://zabbix.media-glass.es/(anIPaddressof192.168.33.11ishardcodedintotheconfiguration).

https://forge.puppetlabs.com/wdijkerman/zabbix

http://zabbix.media-glass.es/

PreparingourhostmachineFortheremainderofthischapter,IwillassumethatyouareusingtheZabbixServerthatisrunningonitsownvagrantinstance.ThishelpstoensurethatyourenvironmentisconsistentwiththeconfigurationoftheZabbixagentwewillbelookingat.

TopassthestatisticsfromourcontainerstotheZabbixagent,whichwilltheninturnexposethemtotheZabbixServer,wewillbeinstallingusingtheZabbix-Docker-MonitoringZabbixagentmodulethathasbeendevelopedbyJanGaraj.Formoreinformationontheproject,seethefollowingURLs:

TheProjectpage:https://github.com/monitoringartist/Zabbix-Docker-Monitoring/TheZabbixsharepage:https://share.zabbix.com/virtualization/docker-containers-monitoring

Togettheagentandmoduleinstalled,configured,andrunning,weneedtoexecutethefollowingsteps:

1. InstalltheZabbixpackagerepository.2. InstalltheZabbixagent.3. Installtheprerequisitesforthemodule.4. AddtheZabbixagentusertotheDockergroup.5. Downloadtheauto-discoverybashscript.6. Downloadtheprecompiledzabbix_module_dockerbinary.7. ConfiguretheZabbixagentwiththedetailsofourZabbixServerandalsotheDocker

module.8. Setthecorrectpermissionsonallthefileswehavedownloadedandcreated.9. StarttheZabbixagent.

WhilethestepsremainthesameforbothCentOSandUbuntu,theactionstakentodotheinitialpackageinstallationdifferslightly.Ratherthangoingthroughtheprocessofshowingthecommandstoinstallandconfiguretheagent,thereisascriptforeachofthehostoperatingsystemsinthe/monitoring_docker/chapter04/folder.Toviewthescripts,runthefollowingcommandfromyourterminal:

cat/monitoring_docker/chapter04/install-agent-centos.sh

cat/monitoring_docker/chapter04/install-agent-ubuntu.sh

Now,youhavetakenalookatthescriptsitstimetorunthem,todothistypeoneofthefollowingcommands.IfyouarerunningCentOS,runthiscommand:

bash/monitoring_docker/chapter04/install-agent-centos.sh

ForUbuntu,runthefollowingcommand:

bash/monitoring_docker/chapter04/install-agent-ubuntu.sh

Toverifythateverythingranasexpected,checktheZabbixagentlogfilebyrunningthefollowingcommand:

https://github.com/monitoringartist/Zabbix-Docker-Monitoring/

https://share.zabbix.com/virtualization/docker-containers-monitoring

cat/var/log/zabbix/zabbix_agentd.log

Youshouldseethattheendofthefileconfirmsthattheagenthasstartedandthatthezabbix_module_docker.somodulehasbeenloaded:

BeforewemoveontotheZabbixwebinterface,let’slaunchafewcontainersusingthedocker-composefilefromChapter2,UsingtheBuilt-inTools:

[vagrant@docker~]$cd/monitoring_docker/chapter02/02-multiple/

[vagrant@docker02-multiple]$docker-composeup-d

[vagrant@docker02-multiple]$docker-composescaleweb=3

[vagrant@docker02-multiple]$docker-composeps

WeshouldnowhavethreewebservercontainersrunningandarunningZabbixagentonthehost.

TheZabbixwebinterfaceOnceyouhaveZabbixinstalledyoucanopentheZabbixwebinterfacebygoingtohttp://zabbix.media-glass.es/inyourbrowser,thislinkwillonlyworkwhenyouhavetheZabbixvagrantboxupandrunning,ifyoudon’thaveitrunningthepagewilltimeout.Youshouldbepresentedwithaloginscreen.Enterthedefaultusernameandpasswordhere,whichisAdminandzabbix(notethattheusernamehasacapitalA),tologin.

Onceloggedin,youwillneedtoaddthehosttemplates.ThesearepreconfiguredenvironmentsettingsandwilladdsomecontextaroundthestatisticsthattheZabbixagentissendingtotheserver,alongwiththeauto-discoveryofcontainers.

Toaddthetemplates,gototheConfigurationtabinthetopmenuandselectTemplate;thiswillbringupalistofallthetemplatesthatarecurrentlyinstalled.ClickontheImportbuttonintheheaderanduploadacopyofthetwotemplatefilesyoucanfindinthe~/Documents/Projects/monitoring-docker/chapter04/templatefolderonyourmainmachine;thereisnoneedtochangetheruleswhenuploadingthetemplates.

Oncebothtemplateshavebeensuccessfullyimported,itistimetoaddourDockerhost.Again,gototheConfigurationtab,butthistimeselectHosts.Here,youneedtoclickonCreatehost.Then,enterthefollowinginformationintheHosttab:

http://zabbix.media-glass.es/

Herearethedetailsoftheprecedinginformation:

Hostname:ThisisthehostnameofourDockerhostVisiblename:Here,thenameserverwillappearasinZabbixGroups:WhichgroupwithinZabbixtheserveryouwouldliketheDockerhosttobepartofAgentInterfaces:ThisistheIPaddressortheDNSnameofourDockerhostEnabled:Thisshouldbeticked

BeforeclickingonAdd,youshouldclickontheTemplatestabandlinkthefollowingtwotemplatestothehost:

TemplateAppDockerTemplateOSLinux

Hereisthescreenshotofthehost:

Onceyouhaveaddedthetwotemplates,clickonAddtoconfigureandenablethehost.Toverifythatthehosthasbeenaddedcorrectly,youshouldgototheMonitoringtabandthenLatestdata.Fromhere,clickonShowfilterandenterthehostmachineintheHostsbox.Youshouldthenstarttoseeitemsappearing:

Don’tworryifyoudon’tseetheDockersectionimmediately,bydefault,Zabbixwillattempttoauto-discovernewcontainerseveryfiveminutes.

DockermetricsForeachcontainer,Zabbixdiscoversthefollowingmetricsthatwillberecorded:

Container(yourContainersname)isrunningCPUsystemtimeCPUusertimeUsedcachememoryUsedRSSmemoryUsedswap

Apartfrom“Usedswap”,thesearethesamemetricsrecordedbycAdvisor.

CreatecustomgraphsYoucanaccessatime-basedgraphforanyofthemetricscollectedbyZabbix;youcanalsocreateyourowncustomgraphs.Inthefollowinggraph,IhavecreatedagraphthatplotsalltheCPUSystemstatsfromthethreewebcontainerswelaunchedearlierinthechapter:

Asyoucansee,IperformedafewtestsusingApacheBenchtomakethegraphalittlemoreinteresting.

Formoreinformationonhowtocreatecustomgraphs,seethegraphssectionofthedocumentationsiteathttps://www.zabbix.com/documentation/2.4/manual/config/visualisation/graphs.

https://www.zabbix.com/documentation/2.4/manual/config/visualisation/graphs

ComparecontainerstoyourhostmachineAsweaddedtheLinuxOStemplateandtheDockertemplatetothehostandwearealsorecordingquitealotofinformationaboutthesystem,herewecantelltheeffectthetestingwithApacheBenchhadontheoverallprocessorload:

Wecandrilldownfurthertogetinformationontheoverallutilization:

TriggersAnotherfeatureofZabbixistriggers:youcandefineactionstohappenwhenametricmeetsacertainsetofcriteria.Inthefollowingexample,ZabbixhasbeenconfiguredwithatriggercalledContainerDown;thischangesthestatusofthemonitoreditemtoProblemwithaseverityofDisaster:

Thischangeinstatusthentriggersane-mailtoinformthat,forsomereasonthecontainerisnolongerupandrunning:

Thiscouldhavealsotriggeredothertasks,suchasrunningacustomscript,sendinganinstantmessageviaJabber,oreventriggeringathird-partyservicesuchasPagerDuty(https://www.pagerduty.com)orSlack(https://slack.com).

https://www.pagerduty.com

https://slack.com

FormoreinformationonTriggers,Events,andNotifications,seethefollowingsectionsofthedocumentation:

https://www.zabbix.com/documentation/2.4/manual/config/triggershttps://www.zabbix.com/documentation/2.4/manual/config/eventshttps://www.zabbix.com/documentation/2.4/manual/config/notifications

https://www.zabbix.com/documentation/2.4/manual/config/triggers

https://www.zabbix.com/documentation/2.4/manual/config/events

https://www.zabbix.com/documentation/2.4/manual/config/notifications

SummarySo,howdoesthistraditionalapproachtomonitoringfitintoacontainer’slifecycle?

GoingbacktothePetsversusCattleanalogy,atfirstglance,ZabbixseemstobegearedmoretowardsPets:itsfeaturesetisbestsuitedtomonitoringservicesthatarestaticoveralongperiodoftime.Thismeansthatthesameapproachtomonitoringapetcanalsobeappliedtolong-runningprocessesrunningwithinyourcontainers.

Zabbixisalsotheperfectoptionformonitoringmixedenvironments.Maybeyouhaveseveraldatabaseserversthatarenotrunningascontainers,butyouhaveseveralhostsrunningDocker,andhaveequipmentsuchasswitchesandSANsthatyouneedtomonitor.Zabbixcanprovideyouwithasinglepaneofglassshowingyoumetricsforallyourenvironments,alongwithbeingabletoalertyoutoproblems.

Sofar,wehavelookedatusingAPIsandmetricsprovidedbyDockerandLXC,butwhataboutothermetricscanweuse?Inthenextchapter,wewilllookatatoolthathooksstraightintothehostmachine’skerneltogatherinformationonyourcontainers.

Chapter5.QueryingwithSysdigTheprevioustoolswehavelookedathaveallreliedonmakingAPIcallstoDockerorreadingmetricsfromLXC.Sysdigworksdifferentlybyhookingitselfintothehostsmachine’skernelwhilethisapproachdoesgoagainstDocker’sphilosophyofeachservicebeingruninitsownisolatedcontainer,theinformationyoucangetbyrunningSysdigonlyforafewminutesfaroutweighsanyargumentsaboutnotusingit.

Inthischapter,wewilllookatthefollowingtopics:

HowtoinstallSysdigandCsysdigonthehostmachineBasicusageandhowtoqueryyourcontainersinrealtimeHowtocapturelogssotheycanbequeriedlater

WhatisSysdig?BeforewestarttogetintoSysdig,let’sfirstunderstandwhatitis.WhenIfirstheardaboutthetool,Ithoughttomyselfthatitsoundedtoogoodtobetrue;thewebsitedescribesthetoolasfollows:

“Sysdigisopensource,system-levelexploration:capturesystemstateandactivityfromarunningLinuxinstance,thensave,filterandanalyze.SysdigisscriptableinLuaandincludesacommandlineinterfaceandapowerfulinteractiveUI,csysdig,thatrunsinyourterminal.Thinkofsysdigasstrace+tcpdump+htop+iftop+lsof+awesomesauce.Withstateoftheartcontainervisibilityontop.”

Thisisquiteaclaimasallthetoolsthatitisclaimingtobeaspowerfulwereallinasetofgotocommandstorunwhenlookingintoproblems,soIwasalittleskepticalatfirst.

Asanyonewhohashadtotryandtrackdownahaywireprocessoftryandtrackdownanissuethatisn’tbeingveryverboseinitserrorlogsonaLinuxserverwillknowthatusingtoolssuchasstrace,lsof,andtcpdumpcangetcomplicatedveryquicklyanditnormallyinvolvescapturingawholelotofdataandthenusingacombinationofseveraltoolstoslowly,andmanually,tracetheproblembyreducingtheamountofdatayoucaptured.

ImaginemydelightwhenSysdig’sclaimsturnedouttobetrue.ItmademewishIhadthetoolbackwhenIwasafrontlineengineer;itwouldhavemademylifealoteasier.

Sysdigcomesintwodifferentflavors,firstistheOpenSourceversionavailableathttp://www.sysdig.org/;thiscomeswithanncursesinterfacesothatyoucaneasilyaccessandquerydatafromaterminal-basedGUI.

NoteWikipediadescribesncurses(newcurses)asaprogramminglibrarythatprovidesanAPIthatallowstheprogrammertowritetext-baseduserinterfacesinaterminal-independentmanner.Itisatoolkitfordeveloping“GUI-like”applicationsoftwarethatrunsunderaterminalemulator.Italsooptimizesscreenchangesinordertoreducethelatencyexperiencedwhenusingremoteshells.

ThereisalsoacommercialservicethatallowsyoutostreamyourSysdigtotheirexternallyhostedservice;thisversionhasaweb-basedinterfaceforviewingandqueryingyourdata.

Inthischapter,wewillbeconcentratingontheopensourceversion.

http://www.sysdig.org/

InstallingSysdigConsideringhowpowerfulSysdigis,ithasoneofthemoststraightforwardinstallationandconfigurationprocessesIhavecomeacross.ToinstallSysdigoneitheraCentOSorUbuntuserver,typethefollowingcommand:

curl-shttps://s3.amazonaws.com/download.draios.com/stable/install-sysdig

|sudobash

Afterrunningtheprecedingcommand,youwillgetthefollowingoutput:

That’sit,youarereadytogo.Thereisnothingmoretoconfigureordo.Thereisamanualinstallationprocessandalsoawayofinstallingthetoolusingcontainerstobuildthenecessarykernelmodules;formoredetails,seetheinstallationguideasfollows:

http://www.sysdig.org/wiki/how-to-install-sysdig-for-linux/

http://www.sysdig.org/wiki/how-to-install-sysdig-for-linux/

UsingSysdigBeforewelookathowtouseSysdig,let’slaunchafewcontainersusingdocker-composebyrunningthefollowingcommand:

cd/monitoring_docker/chapter05/wordpress/

docker-composeup–d

ThiswilllaunchaWordPressinstallationrunningadatabaseandtwowebservercontainersthatareloadbalancedusinganHAProxycontainer.YouwillbeabletoviewtheWordPressinstallationathttp://docker.media-glass.es/oncethecontainershavelaunched.Youwillneedtoentersomedetailstocreatetheadminuserbeforethesiteisvisible;followtheon-screenpromptstocompletethesesteps.


ThebasicsAtitscore,Sysdigisatoolforproducingastreamofdata;youcanviewthestreambytypingsudosysdig(toquit,pressCtrl+c).

Thereisalotinformationtheresolet’sstarttofilterthestreamdownandrunthefollowingcommand:

sudosysdigevt.type=chdir

Thiswilldisplayonlyeventsinwhichauserchangesdirectory;toseeitinaction,openasecondterminalandyouwillseethatwhenyoulogin,youseesomeactivityinthefirstterminal.Asyoucansee,itlooksalotlikeatraditionallogfile;wecanformatoutputtogiveinformationsuchastheusername,byrunningthefollowingcommand:

sudosysdig-p"user:%user.namedir:%evt.arg.path"evt.type=chdir

Then,inyoursecondterminal,changethedirectoryafewtimes:

Asyoucansee,thisisaloteasiertoreadthantheoriginalunformattedoutput.PressCtrl+ctostopfiltering.

CapturingdataIntheprevioussection,welookedatfilteringdatainrealtime;itisalsopossibletostreamSysdigdatatoafilesothatyoucanquerythedataatalatertime.Exitfromyoursecondterminalandrunthefollowingcommandonyourfirstone:

sudosysdig-w~/monitoring-docker.scap

Whilethecommandisrunningonthefirstterminal,logintothehostonthesecondoneandchangethedirectoryafewtimes.Also,whilewearerecording,clickaroundtheWordPresssitewestartedatthebeginningofthissection,theURLishttp://docker.media-glass.es/.Onceyouhavedonethat,stoptherecordingbypressingCrtl+c;youshouldhavenowdroppedbacktoaprompt.YoucancheckthesizeofthefilecreatedbySysdigbyrunningthefollowing:

ls-lha~/monitoring-docker.scap

Now,wecanusethedatathatwehavecapturedtoapplythesamefilteraswedidwhenlookingatthereal-timestream:

sudosysdig-r~/monitoring-docker.scap-p"user:%user.name

dir:%evt.arg.path"evt.type=chdir

Byrunningtheprecedingcommand,youwillgetthefollowingoutput:

Noticehowwegetsimilarresultstowhenwewereviewingthedatainrealtime.

ContainersOneofthethingsthatwasrecordedin~/monitoring-docker.scapwasdetailsonthesystemstate;thisincludesinformationonthecontainerswelaunchedatthestartofthechapter.Let’susethisfiletogetsomestatsonthecontainers.Tolistthecontainersthatwereactiveduringthetime,wecapturedthedatafilerun:

sudosysdig-r~/monitoring-docker.scap-clscontainers

ToseewhichofthecontainersutilizedtheCPUmostofthetime,wewereclickingaroundtheWordPresssiterun:

sudosysdig-r~/monitoring-docker.scap-ctopcontainers_cpu

Tohavealookatthetopprocessesineachofthecontainersthathave“wordpress”intheirnames(whichisalloftheminourcase),runthefollowingcommand:

sudosysdig-r~/monitoring-docker.scap-ctopprocs_cpucontainer.name

containswordpress

Finally,whichofourcontainerstransferredthemostamountofdata?:

sudosysdig-r~/monitoring-docker.scap-ctopcontainers_net

Byrunningtheprecedingcommand,youwillgetthefollowingoutput:

Asyoucansee,wehaveextractedquiteabitofinformationonourcontainersfromthedatawecaptured.Also,usingthefile,youcanremovethe-r~/monitoring-docker.scappartofthecommandtoviewthecontainermetricsinrealtime.

It’salsoworthpointingoutthattherearebinariesforSysdigthatworkonbothOSXandWindows;whilethesedonotcaptureanydata,theycanbeusedtoreaddatathatyouhaverecordedonyourLinuxhost.

FurtherreadingFromthefewbasicexercisescoveredinthissection,youshouldstarttogetanideaofjusthowpowerfulSysdigcanbe.TherearemoreexamplesontheSysdigwebsiteathttp://www.sysdig.org/wiki/sysdig-examples/.Also,Irecommendyoutoreadtheblogpostathttps://sysdig.com/fishing-for-hackers/;itwasmyfirstexposuretoSysdiganditreallydemonstratesitsusefulness.

http://www.sysdig.org/wiki/sysdig-examples/

https://sysdig.com/fishing-for-hackers/

UsingCsysdigAseasyasitistoviewdatacapturedbySysdigusingthecommandlineandmanuallyfilteringtheresults,itcangetmorecomplicatedasyoustarttostringmoreandmorecommandstogether.TohelpmakethedatacapturedbySysdigasaccessibleaspossible,SysdigshipswithaGUIcalledCsysdig.

LaunchingtheCsysdigisdonewithasinglecommand:

sudocsysdig

Oncetheprocesshaslaunched,itshouldinstantlylookfamiliartoanyonewhohasusedtoporcAdvisor(minusthegraphs);itsdefaultviewwillshowyoureal-timeinformationontheprocessesthatarerunning:

Tochangethisview,knownastheProcessesview,pressF2toopentheViewsmenu;fromhere,youcanusetheupanddownarrowsonyourkeyboardtoselectaview.Asyoumayhavealreadyguessed,wewouldliketoseetheContainersview:

However,beforewedrilldownintoourcontainers,let’squitCsysdigbypressingqandloadupthefilewecreatedintheprevioussection.Todothis,typethefollowingcommand:

sudocsysdig-r~/monitoring-docker.scap

OnceCsysdigloads,youwillnoticethatSourcehaschangedfromLiveSystemtothefilepathofourdatafile.Fromhere,pressF2andusetheuparrowtoselectcontainersandthenhitEnter.Fromhere,youcanusetheupanddownarrowstoselectoneofthetwowebservers,thesewouldbeeitherwordpress_wordpress1_1orwordpress_wordpress2_1asshowninthefollowingscreen:

NoteTheremainingpartofthischapterassumesthatyouhaveCsysdigopenin-frontofyou,itwilltalkyouthroughhowtonavigatearoundthetool.Pleasefeelfreetoexploreyourselfaswell.

Onceyouhaveselectedaserver,hitEnterandyouwillbepresentedwithalistofprocessesthatthecontainerwasrunning.Again,youcanusethearrowkeystoselectaprocesstodrilldownfurtherinto.

IsuggestedlookingatoneoftheApacheprocessesthathasavaluelistedintheFilecolumn.Thistime,ratherthanpressingEntertoselecttheprocess,let’s“Echo”whattheprocesswasuptoatthetimewecapturedthedata;withtheprocessselected,pressF5.

Youcanusetheupanddownarrowstoscrollthroughtheoutput:

Tobetterformatthedata,pressF2andselectPrintableASCII.Asyoucanseefromtheprecedingscreenshot,thisApacheprocessperformedthefollowingtasks:

AcceptedanincomingconnectionAccessedthe.htaccessfileReadthemod_rewriterulesGotinformationfromthehostsfileMadeaconnectiontotheMySQLcontainerSenttheMySQLpassword

Byscrollingthroughtheremainderofthedatainthe“Echo”resultsfortheprocess,youshouldbeabletoeasilyfollowtheinteractionswiththedatabaseallthewaythroughtothepagebeingsenttothebrowser.

Toleavethe“Echo”screen,pressBackspace;thiswillalwaystakeyoualevelback.

Ifyouwantamoredetailedbreakdownonwhattheprocesswasdoing,thenpressF6toentertheDigview;thiswilllistthefilesthattheprocesswasaccessingatthetime,alongwiththenetworkinteractionandhowitisaccessingtheRAM.

Toviewafulllistofcommandsandformorehelp,youcanpressF1atanytime.Also,togetabreakdownonanycolumnsthatareonscreen,pressF7.

SummaryAsImentionedatthestartofthischapter,SysdigisprobablyoneofthemostpowerfultoolsIhavecomeacrossinrecentyears.

Partofitspoweristhewaythatitexposesalotofinformationandmetricsinawaythatneverfeelsoverwhelming.It’sclearthatthedevelopershavespentalotoftimeensuringthatboththeUIandthewaythatcommandsarestructuredfeelnaturalandinstantlyunderstandable,evenbythenewestmemberofanoperationsteam.

Theonlydownsideisthat,unlessyouwanttoviewtheinformationinrealtimeorlookintoaproblemindevelopmentstoringtheamountofdatathatisbeinggeneratedbySysdig,itcanbequitecostlyintermsofdiscspacebeingused.

ThisissomethingthatSysdighasrecognized,andtohelpwiththis,thecompanyoffersacloud-basedcommercialservicecalledSysdigCloudforyoutostreamyourSysdigdatainto.Inthenextchapter,wewilllookatthisserviceandalsosomeofitscompetitors.

Chapter6.ExploringThirdPartyOptionsSofar,wehavebeenlookingatthetoolsandservicesyouhostyourself.Alongwiththeseself-hostedtools,alargeamountofcloud-basedsoftwarehasdevelopedaroundDockerasaserviceecosystem.Inthischapter,wewilllookatthefollowingtopics:

WhyuseaSaaSserviceoverself-hostedorreal-timemetrics?Whatservicesareavailableandwhatdotheyoffer?InstallationofagentsforSysdigCloud,Datadog,andNewReliconthehostmachinesConfigurationoftheagentstoshipmetrics

AwordaboutexternallyhostedservicesSofar,toworkthroughtheexamplesinthisbook,wehaveusedlocallyhostedvirtualserversthatarelaunchedusingvagrant.Duringthischapter,wearegoingtouseservicesthatneedtobeabletocommunicatewithyourhostmachine,soratherthantryingtodothisusingyourlocalmachine,itsabouttimeyoutookyourhostmachineintothecloud.

Aswearegoingtostartandstoptheremotehostswhilewelookattheservices,itpaystouseapubliccloud,asweonlygetchargedforwhatweuse.

Thereareseveralpubliccloudservicesthatyoucanusetoevaluatethetoolscoveredinthischapter,whichoneyouchoosetouseisuptoyou,youcoulduse:

DigitalOcean:https://www.digitalocean.com/AmazonWebServices:https://aws.amazon.com/MicrosoftAzure:https://azure.microsoft.com/VMwarevCloudAir:http://vcloud.vmware.com/

Oruseyourownpreferredprovider,theonlypre-requisiteisthatyourserverispublicallyaccessible.

ThischapterassumesthatyouarecapableoflaunchingeitheraCentOS7orUbuntu14.04cloudinstanceandyouunderstandthatyouwilllikelyincurchargeswhilethecloudinstanceisupandrunning.

https://www.digitalocean.com/

https://aws.amazon.com/

https://azure.microsoft.com/

http://vcloud.vmware.com/

DeployingDockerinthecloudOnceyouhavelaunchedyourcloudinstance,youcanbootstrapDockerinthesamewaythatyouinstalledusingvagrant.Inthechapter6folderoftheGitrepository,therearetwoseparatescriptstodownloadandinstalltheDockerengineandcomposeitonyourcloudinstance.

ToinstallDocker,ensurethatyourcloudinstanceisupdatedbyrunning:

sudoyumupdate

FortheCentOSinstanceofyourUbuntu,runthefollowingcommand:

sudoapt-getupdate

Onceupdated,runthefollowingcommandtoinstallthesoftware.Duetothedifferencesinthewaydifferentcloudenvironmentsareconfigured,itisbesttoswitchovertotherootusertoruntheremainderofthecommands,todothis,run:

sudosu-

Nowyouwillbeabletoruntheinstallscriptusingthefollowingcommand:

curl-fsShttps://raw.githubusercontent.com/russmckendrick/monitoring-

docker/master/chapter06/install_docker/install_docker.sh|bash

Tocheckthateverythingworksasexpected,runthefollowingcommand:

dockerrunhello-world

Youshouldseesomethingsimilartotheterminaloutput,asshowninthefollowingscreenshot:

WecanstarttolookattheSasSservicesonceyouhaveDockerupandrunning.

WhyuseaSaaSservice?Youmayhavenoticedwhileworkingwiththeexamplesinthepreviouschaptersthatthetoolswehaveusedcanpotentiallyusemanyresourcesifweneededtostartcollectingmoremetrics,especiallyiftheapplicationswewanttomonitorareinproduction.

TohelpshiftthisloadfrombothstorageandCPU,anumberofcloud-basedSaaSoptionshavestartedofferingsupporttorecordmetricsforyourcontainers.Manyoftheseserviceswerealreadyofferingservicestomonitorservers,soaddingsupportforcontainersseemedanaturalprogressionforthem.

Thesetypicallyrequireyoutoinstallanagentonyourhostmachine,onceinstalled,theagentwillsitinthebackgroundandreporttotheservices,normallycloud-basedandAPIservices.

AfewoftheservicesallowyoutodeploytheagentsasDockercontainers.Theyoffercontainerizedagentssothattheservicecanrunonstrippeddownoperatingsystems,suchas:

CoreOS:https://coreos.com/RancherOS:http://rancher.com/rancher-os/Atomic:http://www.projectatomic.io/UbuntuSnappyCore:https://developer.ubuntu.com/en/snappy/

Theseoperatingsystemsdifferfromtraditionalones,asyoucannotinstallservicesonthemdirectly;theironlypurposeistorunaservice,suchasDocker,sothatyoucanlaunchtheservicesorapplicationsyouneedtoberunascontainers.

Aswearerunningfulloperatingsystemsasourhostsystems,wedonotneedthisoptionandwillbedeployingtheagentsdirectlytothehosts.

TheSaaSoptionsthatwearegoingtolookatinthischapterareasfollows:

SysdigCloud:https://sysdig.com/product/Datadog:https://www.datadoghq.com/NewRelic:http://newrelic.com

Theyallofferfreetrialsandtwoofthemofferfreecut-downversionsofthemainservice.Onthefaceofit,theymightallappeartooffersimilarservices;however,whenyoustarttousethem,youwillimmediatelynoticethattheyareinfactallverydifferentfromeachother.

https://coreos.com/

http://rancher.com/rancher-os/

http://www.projectatomic.io/

https://developer.ubuntu.com/en/snappy/

https://sysdig.com/product/

https://www.datadoghq.com/

http://newrelic.com

SysdigCloudInthepreviouschapter,wehadalookattheopensourceversionofSysdig.WesawthatthereisagreatncursesinterfacecalledcSysdiganditallowsustonavigatethroughallthedatathatSysdigiscollectingaboutourhost.

ThesheeramountofmetricsanddatacollectedbySysdigmeansthatyouhavetotrytostayontopofiteitherbyshippingyourfilesofftheserver,maybetoAmazonSimpleStorageService(S3),ortosomelocalsharedstorage.Inaddition,youcanquerythedatainthecommandlineonthehostitselforonyourlocalmachineusinganinstallationofthecommand-linetools.

ThisiswhereSysdigCloudcomesintoplay;itoffersaweb-basedinterfacetothemetricsthatSysdigcapturesalongwiththeoptionstoshiptheSysdigcapturesoffyourhostmachineeithertoSysdig’sownstorageortoyourS3bucket.

Sysdigcloudoffersthefollowingfunctionality:

ContainerVision™Real-TimeDashboardHistoricalReplayDynamicTopologyAlerting

Aswellas,theoptiontotriggeracaptureonanyofyourhostsandatanytime.

SysdigdescribesContainerVisionas:

“SysdigCloud’spatent-pendingcoretechnology,ContainerVision,istheonlymonitoringtechnologyonthemarketdesignedspecificallytorespecttheuniquecharacteristicsofcontainers.ContainerVisionoffersyoudeepandcomprehensivevisibilityintoallaspectsofyourcontainerizedenvironment-applications,infrastructures,servers,andnetworks-allwithouttheneedtopolluteyourcontainerswithanyextrainstrumentation.Inotherwords,ContainerVisiongivesyou100%visibilityintotheactivityinsideyourcontainers,fromtheoutside.”

BeforewedelveintoSysdigCloudanyfurther,Ishouldpointoutthatthisisacommercialserverandatthetimeofwriting,itcosts$25perhostpermonth.Thereisalsoa14-dayfullyfeaturedtrialavailable.Ifyouwishtoworkthroughtheagentinstallationandfollowtheexampleinthischapter,youwillneedanactiveaccountthatrunseitheronthe14-daytrialorapaidsubscription.

Signupfora14-dayfreetrial:https://sysdig.com/Detailsonpricing:https://sysdig.com/pricing/Introductiontothecompany:https://sysdig.com/company/

https://sysdig.com/

https://sysdig.com/pricing/

https://sysdig.com/company/

InstallingtheagentTheagentinstallationissimilartoinstallingtheopensourceversion;youneedtoensurethatyourcloudhostisrunninganup-to-datekernelandthatyouarealsobootedintothekernel.

Somecloudproviderskeepatightcontrolonthekernelsyoucanbootinto(forexample,DigitalOcean),andtheydonotallowyoutomanageyourkernelonthehostitself.Instead,youneedtochoosethecorrectversionthroughtheircontrolpanel.

Onceyouhavethecorrectkernelinstalled,youshouldbeabletorunthefollowingcommandtoinstalltheagent.Ensurethatyoureplacetheaccesskeyattheendofthecommandwithyourownaccesskey,whichcanbefoundonyourUserProfilepageorontheagentinstallationpages;youcanfindtheseat:

UserProfile:https://app.sysdigcloud.com/#/settings/userAgentInstallation:https://app.sysdigcloud.com/#/settings/agentInstallation

Thecommandtorunis:

curl-shttps://s3.amazonaws.com/download.draios.com/stable/install-agent|

sudobash-s—--access_keywn5AYlhjRhgn3shcjW14y3yOT09WsF7d

Theshelloutputshouldlooklikethefollowingscreen:

Oncetheagenthasbeeninstalled,itwillimmediatelystarttoreportthedatabacktoSysdigCloud.IfyouclickonExplore,youwillseeyourhostmachineandtherunning

https://app.sysdigcloud.com/#/settings/user

https://app.sysdigcloud.com/#/settings/agentInstallation

containers:

Asyoucanseehere,IhavemyhostmachineandfourcontainersrunningaWordPressinstallationsimilartotheoneweusedinthepreviouschapter.Fromhere,wecanstarttodrilldownintoourmetrics.

TolaunchtheWordPressinstallationonyourcloud-basedmachine,runthefollowingcommandsastherootuser:

sudosu-

mkdir~/wordpress

curl-Lhttps://raw.githubusercontent.com/russmckendrick/monitoring-

docker/master/chapter05/wordpress/docker-compose.yml>~/wordpress/docker-

compose.yml

cd~/wordpress

docker-composeup-d

ExploringyourcontainersTheSysdigCloudwebinterfacewillfeelinstantlyfamiliar,asitsharesasimilardesignandoverallfeelingwithcSysdig:

Onceyoustarttodrilldown,youcanseethatabottompaneopensupandthisiswhereyoucanviewthestatistics.OneofthethingsIlikedaboutSysdigCloudisthatitopensupawealthofmetricsandthereshouldbeverylittlethatyouneedtoconfigurefromhere.

Forexample,ifyouwanttoknowwhatprocesseshavebeenconsumingthemostCPUtimeinthelast2hours,clickon2HinthesecondarymenuandthenfromtheViewstabinthebottom-leftclickonSystem:TopProcesses;thiswillgiveyouatableoftheprocesses,orderedbytheonesthathaveusedthemosttime.

Toapplythisviewtoacontainer,clickonacontainerinthetop-sectionandthebottom-sectionwillbeinstantlyupdatedtoreflectthetopCPUutilizationforjustthatcontainer;asmostcontainerswillonlyrunoneortwoprocesses,thismaynotbethatinteresting.So,let’shaveadeeplookattheprocessesthemselves.Let’ssay,weclickedonourdatabase

containerandwewantedinformationonwhatishappeningwithinMySQL.

SysdigCloudcomeswithapplicationoverlays,thesewhenselectedgiveyoumoregranularinformationontheprocesseswithinthecontainer.SelectingtheApp:MySQL/PostgreSQLviewgivesyouaninsightintowhatyourMySQLprocessesarecurrentlydoing:

Here,youcanseethatviewinthebottomsectionhasinstantlyupdatedtogiveawealthofinformationonwhathasbeenhappeninginthelast5minuteswithinMySQL.

SysdigCloudsupportsanumberofapplicationviews,including:

ApacheHAProxy

NGINXRabbitMQRedisTomcat

Eachonegivesyouimmediateaccesstometrics,whicheventhemostexperiencedSysAdminswillfindvaluable.

Youmayhavenoticedthatatthetopofthesecondpaneltherearealsoafewicons,theseallowyouto:

AddAlert:Createsanalertbasedontheviewyouhaveopen;itletsyoutweakthethresholdandalsochoosehowyouarenotified.SysdigCapture:Pressingthisbringsupadialog,whichletsyourecordaSysdigsession.Oncerecorded,thesessionistransferredtoSysdigCloudoryourownS3bucket.Oncethesessionisavailable,youdownloaditorexploreitwithinthewebinterface.SSHConnect:GetsaremoteshellontheserverfromtheSysdigCloudwebinterface;itisusefulifyoudonothaveimmediateaccesstoyourlaptopordesktopmachineandyouwanttodosometroubleshooting.Pintodashboard:Addsthecurrentviewtoacustomdashboard.

Outtheseoptionsicons,the“AddAlert”and“SysdigCapture”optionsareprobablytheonesthatyouwillendupusingthemost.OnefinalviewthatIfoundinteresting,isthetopologyone.Itgivesyouabird’seyeviewofyourhostandcontainers,thisisusefultooseetheinteractionbetweencontainersandhosts:

Here,youcanseemerequestapagefromtheWordPresssite(it’sintheboxontheleft),thisrequesthitsmyhostmachine(theboxontheright).Onceit’sonthehostmachine,itisroutedtotheHAProxycontainer,whichthenpassesthepagerequesttotheWordpress2container.Fromhere,theWordpress2containerinteractswiththedatabasethatisrunningontheMySQLcontainer.

SummaryandfurtherreadingAlthoughSysdigCloudisquiteanewservice,itfeelsinstantlyfamiliarandfullyfeaturedasitisbuiltontopofanalreadyestablishedandrespectedopensourcetechnology.Ifyoulike,thelevelofdetailyougetfromtheopensourceversionofSysdig,thenSysdigCloudisanaturalprogressionforyoutostartstoringyourmetricsoffsiteandalsotoconfigurealerts.SomegoodstartingpointsforlearningmoreaboutSysdigCloudare:

VideoIntroduction:https://www.youtube.com/watch?v=p8UVbpw8n24SysdigCloudBestPractices:http://support.sysdigcloud.com/hc/en-us/articles/204872795-Best-PracticesDashboards:http://support.sysdigcloud.com/hc/en-us/articles/204863385-DashboardsSysdigblog:https://sysdig.com/blog/

TipIfyouhavelaunchedacloudinstanceandarenolongerusingit,nowwouldbeagoodtimetopowertheinstancedownorterminateitaltogether.Thiswillensurethatyoudonotgetbilledforservicesthatyouarenotusing.

https://www.youtube.com/watch?v=p8UVbpw8n24

http://support.sysdigcloud.com/hc/en-us/articles/204872795-Best-Practices

http://support.sysdigcloud.com/hc/en-us/articles/204863385-Dashboards

https://sysdig.com/blog/

DatadogDatadogisafullmonitoringplatform;itsupportsvariousservers,platforms,andapplications.Wikipediadescribestheserviceas:

“DatadogisaSaaS-basedmonitoringandanalyticsplatformforITinfrastructure,operationsanddevelopmentteams.Itbringstogetherdatafromservers,databases,applications,toolsandservicestopresentaunifiedviewoftheapplicationsthatrunatscaleinthecloud.”

Itusesanagentthatisinstalledonyourhostmachine;thisagentsendsmetricsbacktotheDatadogserviceperiodically.Italsosupportmultiplecloudplatforms,suchasAmazonWebServices,MicrosoftAzure,andOpenStacktonameafew.

Theaimistobringallofyourservers,applications,andhostprovidermetricsintoasinglepaneofglass;fromhere,youcancreatecustomdashboardsandalertssothatyoucanbenotifiedofanyproblematanylevelwithinyourinfrastructure.

Youcansignupforafreetrialofthefullserviceathttps://app.datadoghq.com/signup.Youwillneedatleastatrialaccounttoconfigurethealtering,andifyourtrialhasalreadyexpiredtheliteaccountwilldo.FormoredetailonDatadog’spricingstructure,pleaseseehttps://www.datadoghq.com/pricing/.

https://app.datadoghq.com/signup

https://www.datadoghq.com/pricing/

InstallingtheagentTheagentcanbeinstalledeitherdirectlyonthehostmachineorasacontainer.Toinstalldirectlyonthehostmachine,runthefollowingcommandandmakesurethatyouuseyourownuniqueDD_API_KEY:

DD_API_KEY=wn5AYlhjRhgn3shcjW14y3yOT09WsF7dbash-c"$(curl-L

https://raw.githubusercontent.com/DataDog/dd-

agent/master/packaging/datadog-agent/source/install_agent.sh)"

Toruntheagentasacontainer,usethefollowingcommandandagainmakesurethatyouuseyourownDD_API_KEY:

sudodockerrun-d--namedd-agent-h`hostname`-v

/var/run/docker.sock:/var/run/docker.sock-v

/proc/mounts:/host/proc/mounts:ro-v/sys/fs/cgroup/:/host/sys/fs/cgroup:ro

-eAPI_KEY=wn5AYlhjRhgn3shcjW14y3yOT09WsF7ddatadog/docker-dd-agent

Oncetheagenthasbeeninstalled,itwillcallbacktoDatadogandthehostwillappearinyouraccount.

IftheagenthasbeeninstalleddirectlyonthehostmachinethenwewillneedtoenabletheDockerintegration,ifyouinstalledtheagentusingthecontainerthenthiswillhavebeendoneforyouautomatically.

Todothis,youfirstneedtoallowtheDatadogagentaccesstoyourDockerinstallationbyaddingthedd-agentusertotheDockergroupbyrunningthefollowingcommand:

usermod-a-Gdockerdd-agent

Thenextstepistocreatethedocker.yamlconfigurationfile,luckilytheDatadogagentshipswithanexampleconfigurationfilethatwecanuse;copythisinplaceandthenrestarttheagent:

cp-pr/etc/dd-agent/conf.d/docker.yaml.example/etc/dd-

agent/conf.d/docker.yaml

sudo/etc/init.d/datadog-agentrestart

Nowtheagentonourhostmachinehasbeenconfiguredandthefinalstepistoenabletheintegrationthroughthewebsite.Todothis,gotohttps://app.datadoghq.com/andclickonIntegrations,scrolldownandthenclickoninstallonDocker:

https://app.datadoghq.com/

Onceyouclickinstall,youwillbepresentedwithanoverviewoftheintegration,clickontheConfigurationtab,thisgivesinstructionsonhowtoconfiguretheagent;aswehavealreadydonethisstep,youcanclickonInstallIntegration.

YoucanfindmoreinformationoninstallingtheagentandtheintegrationsatthefollowingURLs:

https://app.datadoghq.com/account/settings#agenthttps://app.datadoghq.com/account/settings#integrations

https://app.datadoghq.com/account/settings#agent

https://app.datadoghq.com/account/settings#integrations

ExploringthewebinterfaceNow,youhaveinstalledtheagentandenabledtheDockerintegration,youcanstarttohavealookaroundthewebinterface.Tofindyourhost,clickon“Infrastructure”intheleft-handsidemenu.

Youshouldbetakentoascreenthatcontainsamapofyourinfrastructure.Likeme,youprobablyonlyhaveasinglehostmachinelisted,clickonitandsomebasicstatsshouldappearatthebottomofthescreen:

Ifyoudon’talreadyhavethecontainerslaunched,nowwouldbeagoodtimetodoso,letslaunchtheWordPressinstallationagainusing:

sudosu-

mkdir~/wordpress

curl-Lhttps://raw.githubusercontent.com/russmckendrick/monitoring-

docker/master/chapter05/wordpress/docker-compose.yml>~/wordpress/docker-

compose.yml

cd~/wordpress

docker-composeup-d

Now,gobacktothewebinterface,fromthereyoucanclickonanyoftheserviceslistedonthehexagon.Thiswillbringupsomebasicmetricsfortheserviceyouhaveselected.Ifyouclickondocker,youwillseealinkforaDockerDashboardamongthevariousgraphsandsoon;clickingthiswilltakeyoutoamoredetailedviewofyourcontainers:

Asyoucansee,thisgivesusournowfamiliarbreakdownoftheCPUandmemorymetrics,alongwithinthetoprightofthedashboardabreakdownofthecontaineractivityonthehostmachine;thislogsevents,suchasstoppingandstartingcontainers.

Datadogcurrentlyrecordsthefollowingmetrics:

docker.containers.running

docker.containers.stopped

docker.cpu.system

docker.cpu.user

docker.images.available

docker.images.intermediate

docker.mem.cache

docker.mem.rss

docker.mem.swap

FromtheMetricsexploreroptionintheleft-handsidemenu,youcanstarttographthesemetricsandonceyouhavethegraphs,youcanthenstarttoaddthemtoyourowncustomdashboardsorevenannotatethem.Whenyouannotateagraph,asnapshotiscreatedandthegraphshowsupintheeventsqueuealongwiththeotherevents,thathavebeenrecorded,suchascontainerstoppingandstarting:

Also,withinthewebinterfaceyoucanconfiguremonitors;theseallowyoutodefinetriggers,whichalertyouifyourconditionsarenotmet.Alertscanbesentase-mailsorviathirdpartyservices,suchasSlack,Campfire,orPagerDuty.

SummaryandfurtherreadingWhileDatadog’sDockerintegrationonlygivesyouthebasicmetricsonyourcontainers,itdoeshaveawealthoffeaturesandintegrationwithotherapplicationsandthirdparties.IfyouneedtomonitoranumberofdifferentservicesalongsideyourDockercontainers,thenthisservicecouldbeforyou:

Homepage:https://www.datadoghq.comOverview:https://www.datadoghq.com/product/MonitoringDockerwithDatadog:https://www.datadoghq.com/blog/monitor-docker-datadog/Twitter:https://twitter.com/datadoghq

TipPleaseRemember

Ifyouhavelaunchedacloudinstanceandarenolongerusingitthennowwouldbeagoodtimetopowertheinstancedownorterminateitaltogether.Thiswillensurethatyoudonotgetbilledforanyservicesyouarenotusing.

https://www.datadoghq.com

https://www.datadoghq.com/product/

https://www.datadoghq.com/blog/monitor-docker-datadog/

https://twitter.com/datadoghq

NewRelicNewReliccouldbeconsideredthegranddaddyofSaaSmonitoringtools,chancesarethatifyouareadeveloperyouwillhaveheardofNewRelic.IthasbeenaroundforawhileanditisthestandardtowhichotherSaaStoolscomparethemselves.

NewRelichasgrownintoseveralproductsovertheyear,currently,theyoffer:

NewRelicAPM:Themainapplicationperformance-monitoringtool.ThisiswhatmostpeoplewillknowNewRelicfor;thistollgivesyouthecodelevelvisibilityofyourapplication.NewRelicMobile:Asetoflibrariestoembedintoyournativemobileapps,givingAPMlevelsofdetailforyouriOSandandroidapplication.NewRelicInsights:Ahigh-levelviewofallofthemetricscollectedbyotherNewRelicservices.NewRelicServers:Monitorsyourhostservers,recordingmetricsaroundCPU,RAM,andstorageutilization.NewRelicBrowser:Givesyouaninsightintowhathappenswithyourweb-basedapplicationsoncetheyleaveyourserversandenteryourenduser’sbrowserNewRelicSynthetics:Monitorsyourapplicationsresponsivenessfromvariouslocationsaroundtheworld.

RatherthanlookingatalloftheseofferingsthatgiveusaninsightintowhatishappeningwithourDocker-basedcode,asthat’sprobablyawholebookonitsown,wearegoingtotakealookattheserverproduct.

TheservermonitoringserviceofferedbyNewRelicisavailablefreeofcharge,youjustneedanactiveNewRelicaccount,youcansignupforanaccountathttps://newrelic.com/signup/detailsonNewRelicspricingcanbefoundattheirhomepageathttp://newrelic.com/.

https://newrelic.com/signup/

http://newrelic.com/

InstallingtheagentLiketheotherSaaSofferingswehavelookedatinthischapter,NewRelicServershasahost-basedclient,whichneedstobeabletoaccesstheDockerbinary.ToinstallthisonaCentOSmachine,runthefollowing:

yuminstallhttp://download.newrelic.com/pub/newrelic/el5/i386/newrelic-

repo-5-3.noarch.rpm

yuminstallnewrelic-sysmond

ForUbuntu,runthefollowingcommand:

echo'debhttp://apt.newrelic.com/debian/newrelicnon-free'|sudotee

/etc/apt/sources.list.d/newrelic.list

wget-O-https://download.newrelic.com/548C16BF.gpg|sudoapt-keyadd-

apt-getupdate

apt-getinstallnewrelic-sysmond

Nowthatyouhavetheagentinstalled,youneedtoconfiguretheagentwithyourlicensekey.Youcandothiswiththefollowingcommandandmakesurethatyouaddyourlicense,whichcanbefoundinyoursettingspage:

nrsysmond-config--setlicense_key=wn5AYlhjRhgn3shcjW14y3yOT09WsF7d

Nowthattheagentisconfigured,weneedtoaddthenewrelicusertothedockergroupsothattheagenthasaccesstoourcontainerinformation:

usermod-a-Gdockernewrelic

Finally,weneedtostarttheNewRelicServeragentandrestartDocker:

/etc/init.d/newrelic-sysmondrestart

/etc/init.d/dockerrestart

TipRestartingDockerwillstoptherunningcontainersthatyouhave;makesurethatyoumakeanoteoftheseusingdockerpsandthenstartthemmanuallyandbackuponcetheDockerservicerestarts.

YoushouldseeyourserverappearonyourNewReliccontrolpanelafterafewminutes.

ExploringthewebinterfaceOnceyouhavetheNewRelicserveragentinstalled,configured,andrunningonyourhostmachine,youwillseesomethingsimilartothefollowingscreenshotwhenclickingonServersinthetopmenu:

Selectingtheserverwillallowyoutostartexploringthevariousmetricsthattheagentisrecording:

Fromhere,youhavetheoptiontodrilldownfurther:

Overview:GivesaquickoverviewofyourhostmachineProcesses:ListsalloftheprocessesthatarerunningbothonthehostmachineandwithinyourcontainersNetwork:LetsyouseethenetworkactivityforyourhostmachineDisks:GivesyoudetailsonhowmuchspaceyouareusingDocker:ShowsyoutheCPUandmemoryutilizationforyourcontainers

Asyoumayhaveguessed,wearegoingtobelookingattheDockeritemnext,clickonitandyouwillseealistofyouractiveimages:

YoumayhavenoticedadifferencebetweenNewRelicandtheotherservices,asyoucanseeNewRelicdoesnotshowyoutherunningcontainers,insteaditshowsyoutheutilizationbyDockerimage.

Intheprecedingscreenshot,IhavefourcontainersactiveandrunningtheWordPressinstallationwehaveusedelsewhereinthebook.IfIwantedabreakdownpercontainer,thenIwouldbeoutofluck,asdemonstratedbythefollowingscreen:

It’saprettydullscreen,butitgivesyouanideaaboutwhatyouwillseeifyouarerunningmultiplecontainersthathavebeenlaunchedusingthesameimage.Sohowisthisuseful?Well,coupledwiththeotherservicesofferedbyNewRelic,itcangiveyouanindicationofwhatyourcontainerswereuptowhenaproblemoccurredwithinyourapplication.IfyourememberthePetsversusCattleversusChickensanalogyfromChapter1,IntroductiontoDockerMonitoring,wedon’tnecessarilycarewhichcontainerdidwhat;wejustwanttoseetheimpactithadduringtheissuewearelookinginto.

SummaryandfurtherreadingDuetotheamountofproductsitoffers,NewReliccanbealittledauntingatfirst,butifyouworkwithadevelopmentteamthatactivelyusesNewRelicwithintheirday-to-dayworkflow,thenhavingalloftheinformationaboutyourinfrastructurealongsidethisdatacanbebothvaluableandnecessary,especiallyduringanissue:

NewRelicServermonitoring:http://newrelic.com/server-monitoringNewRelicandDocker:http://newrelic.com/docker/Twitter:https://twitter.com/NewRelic

TipIfyouhavelaunchedacloudinstanceandarenolongerusingitthen,nowisagoodtimetopowertheinstancedownorterminateitaltogether,thiswillensureyoudonotgetbilledforanyservicesyouarenotusing.

http://newrelic.com/server-monitoring

http://newrelic.com/docker/

https://twitter.com/NewRelic

SummaryWhichSaaSserviceyouchoosedependsonyourcircumstances,thereareanumberofquestionsyoushouldaskyourselfbeforeyoustartevaluatingtheSaaSofferings:

Howmanycontainerswouldyouliketomonitor?Howmanyhostmachinesdoyouhave?Isthereanon-containerizedinfrastructureyouneedtomonitor?Whatmetricsdoyouneedfromthemonitoringservice?Howlongshouldthedataberetainedfor?Couldotherdepartments,suchasdevelopmentandutilizetheservice?

WecoveredjustthreeoftheavailableSaaSoptionsinthischapter,thereareotheroptionsavailable,suchas:

Ruxit:https://ruxit.com/docker-monitoring/Scout:https://scoutapp.com/plugin_urls/19761-docker-monitorLogentries:https://logentries.com/insights/server-monitoring/Sematext:http://sematext.com/spm/integrations/docker-monitoring.html

Monitoringserversandservicesareonlyasgoodasthemetricsyoucollect,ifpossibleandifyourbudgetallows,youshouldtakefulladvantageoftheservicesofferedbyyourchosenproviders,asmoredatabeingrecordedbyasingleproviderwillonlybenefityouwhenitcomestoanalyzingproblemswithnotonlyyourcontainerizedapplications,butalsowithyourinfrastructure,codeandevenyourcloudprovider.

Forexample,ifyouaremonitoringyourhostmachineusingthesameserviceasyouusetomonitoryourcontainers,thenbyusingthecustomgraphingfunctions,youshouldbeabletocreateoverlaygraphsofCPUloadspikesofbothyourhostmachineandyourcontainer.Thisisalotmoreusefulthantryingtocomparetwodifferentgraphsfromdifferentsystemssidebyside.

Inthenextchapter,wewilllookatanoften-overlookedpartofmonitoring:shippingyourlogfilesawayfromyourcontainers/hoststoasinglelocationsothattheycanbemonitoredandreviewed.

https://ruxit.com/docker-monitoring/

https://scoutapp.com/plugin_urls/19761-docker-monitor

https://logentries.com/insights/server-monitoring/

http://sematext.com/spm/integrations/docker-monitoring.html

Chapter7.CollectingApplicationLogsfromwithintheContainerOneofthemostoverlookedpartsofmonitoringarelogfilesgeneratedbytheapplicationorservicessuchasNGINX,MySQL,Apache,andsoon.SofarwehavelookedatvariouswaysofrecordingtheCPUandRAMutilizationoftheprocesseswithinyourcontainersareatapointintime,nowitstimetodothesameforthelogfiles.

IfyouarerunningyourcontainersasCattleorChickens,thenthewayyoudealwiththeissuestodestroyandrelaunchyourcontainereithermanuallyorautomaticallyisimportant.Whilethisshouldfixtheimmediateproblem,itdoesnothelpwithtrackingdowntherootcauseoftheissueandifyoudon’tknowthatthenhowcanyouattempttoresolveitsothatitdoesnotreoccur.

Inthischapter,wewilllookathowwecangetthecontentofthelogfilesfortheapplicationsrunningwithinourcontainerstothecentrallocationsothattheyareavailable,evenifyouhavetodestroyandreplaceacontainer.Wearegoingtocoverthefollowingtopicsinthischapter:

Howtoviewcontainerlogs?Deployingan“ELK”stackusingaDockercontainersstacktoshipthelogstoReviewingyourlogsWhatthirdpartyoptionsareavailable?

ViewingcontainerlogsLikethedockertopcommand,thereisaverybasicwayofviewinglogs.Whenyouusethedockerlogscommand,youareactuallyviewingtheSTDOUTandSTDERRoftheprocessesthatarerunningwithinthecontainer.

NoteFormoreinformationonStandardStreams,pleaseseehttps://en.wikipedia.org/wiki/Standard_streams.

Asyoucanseefromthefollowingscreenshot,thesimplestthingyouhavetodoisrundockerlogsfollowedbyyourcontainername:

Toseethisonyourownhost,let’slaunchtheWordPressinstallationfromchapter05usingthefollowingcommands:

cd/monitoring_docker/chapter05/wordpress/

docker-composeup–d

dockerlogswordpress_wordpress1_1

Youcanextendthedockerlogscommandbyaddingthefollowingflagsbeforeyourcontainername:

-for--followwillstreamthelogsinrealtime-tor--timestampswillshowatimestampatthestartofeachline--tail="5"willshowthelastxnumberoflines--since="5m00s"willshowonlytheentriesforthelast5minutes

UsingtheWordPressinstallationthatwehavejustlaunched,tryrunningthefollowingcommands:

dockerlogs--tail="2"wordpress_wordpress1_1

Thiswillshowthelasttwolinesofthelogs,youcanaddtimestampsusing:

https://en.wikipedia.org/wiki/Standard_streams

dockerlogs--tail="2"–timestampswordpress_wordpress1_1

Asyoucanseeinthefollowingterminaloutput,youcanalsostringcommandstogethertoformaverybasicquerylanguage:

Thedownsideofusingdockerlogsisexactlythesameasusingdockertop,inthatitisonlyavailablelocallyandthelogsareonlypresentforthetimethecontainerisaround,youcanviewthelogsofastoppedcontainer,butoncethecontainerisremoved,soarethelogs.

ELKStackSimilartosomeofthetechnologiesthatwehavecoveredinthisbook,anELKstackreallydeservesabookbyitself;infact,therearebooksforeachoftheelementsthatmakeanELKstack,theseelementsare:

Elasticsearchisapowerfulsearchserver,whichhasbeendevelopedwithmodernworkloadsinmindLogstashsitsbetweenyourdatasourceandElasticsearchservices;ittransformsyourdatainrealtimetoaformat,whichElasticsearchcanunderstand.KibanaisinfrontofyourElasticsearchservicesandallowsyoutoqueryyourdatainafeature-richweb-baseddashboard.

TherearealotofmovingpartswithanELKstack,sotosimplifythings,wewilluseaprebuiltstackforthepurposeoftesting;however,youprobablydon’twanttousethisstackinproduction.

StartingthestackLet’slaunchafreshvagranthostonwhichtoruntheELKstack:







.....







Now,wehaveacleanhostthatisupandrunning,wecanstartthestackbyrunningthefollowingcommands:

[vagrant@docker~]$cd/monitoring_docker/chapter07/elk/

[vagrant@dockerelk]$docker-composeup-d

Asyoumayhavenoticed,itdidmorethatjustpulldownsomeimages;whathappenedwas:

AnElasticsearchcontainerwaslaunchedusingtheofficialimagefromhttps://hub.docker.com/_/elasticsearch/.ALogstashcontainerwaslaunchedusingtheofficialimagefromhttps://hub.docker.com/_/logstash/,itwasalsolaunchedwithourownconfiguration,whichmeansthatourinstallationlistensforlogssentfromLogspout(moreaboutthatinaminute).AcustomKibanaimagewasbuiltusingtheofficialimagefromhttps://hub.docker.com/_/kibana/.AllitdidwasaddasmallscripttoensurethatKibanadoesn’tstartuntilourElasticsearchcontainerisfullyupandrunning.Itwasthenlaunchedwithacustomconfigurationfile.AcustomLogspoutcontainerwasbuiltusingtheofficialimagefromhttps://hub.docker.com/r/gliderlabs/logspout/andthenweaddedacustommodulesothatLogspoutcouldtalktoLogstash.

Oncedocker-composehasfinishedbuildingandlaunchingthestackyoushouldbeabletoseethefollowingwhenrunningdocker-composeps:

https://hub.docker.com/_/elasticsearch/

https://hub.docker.com/_/logstash/

https://hub.docker.com/_/kibana/

https://hub.docker.com/r/gliderlabs/logspout/

WenowhaveourELKstackupandrunning,asyoumayhavenoticed,thereisanadditionalcontainerrunningandgivingusanELK-Lstack,sowhatisLogspout?

LogspoutIfweweretolaunchElasticsearch,Logstash,andKibanacontainers,weshouldhaveafunctioningELKstackbutwewillhavealotofconfigurationtodotogetourcontainerlogsintoElasticsearch.

SinceDocker1.6,youhavebeenabletoconfigureloggingdrivers,thismeantthatitispossibletolaunchacontainerandhaveitsenditsSTDOUTandSTDERRtoaSyslogServer,whichwillbeLogstashinourcase;however,thismeansthatyouwillhavetoaddsomethingsimilartothefollowingoptionseachtimewelaunchacontainer:

--log-driver=syslog--log-optsyslog-address=tcp://elk_logstash_1:5000

ThisiswhereLogspoutcomesin,ithasbeendesignedtocollectalloftheSTDOUTandSTDERRmessagesonahostmachinebyinterceptingthemessagesthatarebeingcollectedbytheDockerprocessandthenitroutesthemtoourLogstashinstanceinaformatthatisunderstoodbyElasticsearch.

Justasthelog-driver,itsupportsSyslogoutofthebox;however,thereisathirdpartymodulethattransformstheoutputtoJSON,whichLogstashunderstands.Asapartofourbuildwedownloaded,compiledandconfiguredthemodule.

YoucanfindoutmoreaboutLogspoutandloggingdriversatthefollowing:

OfficialLogspoutimage:https://hub.docker.com/r/gliderlabs/logspout/LogspoutProjectpage:https://github.com/gliderlabs/logspoutLogspoutLogstashmodule:https://github.com/looplab/logspout-logstashDocker1.6releasenotes:https://blog.docker.com/2015/04/docker-release-1-6/DockerLoggingDrivers:https://docs.docker.com/reference/logging/overview/

https://hub.docker.com/r/gliderlabs/logspout/

https://github.com/gliderlabs/logspout

https://github.com/looplab/logspout-logstash

https://blog.docker.com/2015/04/docker-release-1-6/

https://docs.docker.com/reference/logging/overview/

ReviewingthelogsSonow,wehaveourELKrunningandamechanisminplacetostreamalloftheSTDOUTandSTDERRmessagesgeneratedbyourcontainersintoLogstash,whichinturnroutesthedataintoElasticsearch.NowitstimetoviewthelogsinKibana.ToaccessKibanagotohttp://192.168.33.10:8080/inyourbrowser;whenyouaccessthepage,youwillbeaskedtoConfigureanindexpattern,thedefaultindexpatternwillbefineforourneedssojustclicktheCreatebutton.

Onceyoudo,youwillseealistoftheindexpatterns,thesearetakendirectlyfromtheLogspoutoutput,andyoushouldnoticethefollowingitemsintheindex:

docker.name:Thenameofcontainerdocker.id:ThefullcontainerIDdocker.image:Thenameoftheimageusedtolaunchtheimage

Fromhere,ifyouweretoclickonDiscoverinthetopmenuyouwouldseesomethingsimilartothefollowingpage:

Inthescreenshot,youwillseethatIhaverecentlylaunchedtheWordPressstackandwehavebeenusingitthroughoutthebook,usingthefollowingcommands:

[vagrant@dockerelk]$cd/monitoring_docker/chapter05/wordpress/

[vagrant@dockerwordpress]$docker-composeup–d

Togiveyouanideaofwhatisbeinglogged,hereistherawJSONtakenfromElasticseachforrunningtheWordPressinstallationscript:

{

"_index":"logstash-2015.10.11",

"_type":"logs",

"_id":"AVBW8ewRnBVdqUV1XVOj",

"_score":null,

"_source":{

"message":"172.17.0.11--[11/Oct/2015:12:48:26+0000]\"POST/wp-

admin/install.php?step=1HTTP/1.1\"2002472\"http://192.168.33.10/wp-

admin/install.php\"\"Mozilla/5.0(Macintosh;IntelMacOSX10_11)

AppleWebKit/601.1.56(KHTML,likeGecko)Version/9.0Safari/601.1.56\"",

"docker.name":"/wordpress_wordpress1_1",

"docker.id":

"0ba42876867f738b9da0b9e3adbb1f0f8044b7385ce9b3a8a3b9ec60d9f5436c",

"docker.image":"wordpress",

"docker.hostname":"0ba42876867f",

"@version":"1",

"@timestamp":"2015-10-11T12:48:26.641Z",

"host":"172.17.0.4"

},

"fields":{

"@timestamp":[

1444567706641

]

},

"sort":[

1444567706641

]

}

Fromhere,youcanstarttousethefreetextsearchboxandbuildupsomequitecomplexqueriestodrilldownintoyourcontainer’sSTDOUTandSTDERRlogs.

Whataboutproduction?Asmentionedatthetopofthissection,youprobablydon’twanttorunyourproductionELKstackusingthedocker-composefile,whichaccompaniesthischapter.Firstofall,youwillwantyourElasticsearchdatatobestoredonapersistentvolumeandyoumorethanlikelywantyourLogstashservicetobehighlyavailable.

TherearenumerousguidesonhowtoconfigureahighlyavailableELKstack,aswellas,thehostedservicesfromElastic,whichisthecreatorofElasticsearch,andalsoAmazonWebServices,whichoffersanElasticsearchservice:

ELKtutorial:https://www.youtube.com/watch?v=ge8uHdmtb1MFoundfromElastic:https://www.elastic.co/foundAmazonElasticsearchService:https://aws.amazon.com/elasticsearch-service/

https://www.youtube.com/watch?v=ge8uHdmtb1M

https://www.elastic.co/found

https://aws.amazon.com/elasticsearch-service/

LookingatthirdpartyoptionsThereareafewoptionswhenitcomestohostingcentralloggingforyourcontainersexternaltoyourownserverinstances.Someoftheseare:

LogEntries:https://logentries.com/Loggly:https://www.loggly.com/

Bothoftheseservicesofferafreetier.LogEntriesalsooffersa“LogentriesDockerFree”accountthatyoucanfindoutmoreaboutathttps://logentries.com/docker/

NoteAsrecommendedintheExploringThirdPartyOptionschapter,itisbesttouseacloudservicewhenevaluatingthirdpartyservices.Theremainderofthischapterassumesthatyouarerunningacloudhost.

Let’slookatconfiguringtheLogEntriesonanexternalserver,firstofallyouneedtohavesignedupforanaccountathttps://logentries.com/.Onceyouhavesignedup,youshouldbetakentoapageinwhichyourlogswilleventuallybedisplayed.

Tostart,clickontheAddnewlogbuttoninthetop-rightcornerofthepageandthenclicktheDockerlogointhePlatformssection.

YouhavetonameyoursetoflogsintheSelectsetsection,sogiveanametoyourlogset.YounowhavethechoiceofbuildingyourowncontainerlocallyusingtheDockerfilefromhttps://github.com/logentries/docker-logentries:

gitclonehttps://github.com/logentries/docker-logentries.git

cddocker-logentries

dockerbuild-tdocker-logentries.

Afterrunningtheprecedingcommand,youwillgetthefollowingoutput:

https://logentries.com/

https://www.loggly.com/

https://logentries.com/docker/

https://logentries.com/

https://github.com/logentries/docker-logentries

Beforeyoustartyourcontainer,youwillneedtogenerateanaccesstokenforyourlogsetbyclickingonGenerateLogToken.Onceyouhavethis,youcanlaunchyourlocallybuiltcontainersusingthefollowingcommand(replacethetokenwiththeoneyouhavejustgenerated):

dockerrun-d-v/var/run/docker.sock:/var/run/docker.sockdocker-

logentries-twn5AYlh-jRhgn3shc-jW14y3yO-T09WsF7d-j

YoucandownloadtheimagestraightfromtheDockerhubbyrunning:

dockerrun-d-v/var/run/docker.sock:/var/run/docker.sock

logentries/docker-logentries-twn5AYlh-jRhgn3shc-jW14y3yO-T09WsF7d–j

It’sworthpointingoutthattheautomaticallygeneratedinstructionsgivenbyLogEntrieslaunchesthecontainerintheforeground,ratherthandetachingfromthecontaineronceithasbeenlaunchedliketheprecedinginstructions.

Onceyouhavethedocker-logentriescontainerupandrunning,youshouldstarttoseelogsfromyourcontainerstreamedinreal-timetoyourdashboard:

Fromhere,youwillbeabletoqueryyourlogs,createdashboards,andcreatealertsdependingontheaccountoptionyougofor.

SummaryInthischapter,wehavecoveredhowtoquerytheSTDOUTandSTDERRoutputfromyourcontainersusingthetoolbuiltintoDocker,howtoshipthemessagestoanexternalsource,ourELKstack,andhowtostorethemessagesevenafterthecontainerhasbeenterminated.Finally,wehavelookedatafewofthethird-partyserviceswhoofferservicestowhichyoucanstreamyourlogs.

Sowhygotoallofthiseffort?Monitoringisn’tjustaboutkeepingandqueryingCPU,RAM,HDD,andNetworkutilizationmetrics;thereisnopointinknowingiftherewasaCPUspikeanhouragoifyoudon’thaveaccesstothelogfilestoseeifanyerrorswerebeinggeneratedatthattime.

Theserviceswehavecoveredinthischapterofferthequickestandmostefficientinsightsintowhatcanquicklybecomeacomplexdataset.

Inthenextchapter,wewilllookatalloftheservicesandconceptswehavecoveredinthebookandapplythemtosomerealworldscenarios.

Chapter8.WhatAretheNextSteps?Inthisfinalchapter,wewilllookatthenextstepsyoucantaketomonitoryourcontainers,bytalkingaboutthebenefitsofaddingalertstoyourmonitoring.Also,wewillcoversomedifferentscenariosandalsowhichtypeofmonitoringisappropriateforeachofthem:

Commonproblems(performance,availability,andsoon)andwhichtypeofmonitoringisbestforyoursituation.Whatarethebenefitsofalertingonthemetricsyouarecollectingandwhataretheoptions?

SomescenariosTolookatwhichtypeofmonitoringyoumightwanttoimplementforyourcontainer-basedapplications,weshouldworkthroughafewdifferentexampleconfigurationsthatyourcontainer-basedapplicationscouldbedeployinginto.First,let’sremindourselvesaboutPets,Cattle,Chickens,andSnowflakes.

Pets,Cattle,Chickens,andSnowflakesBackintheChapter1,IntroductiontoDockerMonitoring,wespokeaboutPets,Cattle,Chickens,andSnowflakes;inthatchapter,wedescribedwhateachtermmeantwhenitwasappliedtomodernclouddeployments.Here,wewillgointoalittlemoredetailabouthowthetermscanbeappliedtoyourcontainers.

PetsForyourcontainerstobeconsideredaPet,youwillbemorethanlikelytoberunningeitherasingleorasmallnumberoffixedcontainersonadesignatedhost.

Eachoneofthesecontainerscouldbeconsideredasinglepointoffailure;ifanyoneofthemgoesdown,itwillmorethanlikelyresultinerrorsforyourapplication.Worststill,ifthehostmachinegoesdownforanyreason,yourentireapplicationwillbeoffline.

ThisisatypicaldeploymentmethodformostofourfirststepswithDocker,andinnowayshoulditbeconsideredbad,frownedupon,ornotrecommend;aslongasyouareawareofthelimitations,youwillbefine.

Thispatterncanalsobeusedtodescribemostdevelopmentenvironments,asyouareconstantlyreviewingitshealthandtuningasneeded.

YouwillmorethanlikelybehostingthemachineonyourlocalcomputeroronahostingservicesuchasDigitalOcean(https://www.digitalocean.com/).

CattleForthebulkofproductionorbusinesscriticaldeployments,youshouldaimtolaunchyourcontainersinaconfigurationthatallowsthemtoautomaticallyrecoverthemselvesafterafailure,or,whenmorecapacityisneeded,additionalcontainersarelaunchedandthenterminatedwhenthescalingeventisover.

Youwillmorethanlikelybeusingapubliccloud-basedserviceasfollows:

AmazonEC2ContainerService:https://aws.amazon.com/ecs/GoogleContainerEngine:https://cloud.google.com/container-engine/JoyentTriton:https://www.joyent.com/blog/understanding-triton-containers/

Alternatively,youwillbehostingonyourownserversusingaDocker-friendlyandcluster-awareoperatingsystemasfollows:

CoreOS:https://coreos.com/RancherOS:http://rancher.com/rancher-os/

Youwon’tcaresomuchastowhereacontainerislaunchedwithinyourclusterofhosts,aslongasyoucanroutetraffictoit.Toaddmorecapacitytothecluster,youwillbebringingupadditionalhostswhenneededandremovingthemfromtheclusterwhennotneededinordertosaveoncosts.

Chickens

https://www.digitalocean.com/

https://aws.amazon.com/ecs/

https://cloud.google.com/container-engine/

https://www.joyent.com/blog/understanding-triton-containers/

https://coreos.com/

http://rancher.com/rancher-os/

Itsmorethanlikelyyouwillbeusingcontainerstolaunch,processdata,andthenterminate.Thiscanhappenanytimefromonceadaytoseveraltimesaminute.Youwillbeusingadistributedschedulerasfollows:

KubernetesbyGoogle:http://kubernetes.io/ApacheMesos:http://mesos.apache.org/

Becauseofthis,youwillhavealargenumberofcontainerslaunchingandterminatingwithinyourcluster;youdefinitelywon’tcareaboutwhereacontainerislaunchedorevenhowtrafficisroutedtoit,aslongasyourdataisprocessedcorrectlyandpassedbacktoyourapplication.

LiketheclusterdescribedintheCattlesection’sdescription,hostswillbeaddedandremovedautomatically,probablyinresponsetoscheduledpeakssuchasendofmonthreportingorseasonalsalesandsoon.

SnowflakesIhopeoneofthethingsyoutookawayfromthefirstchapteristhatifyouhaveanyserversorservicesthatyouconsiderbeingSnowflakes,thenyoushoulddosomethingtoretirethemassoonaspossible.

Luckily,duetothewaythecontainerizingofyourapplicationsworks,youshouldneverbeabletocreateasnowflakeusingDocker,asyourcontainerizedenvironmentshouldalwaysbereproducible,eitherbecauseyouhavetheDockerfile(everyonemakesbackupsright?)oryouhaveaworkingcopyofthecontainerimagebecauseyouhaveexportedthecontainerasawholeusingthebuilt-intools.

NoteSometimesitmaynotbepossibletocreateacontainerusingaDockerfile.Instead,youcanbackupormigrateyourcontainersbyusingtheexportcommand.Formoreinformationonexportingyourcontainers,seethefollowingURL:

https://docs.docker.com/reference/commandline/export/

Ifyoufindyourselfinthisposition,letmebethefirsttocongratulateyouonmitigatingafuturedisasterbypromotingyourSnowflakeintoaPetorevenCattleaheadofanyproblems.

TipStillrunningaSnowflake?

IfyoufindyourselfstillrunningaSnowflakeserverorservice,Icannotstressenoughthatyoulookatdocumenting,migrating,orupdatingtheSnowflakeassoonaspossible.Thereisnopointinmonitoringaservicethatmaybeimpossibleforyoutorecover.Rememberthattherearecontainersforoldtechnologies,suchasPHP4,ifyoureallyneedtorunthem.

http://kubernetes.io/

http://mesos.apache.org/

https://docs.docker.com/reference/commandline/export/

ScenariooneYouarerunningapersonalWordPresswebsiteusingtheofficialcontainersfromtheDockerHub;thecontainershavebeenlaunchedusingaDockerComposefileliketheonewehaveusedseveraltimesthroughoutthisbook.

YouhavetheDockerComposefilestoredinaGitHubrepositoryandyoucantakesnapshotsofthehostmachineasabackup.Asit’syourownblog,youarefinerunningitonasinglecloud-basedhost.

Asuitablemonitoringwillbeasfollows:

DockerstatsDockertopDockerlogscAdvisorSysdig

Asyouarerunningasinglehostmachinethatyouaretreatingasabackup,thereisnorealneedforyoutoshipyourlogfilestoacentrallocationasoddsareyourhostmachines;likethecontainers,itshostingwillbeonlineformonthsorpossiblyevenyears.

Itisunlikelythatyouwillneedtodigtoodeeplyintoyourcontainers’historicalperformancestats,asmostofthetuningandtroubleshootingwillbedoneinrealtimeasproblemsoccur.

Withthemonitoringtoolssuggested,youwillbeabletogetagoodinsightintowhatishappeningwithinyourcontainersinrealtime,andtogetmorethanenoughinformationonprocessesthatareconsumingtoomuchRAMandCPU,alongwithanyerrormessagesfromwithinthecontainers.

YoumaywanttoenableaservicesuchasPingdom(https://www.pingdom.com/)orUptimeRobot(http://uptimerobot.com/).TheseservicespollyourwebsiteeveryfewminutestoensurethattheURLyouconfigurethemto,checkwhetheritsloadingwithinacertaintimeoratall.Iftheydetectanyslowdownorfailureswiththepageloading,theycanbeconfiguredtosendaninitialalerttonotifyyouthatthereisapotentialissue,suchasboththeservicesmentionedhaveafreetier.

https://www.pingdom.com/

http://uptimerobot.com/

ScenariotwoYouarerunningacustome-commerceapplicationthatneedstobehighlyavailableandalsoscaleduringyourpeaktimes.Youareusingapubliccloudserviceandthetoolsetthatcomeswithittolaunchcontainersandroutetraffictothem.


cAdvisor+PrometheusZabbixSysdigCloudNewRelicServerMonitoringDatadogELK+LogspoutLogEntriesLoggly

Withthisscenario,thereisabusinessneedtonotonlybenotifiedaboutcontainerandhostfailures,butalsotoholdyourmonitoringdataandlogsawayfromyourhostserverssothatyoucanproperlyreviewhistoricalinformation.YoumayalsoneedtokeeplogsforPCIcomplianceorinternalauditingforafixedperiodoftime.

Dependingonyourbudget,youcanachievethisbyhostingyourownmonitoring(ZabbixandPrometheus)andcentrallogging(ELK)stackssomewherewithinyourinfrastructure.

Youcanalsochoosetorunafewdifferentthird-partytoolssuchascombiningtoolsthatmonitorperformance,forexample,SysdigCloudorDatadog,withacentralloggingservice,suchasLogEntriesorLoggly.

Ifappropriate,youcanalsorunacombinationofself-hostedandthird-partytools.

Whiletheself-hostedoptionmayappeartobethemostbudget-friendlyoption,therearesomeconsiderationstotakeintoaccount,asfollows:

Yourmonitoringneedstobehostedawayfromyourapplication.Thereisnopointinhavingyourmonitoringinstalledonthesamehostasyourapplication;whatwillalertyouifthehostfails?Yourmonitoringneedstobehighlyavailable;doyouhavetheinfrastructuretodothis?Ifyourapplicationneedstobehighlyavailable,thensodoesyourmonitoring.Youneedtohaveenoughcapacity.Doyouhavethecapacitytobeabletostorelogfilesandmetricsgoingbackamonth,6months,orayear?

Ifyouaregoingtohavetoinvestinanyoftheprecedingoptions,thenitwillbeworthweighingupthecostsofinvestinginboththeinfrastructureandthemanagementofyourownmonitoringsolutionagainstusingathird-partythatwilloffertheprecedingoptionsasaservice.

Ifyouareusingacontainer-onlyoperatingsystemsuchasCoreOSorRancherOS,thenyouwillneedtochooseaservicewhoseagentorcollectorcanbeexecutedfromwithinacontainer,asyouwillnotbeabletoinstalltheagentbinariesdirectlyontheOS.

Youwillalsoneedtoensurethatyourhostmachineisconfiguredtostarttheagents/collectorsonboot.Thiswillensurethatassoonasthehostmachinejoinsacluster(whichistypicallywhencontainerswillstarttopopuponthehost),itisalreadysendingmetricstoyourchosenmonitoringservices.

ScenariothreeYourapplicationlaunchesacontainereachtimeyourAPIiscalledfromyourfrontendapplication;thecontainertakestheuserinputfromadatabase,processesit,andthenpassestheresultsbacktoyourfrontendapplication.Oncethedatahasbeensuccessfullyprocessed,thecontaineristerminated.Youareusingadistributedschedulingsystemtolaunchthecontainers.


ZabbixSysdigCloudDatadogELK+LogspoutLogEntriesLoggly

Inthisscenario,youmorethanlikelydonotwanttomonitorthingssuchasCPUandRAMutilization.Thesecontainersafterallshouldonlybearoundforafewminutes,andalsoyourschedulerwilllaunchthecontaineronthehostmachinewherethereisenoughcapacityforthetasktoexecute.

Instead,youwillprobablywanttokeeparecordtoverifythatthecontainerlaunchedandterminatedasexpected.YouwillalsowanttomakesurethatyoulogtheSTDOUTandSTDERRfromthecontainerwhileitisactive,asoncethecontainerhasbeenterminated,itwillbeimpossibleforyoutogetthesemessagesback.

Withthetoolslistedintheprecedingpoints,youshouldbeabletobuildsomequiteusefulqueriestogetadetailedinsightintohowyourshortrunprocessesareperforming.

Forexample,youwillbeabletogettheaveragelifetimeofacontainer,asyouknowthetimethecontainerwaslaunchedandwhenitwasterminated;knowingthiswillthenallowyoutosetatriggertoalertyouifanycontainersarearoundforanylongerthanyouwouldexpectthemtobe.

AlittlemoreaboutalertingAlotofthetoolswehavelookedatinthisbookofferatleastsomesortofbasicalertingfunctionality;themillion-dollarquestionisshouldyouenableit?

Alotofthisisdependentonthetypeofapplicationyouarerunningandhowthecontainershavebeendeployed.Aswehavealreadymentionedafewtimesinthischapter,youshouldneverreallyhaveaSnowflakecontainer;thisleavesuswithPets,Cattle,andChickens.

ChickensAsalreadydiscussedintheprevioussection,youprobablydon’tneedtoworryaboutgettingalertsforRAM,CPU,andharddriveperformanceonaclusterthatisconfiguredtorunChickens.

Yourcontainersshouldnotbeuplongenoughtoexperienceanyrealproblems;however,shouldtherebeanyunexpectedspikes,yourschedulerwillprobablyhaveenoughintelligencetodistributeyourcontainerstohoststhathavethemostavailableresourcesatthattime.

Youwillneedtoknowifanyofyourcontainershavebeenrunninglongerthanyouexpectthemtobeup;forexample,aprocessinacontainerthatnormallytakesnomorethan60secondsisstillrunningafter5minutes.

Thisnotonlymeansthatthereisapotentialproblem,italsomeansthatyoufindyourselfrunninghoststhatonlycontainstalecontainers.

CattleandPetsWhenitcomestosettingupalertsonCattleorPets,youhaveafewoptions.

YouwillmorethanlikelywanttoreceivealertsbasedonCPUandRAMutilizationforboththehostmachineandthecontainers,asthiscouldindicateapotentialproblemthatcouldcauseslowdownwithintheapplicationandalsolossofbusiness.

Asmentionedpreviously,youwillprobablyalsowanttobealertedifyourapplicationstartstoservethecontentthatisunexpected.Forexample,ahostandacontainerwillquitehappilysitthereservinganapplicationerror.

YoucanuseaservicesuchasPingdom,Zabbix,orNewRelictoloadapageandcheckforthecontentinthefooter;ifthiscontentismissing,thenanalertcanbesent.

Dependingonhowfluidyourinfrastructureis,inaCattleconfiguration,youwillprobablywanttobealertedwhencontainersspinupanddown,asthiswillindicateperiodsofhightraffic/transactions.

SendingalertsSendingalertsdiffersforeachtool,forexample,analertcouldbeassimpleassendinganemailtoinformyouthatthereisanissuetothesoundingofanaudiblealertinaNetworkOperationsCenter(NOC)whentheCPUloadofacontainergoesabovefive,ortheloadonthehostgoesabove10.

Forthoseofyouwhorequireanon-callteamtobealerted,mostofthesoftwarewehavecoveredhassomelevelofintegrationalertaggregationservicessuchasPagerDuty(https://www.pagerduty.com).

TheseaggregationserviceseitherinterceptyouralertemailsorallowservicestomakeAPIcallstothem.Whentriggered,theycanbeconfiguredtoplacephonecalls,sendSMSmessages,andevenescalatetosecondaryon-calltechnicianifanalerthasnotbeenflaggeddownwithinadefinabletime.

Ican’tthinkofanycaseswhereyoushouldn’tlookatenablingalerting,afterall,it’salwaysbesttoknowaboutanythingthatcouldeffectyourapplicationbeforeyourendusersdo.

Howmuchalertingyouenableisreallydowntowhatyouareusingyourcontainersfor;however,Iwouldrecommendthatyoureviewallyouralertsregularlyandalsoactivelytuneyourconfiguration.

Thelastthingyouwantisaconfigurationthatproducestoomanyfalsepositivesoronethatistootwitchy,asyoudonotwanttheteamwhoreceivesyouralertstobecomedesensitizedtothealertsthatyouaregenerating.

Forexample,ifacriticalCPUalertistriggeredevery30minutesbecauseofascheduledjob,thenyouwillprobablyneedtoreviewthesensitivityofthealert,otherwiseitiseasyfortheengineertosimplydismissacriticalalertwithoutthinkingaboutit,as“thisalertcomeseveryhalfanhourandwillbeokinafewminutes”,whenyourentireapplicationcouldbeunresponsive.

https://www.pagerduty.com

KeepingupWhileDockerhasbeenbuiltontopofwell-establishedtechnologiessuchasLinuxContainers(LXC),thesehavetraditionallybeendifficulttoconfigureandmanage,especiallyfornon-systemadministrators.

Dockerremovesalmostallthebarrierstoentry,allowingeveryonewithasmallamountofcommand-lineexperiencetolaunchandmanagetheirowncontainer-basedapplications.

Thishasforcedalotofthesupportingtoolstoalsolowertheirbarriertoentry.Softwarethatoncerequiredcarefulplanningtodeploy,suchassomeofthemonitoringtoolswecoveredinthisbook,cannowbedeployedandconfiguredinminutesratherthanhours.

Dockerisalsoaveryfast-movingtechnology;whileithasbeenconsideredproduction-readyforawhile,newfeaturesarebeingaddedandexistingfeaturesareimprovedwithregularupdates.

Sofar,in2015,therehavebeen11releasesofDockerEngine;ofthese,onlysixhavebeenminorupdatesthatfixbugs,andtheresthaveallbeenmajorupdates.Detailsofeachreleasecanbefoundintheproject’sChangelog,whichcanbefoundathttps://github.com/docker/docker/blob/master/CHANGELOG.md.

BecauseofthepaceofdevelopmentofDocker,itisimportthatyoualsoupdateanymonitoringtoolsyoudeploy.Thisisnotonlytokeepupwithnewfeatures,butalsotoensurethatyoudon’tlooseanyfunctionalityduetochangesinthewayinwhichDockerworks.

Thisattitudeofupdatingmonitoringclients/toolscanbeabitofachangeforsomeadministratorswhomaybeinthepastwouldhaveconfiguredamonitoringagentonaserverandthennotthoughtaboutitagain.

https://github.com/docker/docker/blob/master/CHANGELOG.md

SummaryAsdiscussedinthischapter,Dockerisafastmovingtechnology.Whilethisbookhasbeeninproduction,therehavebeenthreemajorversionsreleasedfrom1.7to1.9;witheachreleaseDockerhasbecomemorestableandmorepowerful.

Inthischapter,wehavelookedatdifferentwaystoimplementthetechnologiesthathavebeendiscussedinthepreviouschaptersofthisbook.Bynow,youshouldhaveanideaofwhichapproachisappropriatetomonitoryourcontainersandhostmachines,forbothyourapplicationandforthewaytheapplicationhasbeendeployedusingDocker.

Nomatterwhichapproachyouchosetotake,itisimportantthatyoustayup-to-datewithDocker’sdevelopmentandalsothenewmonitoringtechnologiesastheyemerge,thefollowinglinksaregoodstartingpointstokeepyourselfinformed:

DockerEngineeringBlog:http://blog.docker.com/category/engineering/DockeronTwitter:https://twitter.com/dockerDockeronReddit:https://www.reddit.com/r/dockerDockeronStackOverflow:http://stackoverflow.com/questions/tagged/docker

OneofthereasonswhytheDockerprojecthasbeenembracedbydevelopers,systemadministratorsandevenenterprisecompaniesisbecauseitisabletomoveataquickpace,whileaddingmorefeaturesandveryimpressivelymaintainingitseaseofuseandflexibility.

Overthenext12months,thetechnologyissettobeevenmorewidespread;theimportanceofensuringthatyouarecapturingusefulperformancemetricsandlogsfromyourcontainerswillbecomemorecriticalandIhopethatthisbookhashelpedyoustartyourjourneyintomonitoringDocker.

http://blog.docker.com/category/engineering/

https://twitter.com/docker

https://www.reddit.com/r/docker

http://stackoverflow.com/questions/tagged/docker

IndexA

alertsenabling/Alittlemoreaboutalertingsettingup,onChickens/Chickenssettingup,onCattle/CattleandPetssettingup,onPets/CattleandPetssending/Sendingalerts

AmazonEC2ContainerServiceURL/Cattle

AmazonWebServicesURL/Awordaboutexternallyhostedservices

ApacheBenchURL/RunningDockerstats

ApacheMesosURL/Chickens

AtomicURL/WhyuseaSaaSservice?

Bbackslash

using/RunningcAdvisorusingacontainer

CcAdvisor

about/WhatiscAdvisor?executing,containerused/RunningcAdvisorusingacontainercompiling,fromsource/CompilingcAdvisorfromsourcereferencelink/Alternatives?

Cattleabout/Cattlecontainers,deployingonto/Cattlealerts,settingup/CattleandPets

Changelog,DockerURL/Keepingup

Chickensabout/Chickenscontainers,deployingonto/Chickensalerts,settingup/Chickens

cloudDocker,deployingin/DeployingDockerinthecloud

CloudComputingreferencelink/Chickens

Cloudscalingreferencelink/Pets,Cattle,Chickens,andSnowflakes

containerused,forexecutingcAdvisor/RunningcAdvisorusingacontainer

ContainerDownabout/Triggers

containerlogsviewing/Viewingcontainerlogs

containersresourceutilization,tracking/Whatjusthappened?comparing,tohostmachine/Comparecontainerstoyourhostmachinedeploying,ontoPets/Petsdeploying,ontoCattle/Cattledeploying,ontoChickens/Chickensdeploying,ontoSnowflakes/Snowflakes

containers,monitoringscenarios/Scenarioone,Scenariotwo,Scenariothree

containerstatsviewing/Viewingcontainerstatssubcontainers/Subcontainersdriverstatus/Driverstatusimages/Images

CoreOS

URL/WhyuseaSaaSservice?,CattleCsysdig

about/UsingCsysdigusing/UsingCsysdig

customgraphscreating/Createcustomgraphsreferencelink/Createcustomgraphs

DDashboard

launching/DashboardDatadog

URL/WhyuseaSaaSservice?,Datadogabout/Datadogagent,installing/Installingtheagentagentinstallation,URL/Installingtheagentwebinterface,exploring/Exploringthewebinterfacereferences/Summaryandfurtherreading

datavolumecontainercreating/Thenextsteps

DigitalOceanURL/Pets

DigitalOceanURL/Awordaboutexternallyhostedservices

Dockerabout/DockerURL,foruserguide/Runningavirtualserver,Thenextstepsdeploying,incloud/DeployingDockerinthecloudadvancements/Keepingup

docker-composefileused,forexecutingELKstack/Whataboutproduction?

DockerComposeabout/Docker

DockerEngineabout/Docker

Dockerexecabout/Dockerexec

dockerexportcommandURL/Snowflakes

DockerHubabout/DockerURL/Docker,WhatiscAdvisor?,Usingcontainers

DockerimagesURL/Driverstatus

Dockerstatsabout/Dockerstatsexecuting/RunningDockerstats

Dockertopabout/Dockertop

driverstatusobtaining/Driverstatus

EElasticsearch

URL,forlaunching/StartingthestackELKstack

about/ELKStackstarting/StartingthestackLogspout/Logspoutlogs,reviewing/Reviewingthelogsexecuting,docker-composefileused/Whataboutproduction?references/Whataboutproduction?

environmentcloning/Cloningtheenvironment

externallyhostedservicesabout/AwordaboutexternallyhostedservicesDocker,deployingincloud/DeployingDockerinthecloud

FFig

about/Docker

GGo

URL/WhatiscAdvisor?GoogleContainerEngine

URL/Cattle

Hhostmachine

containers,comparingto/Comparecontainerstoyourhostmachine

Iimages

listing/ImagesInfluxDB

about/Alternatives?URL/Alternatives?

JJoyentTriton

URL/Cattle

KKibana

URL,forlaunching/StartingthestackKubernetes

URL/Chickens

LLiquidWaxEster(LXE)

about/Keepinguplmctfy

URL/WhatiscAdvisor?localenvironment

launching/LaunchingalocalenvironmentLogEntries

URL/Lookingatthirdpartyoptionsconfiguring/Lookingatthirdpartyoptions

LogglyURL/Lookingatthirdpartyoptions

logsreviewing/Reviewingthelogs

LogspoutURL,forlaunching/Startingthestackabout/Logspoutreferences/Logspout

LogstashURL,forlaunching/Startingthestack

MMariaDB10

URL/Usingcontainersmetrics

collecting/Collectingmetricsrecording/Dockermetricscustomgraphs,creating/Createcustomgraphscontainers,comparingtohostmachine/Comparecontainerstoyourhostmachinetriggers,defining/Triggers

MicrosoftAzureURL/Awordaboutexternallyhostedservices

MonitoringDockerrepositoryURL/Cloningtheenvironment

NNagios

URL/Zabbixncurses

about/WhatisSysdig?NetworkOperationsCenter(NOC)

about/SendingalertsNewRelic

URL/WhyuseaSaaSservice?,NewRelicabout/NewRelicagent,installing/Installingtheagentwebinterface,exploring/Exploringthewebinterfacereferences/Summaryandfurtherreading

P—privilegedflag

about/RunningcAdvisorusingacontainerURL/RunningcAdvisorusingacontainer

PagerDutyURL/Triggers,Sendingalerts

Petsabout/Petscontainers,deployingonto/Petsalerts,settingup/CattleandPets

phpinfoURL/RunningDockerstats

PingdomURL/Scenarioone

processesisolating/Whataboutprocesses?Dockertop/DockertopDockerexec/Dockerexec

Prometheusabout/Prometheuslaunching/LaunchingPrometheusquerying/QueryingPrometheusDashboard,launching/DashboardURL,fordocumentation/Dashboarddatavolumecontainer,creating/Thenextstepsreferences/Thenextsteps

PuppetForgeURL/Usingvagrant

RRancherOS

URL/WhyuseaSaaSservice?,Cattle

SSaaSservice

using/WhyuseaSaaSservice?serverstyles

selecting/Sowhatdoesthisallmean?Slack

URL/TriggersSnowflakes

about/Snowflakesreferencelink/Snowflakescontainers,deployingonto/Snowflakes

StandardStreamsreferencelink/Viewingcontainerlogs

subcontainersabout/Subcontainers

SysAdminabout/Pets,Cattle,Chickens,andSnowflakes

Sysdigabout/WhatisSysdig?installing/InstallingSysdigURL/InstallingSysdigusing/UsingSysdigbasics/Thebasicsdata,capturing/Capturingdatacontainers,listing/Containersreferences/Furtherreading

SysDigCloudURL/WhyuseaSaaSservice?

SysdigCloudabout/SysdigCloudfunctionality/SysdigCloudreferences/SysdigCloud,Summaryandfurtherreadingagent,installing/Installingtheagentagentinstallation,URL/Installingtheagentcontainers,exploring/Exploringyourcontainers

Ttimeseries

about/Prometheusreferencelink/Prometheus

triggersdefining/Triggers

UUbuntuSnappyCore

URL/WhyuseaSaaSservice?UptimeRobot

URL/Scenarioone

VVagrant

URL/Launchingalocalenvironmentvagrantbox

URL/RunningavirtualserverVirtualBox

URL/Launchingalocalenvironmentvirtualserver

executing/Runningavirtualserverhalting/Haltingthevirtualserver

VMwarevCloudAirURL/Awordaboutexternallyhostedservices

WWebinterface

about/TheWebinterfaceoverview/OverviewProcesses/ProcessesCPU/CPUMemory/MemoryNetwork/NetworkFilesystem/Filesystemdrawbacks/Thisisallgreat,what’sthecatch?

WordPressURL,forinstallation/UsingSysdig

YYAML

URL/Docker

ZZabbix

about/ZabbixURL/Zabbix,Usingvagrant,TheZabbixwebinterfacereferences/Zabbixinstalling/InstallingZabbixcontainers,using/Usingcontainersvagrant,using/Usingvagranthostmachine,preparing/Preparingourhostmachinehosttemplates,adding/TheZabbixwebinterface

ZenossURL/Zabbix