Top Banner
309

Implementing DevOps on AWS - pdf.ebook777.compdf.ebook777.com/058/1786460149.pdf · except in the case of brief quotations embedded in critical ... Content Development Editor Abhishek

Jul 18, 2018

Download

Documents

hoangcong
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript

  • ImplementingDevOpsonAWS

  • TableofContents

    ImplementingDevOpsonAWSCreditsAbouttheAuthorAbouttheReviewerwww.PacktPub.com

    Whysubscribe?CustomerFeedbackPreface

    WhatthisbookcoversWhatyouneedforthisbookWhothisbookisforConventionsReaderfeedbackCustomersupport

    DownloadingtheexamplecodeDownloadingthecolorimagesofthisbookErrataPiracyQuestions

    1.WhatisDevOpsandShouldYouCare?WhatisDevOps?

    AcommongoalSharedknowledge(nosilos)TrustandsharedresponsibilityRespectAutomationReproducibleinfrastructureMetricsandmonitoringContinuousIntegration,Delivery,andDeploymentEmbracingfailure

    ShouldyoucareIsthistherighttime?Willitwork?Isitworthit?Doyouneedit?

    Summary2.StartTreatingYourInfrastructureasCode

    IaCusingTerraformConfigurationTemplatedesign

    Resources

  • VariablesOutputs

    OperationsValidationDry-runDeploymentUpdatesRemoval

    IaCusingCloudFormationConfigurationTemplatedesign

    ParametersResourcesOutputs

    OperationsTemplatevalidationDeployingaStackUpdatingastackDeletingastack

    Summary3.BringingYourInfrastructureUnderConfigurationManagement

    IntroductiontoSaltStackPreparation

    WritingConfigurationManagementcodeStatesPillarsGrainsTopfiles

    BootstrappingnodesunderConfigurationManagement(end-to-endIaC)Summary

    4.Build,Test,andReleaseFasterwithContinuousIntegrationPrepareIaC

    TerraformtemplatesVariablesVariables(values)Resources

    CreatetheVPCAddnetworkingcomponentsAddEC2nodeandrelatedresources

    OutputsSaltStackcode

    StatesPillarsMinionconfiguration

  • DeployIaCSetupCI

    JenkinsinitializationWritingademoappDefiningthepipelineSettingupthepipeline

    Summary5.Ever-ReadytoDeployUsingContinuousDelivery

    PreparingTerraformtemplatesResourcesVariablesVariables(values)Outputs

    PrepareingSaltcodeStates

    top.slsjenkinsnginx

    PillarsMinionconfiguration

    PreparingJenkinscodePacker

    demo-app.jsondemo-app_vars.jsondemo-app_userdata.sh

    Serverspecspec/localhost/demo-app_spec.rb

    demo-appdemo-app-cdelivery

    PreparingCodeCommitrepositoriesDeployTerraformtemplatesInitializingJenkinsConfiguringJenkinsjobs

    demo-apppipelinedemo-app-cdeliverypipeline

    Summary6.ContinuousDeployment-AFullyAutomatedWorkflow

    Terraformcode(resources.tf)outputs.tf

    DeploymentJenkinspipelines

    ContinuousDeploymentpipelinecdeployment.sh

    Summary

  • 7.Metrics,LogCollection,andMonitoringCentralizedlogging

    IngestingandstoringlogswithLogstashandElasticsearchCollectinglogswithElasticsearchFilebeatVisualizinglogswithKibana

    MetricsIngestingandstoringmetricswithPrometheusGatheringOSandapplicationmetricswithTelegrafVisualizingmetricswithGrafana

    MonitoringAlertingwithPrometheusSelf-remediationwithPrometheusandJenkins

    Summary8.OptimizeforScaleandCost

    ArchitecturalconsiderationsTheVPC

    CIDRSubnetsandAvailabilityZonesVPClimits

    ThefrontendlayerThebackendlayerTheobjectstoragelayer

    TheloadbalancinglayerCross-zoneloadbalancingELBpre-warmingTheCDNlayerSpotinstancesAWSCalculators

    Summary9.SecureYourAWSEnvironment

    ManagingaccessusingIAMSecuringtherootaccount

    VPCsecuritySecurityGroupsNetworkACLsVPNgatewayVPCpeering

    EC2securityIAMRolesSSHaccess

    IndividualkeysEntrypoint

    ELBseverywhereHTTPSbydefault

  • EncryptedstorageEBSvolumesS3objects

    OSupdatesSecurityauditing

    VPCFlowLogsCloudTrailTrustedAdvisorAWSConfigSelfpentesting

    Summary10.AWSTipsandTricks

    UsingVPCsKeeptheMainroutetableasafallbackStayingwithintheVPCCreatingIAMrolesinadvanceGroupsoverusersKnowingtheAWSservicelimitsPre-warmELBsifneededUsingterminationprotectionTaggingwhatyoucanDeployingacrossmultiplezonesEnhancingyourELBhealth-checksOffloadingSSLontotheELBEIPversuspublicIPMindthefull-hourbillingUsingRoute53ALIASrecordsTheS3bucketnamespaceisglobal-versus.intheS3bucketnameRandomizingS3filenamesInitializing(pre-warm)EBSvolumesSummary

  • ImplementingDevOpsonAWS

  • ImplementingDevOpsonAWSCopyright2017PacktPublishing

    Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

    Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

    PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

    Firstpublished:January2017

    Productionreference:1190117

    PublishedbyPacktPublishingLtd.

    LiveryPlace

    35LiveryStreet

    Birmingham

    B32PB,UK.

    ISBN978-1-78646-014-1

    www.packtpub.com

    http://www.packtpub.com

  • Credits

    Author

    VeselinKantsev

    CopyEditor

    SafisEditing

    Reviewer

    MadhuJoshi

    ProjectCoordinator

    JudieJose

    CommissioningEditor

    KartikeyPandey

    Proofreader

    SafisEditing

    AcquisitionEditor

    NamrataPatil

    Indexer

    PratikShirodkar

    ContentDevelopmentEditor

    AbhishekJadhav

    Graphics

    KirkD'Penha

    TechnicalEditor

    MohdRiyanKhan

    ProductionCoordinator

    NileshMohite

  • AbouttheAuthorVeselinKantsevisaDevOpsprofessionalandaLinuxenthusiastwholivesinLondon,UK.

    HisintroductiontoLinuxwasasaSystemAdministratorbackin2006.HisfocusforthepastfewyearshasbeenmostlyoncloudtechnologiesandthetransitionofthecommunityfromanOpstoaDevOpsculture.

    HehasworkedwithcompaniesinvarioussectorssuchasDesign,Media,andFinance,specializinginthemigrationofinfrastructureontoAWSandthepromotionofDevOpsprinciplesandpractices.

  • AbouttheReviewerMadhuJoshi,theVPofCloudServicesatTrillionTechnologySolutions,hasbuilthigh-profilewebsitesinAWSforDoD,Verato,NationalGeospatialAgency(NGA),DHSFEMA,ProQuestSearchSolutions,TheCocaColaCompany,FoodNetwork,Scripps,SpecialOlympics,HomeandGardenTV,andMonumentalSports.Hedesignedthesesitestohandlemillionsofvisitorspermonthandhasusedindustrybestpracticestoprovidepeakcapacityrequiredtoservetheglobalaudience.HealsohasexpertiseinmigratinglargeapplicationssuchastheKickAppssocialmediaplatformandfinancialservicesfromatraditionaldatacentertoAmazon'scloud.

    IwouldliketothankPacktPublishingandJudieJoseforgivingmetheopportunitytoreviewthebook.Thishasturnedouttobeagreatbook,andIwanttocongratulatetheauthoronputtingonlotofhardworktomakethebookveryuseful.IamsurethatthereaderswillhaveasmuchfunasIhadreviewingit.

  • www.PacktPub.comForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.

    DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusatservice@packtpub.comformoredetails.

    Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

    https://www.packtpub.com/mapt

    Getthemostin-demandsoftwareskillswithMapt.MaptgivesyoufullaccesstoallPacktbooksandvideocourses,aswellasindustry-leadingtoolstohelpyouplanyourpersonaldevelopmentandadvanceyourcareer.

    http://www.PacktPub.comhttp://www.PacktPub.comhttp://www.PacktPub.comhttps://www.packtpub.com/mapt

  • Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser

  • CustomerFeedbackThankyouforpurchasingthisPacktbook.Wetakeourcommitmenttoimprovingourcontentandproductstomeetyourneedsseriouslythat'swhyyourfeedbackissovaluable.Whateveryourfeelingsaboutyourpurchase,pleaseconsiderleavingareviewonthisbook'sAmazonpage.Notonlywillthishelpus,moreimportantlyitwillalsohelpothersinthecommunitytomakeaninformeddecisionabouttheresourcesthattheyinvestintolearn.

    Youcanalsoreviewforusonaregularbasisbyjoiningourreviewers'club.Ifyou'reinterestedinjoining,orwouldliketolearnmoreaboutthebenefitsweoffer,pleasecontactus:[email protected].

  • PrefaceDevOpsandAWSaretwokeysubjectsinthetechindustrythathavebeensteadilygrowinginpopularityinrecentyearsandforagoodreason.

    DevOpsisgraduallybecomingthedefactomethodologyorframeworkandisadoptedbyorganizationsofallsizes.Ithasenabledtechnologyteamstoworkmoreefficientlyandmadetheirworkmorerewardingbytighteningthefeedbackloopbetweenthedeveloperandtheenduser.Teammembersenjoyamorepleasant,moreproductiveworkenvironmentthroughmuchincreasedcollaboration.

    Inthisbook,wewillfirstexaminethephilosophybehindDevOps,thenproceedwithsomepracticalexamplesofitsmostpopularprinciples.

    AWSisnowadayssynonymouswithCloudComputing,sittingatthetopoftheindustrychartswithits31%marketshare.Startingbackin2006,AmazonWebServiceshasevolvedintoalarge,independent,sophisticatedecosystemintheCloud.Itisandhasbeenlaunchingnewservicesatanastonishingrate.TheAWSproductcategoriesrangefromrawcomputeanddatabaseresourcestostorage,analytics,AI,gamedevelopment,andmobileservicestoIoTsolutions.

    WewilluseAWSasaplatformtoapplyDevOpstechniqueson.Inthechapterstofollow,wewillseehowtheconvenienceandelasticityofAWSgreatlycomplementstheinnovativeapproachofDevOpstosystemadministrationandapplicationdevelopment.

  • WhatthisbookcoversChapter1,WhatIsDevOpsandShouldYouCare?,introducestheDevOpsphilosophy.

    Chapter2,StartTreatingYourInfrastructureasCode,offersexamplesonhowtodeployinfrastructureascodeusingTerraformorCloudFormation.

    Chapter3,BringYourInfrastructureunderConfigurationManagement,demonstrateshowtoconfigureEC2instancesusingSaltStack.

    Chapter4,Build,Test,andReleaseFasterwithContinuousIntegration,describestheprocessofsettingupaCIworkflowusingaJenkinsCIserver.

    Chapter5,Ever-ReadytoDeployUsingContinuousDelivery,showshowtoextendaCIpipelinetoproducedeployment-readyEC2AMIsusingPackerandServerspec.

    Chapter6,ContinuousDeployment-AFullyAutomatedWorkflow,OffersafullyautomatedworkflowandcompletestheCI/CDeliverypipelinebyaddingthefunctionalityneededforAMIdeployment.

    Chapter7,Metrics,LogCollection,andMonitoring,introducesPrometheus,Logstash,Elasticsearch,andrelatedDevOpstools.

    Chapter8,OptimizeforScaleandCost,offersadviceonhowtoplananAWSdeploymentwithscalabilityandcostefficiencyinmind.

    Chapter9,SecureYourAWSEnvironment,coversbestpracticesinordertoimprovethesecurityofanAWSdeployment.

    Chapter10,AWSTipsandTricks,containsaselectionofusefultipsforabeginnertoanintermediateAWSusers.

  • WhatyouneedforthisbookThepracticalexamplesfoundinthisbookinvolvetheuseofAWSresources,thusanAWSaccountwillberequired.

    Theclient-sidetoolsusedintheexamples,suchastheAWSCLIandTerraform,aresupportedonmostcommonoperatingsystems(Linux/Windows/MacOS).

  • WhothisbookisforThisbookisforsystemadministratorsanddeveloperswhomanageAWSinfrastructureandenvironmentsandareplanningtoimplementDevOpsintheirorganizations.ThoseaimingfortheAWSCertifiedDevOpsEngineercertificationwillalsofindthisbookuseful.PriorexperienceofoperatingandmanagingAWSenvironmentsisexpected.

  • ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.

    Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:"WeneedtoSSHintothenodeandretrievetheadminpasswordstoredin/var/lib/jenkins/secrets/initialAdminPassword."

    Ablockofcodeissetasfollows:

    aws-region="us-east-1"vpc-cidr="10.0.0.0/16"vpc-name="Terraform"aws-availability-zones="us-east-1b,us-east-1c"

    Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

    aws-region="us-east-1"vpc-cidr="10.0.0.0/16"vpc-name="Terraform"aws-availability-zones="us-east-1b,us-east-1c"

    Anycommand-lineinputoroutputiswrittenasfollows:

    $terraformvalidate$terraformplanRefreshingTerraformstatepriortoplan......Plan:11toadd,0tochange,0todestroy.$terraformapplyaws_iam_role.jenkins:Creating......Applycomplete!Resources:11added,0changed,0destroyed.Outputs:JENKINSEIP=x.x.x.xVPCID=vpc-xxxxxx

    Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:"WeselectPipelineasajobtypeandpickanameforit."

    Note

    Warningsorimportantnotesappearinaboxlikethis.

    Tip

  • Tipsandtricksappearlikethis.

  • ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook-whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.

    Tosendusgeneralfeedback,[email protected],andmentionthebook'stitleinthesubjectofyourmessage.

    Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.

    http://www.packtpub.com/authors

  • CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

  • DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesforthisbookfromyouraccountathttp://www.packtpub.com.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

    Youcandownloadthecodefilesbyfollowingthesesteps:

    1. Loginorregistertoourwebsiteusingyoure-mailaddressandpassword.2. HoverthemousepointerontheSUPPORTtabatthetop.3. ClickonCodeDownloads&Errata.4. EnterthenameofthebookintheSearchbox.5. Selectthebookforwhichyou'relookingtodownloadthecodefiles.6. Choosefromthedrop-downmenuwhereyoupurchasedthisbookfrom.7. ClickonCodeDownload.

    Oncethefileisdownloaded,pleasemakesurethatyouunziporextractthefolderusingthelatestversionof:

    WinRAR/7-ZipforWindowsZipeg/iZip/UnRarXforMac7-Zip/PeaZipforLinux

    ThecompletesetofcodecanalsobedownloadedfromthefollowingGitHubrepository:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS.

    http://www.packtpub.comhttp://www.packtpub.com/supporthttps://github.com/PacktPublishing/Implementing-DevOps-on-AWS

  • DownloadingthecolorimagesofthisbookWealsoprovideyouwithaPDFfilethathascolorimagesofthescreenshots/diagramsusedinthisbook.Thecolorimageswillhelpyoubetterunderstandthechangesintheoutput.Youcandownloadthisfilefromhttps://www.packtpub.com/sites/default/files/downloads/ImplementingDevOpsonAWS_ColorImages.pdf

    https://www.packtpub.com/sites/default/files/downloads/ImplementingDevOpsonAWS_ColorImages.pdf

  • ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks-maybeamistakeinthetextorthecode-wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.

    Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.

    http://www.packtpub.com/submit-erratahttps://www.packtpub.com/books/content/support

  • PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

    Pleasecontactusatcopyright@packtpub.comwithalinktothesuspectedpiratedmaterial.

    Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.

  • QuestionsIfyouhaveaproblemwithanyaspectofthisbook,[email protected],andwewilldoourbesttoaddresstheproblem.

  • Chapter1.WhatisDevOpsandShouldYouCare?DevOpscanbeseenasanextensionofthesuccessfulandwellestablishedAgilemethodology.Bringingoperationsintothepicturehelpstheadvancefromcontinuous(Agile)developmenttointegrationtodeployment,butmoreimportantlyithelpsbuildabetterworkingenvironment,onewithstrongercross-teamrelationships.

    IfIhadtodescribeDevOpsinoneword,itwouldbecollaboration.ThegenuinewillingnessofboththeDevandOpscampstoworktogetheristhefoundation,themostimportantaspectofthephilosophy.

    DevOpsappearsasthemeetingpointinthefollowingdiagram:

    Inthischapterwewillgothroughthefollowingtopics:

    WhatisDevOps?Questionsyoushouldaskyourselfbeforeadoptingit

  • WhatisDevOps?So,letusexaminethevariousprinciplecharacteristicsofaDevOpsenvironment.

    Whatfollowsisaseriesofgenerallyaccepteddefinitions,invariablymixedwithpersonalopinions-youhavebeenwarned.

  • AcommongoalThealignmentofefforttowardincreasingsystemperformanceandstability,reducingthetimeittakestodeployorimprovingtheoverallqualityoftheproduct,willresultinhappiercustomersandproudengineers.

    Thegoalneedstoberepeated,clarified,andsimplifieduntilitisfullyunderstood,arguedagainst,andeventuallycommittedtobyeverybody.

    DevOpsshiftsfocusawayfromself-interestandtowardthatgoal.Itdirectspraiseatgroupachievementsratherthanthoseoftheindividual;KPIsandEmployeeoftheMonthinitiativesperhapsnotsomuch.

    Allowpeopletolookatthebiggerpicturepasttherealmoftheircubicle.Trustthem.

  • Sharedknowledge(nosilos)Thechancesareyouhavealreadyheardstoriesorreadbooksaboutthenotoriousorganizationalsilos.

    Intheworstcase,itwouldbesomebodywhorefusestoletgoandoftenbecomesthemainbottleneckinadevelopmentlifecycle.Theycanbefiercelyterritorial,safe-guardingwhatexclusiveknowledgetheymighthaveinagivenfield,likely(Ispeculate)becausethisprovidesthemwithasenseofimportance,furthercateringtotheirego.

    Ontheotherhand,therearealsoexamplesofpeoplewhofindthemselvesinasilopurelyduetounfortunatecircumstances.Myrespectgoesouttothemanyengineersstuckwithsupportinginheritedlegacysystemsallbythemselves.

    Fortunately,DevOpsblurssuchbordersofexpertisewithconceptslikecross-functionalteamsandfull-stackengineers.Itisimportanttonoteherethatthisdoesnottranslateintoanopportunitytocutcostsbyexpectingpeopletobetechninjaexpertsateverysinglething(whichinreallifeequatestoprecedingaverage).But,asinoneofthoseVenndiagrams,itisthecross-overbetweenaDevandanOpssetofskills.

    Silosareavoidedbyencouragingknowledgesharing.Peerreviews,demostand-ups,orshareddocumentationareafewwaystoensurethatnotaskorpieceofknow-howislimitedtoaspecificperson.YoucouldevenadoptPairProgramming.Itseemsabitheavy,butitevidentlyworks!

  • TrustandsharedresponsibilityShoulddevelopersbegivenproductionaccess?

    Therearegoodreasonsformaintainingstrictrole-basedpermissions;oneofthemissecurityanotherisintegrity.Thisstandpointremainsvalidforaslongaswemaintainthestereotypeofthedeveloperwhoissousedtoworkingindevlocal;tothem,conceptssuchaspassphrase-protectedSSHkeysornotmanuallyeditingallofthefilestakeabackseat.

    IntheeraofDevOps,thisisnolongerthecase.Sharedknowledgeandresponsibilitymeansoperationsengineerscanrelyontheirdevelopercolleaguestofollowthesamecodeofconductwhenworkingincritical,productionenvironments.

    DevandOpsteamshaveaccesstothesamesetoftoolsandenvironments.DeploymentsarenolongeraspecialtaskreservedfortheOpsteamandscheduleddaysinadvance.

    Inateamwithsuchknowledge-sharinghabits,I,asanoperationsengineer,canbeconfidentaboutmyDevcolleague'sabilitytoperformmytasks,andviceversa.

    Developersparticipateintheon-callrota,supportingthesoftwaretheyproduce.

    Thisisnottobeseenasanadditionalburden,butasasignoftrustandanopportunitytoincreasecollaboration.Nobodyisthrowingcodeoverthewallanymore.Responsibilityandasenseofautonomymotivatespeopletodomorethanisexpectedofthem.

  • RespectAswespendmoretimetalkingtoeachotheraboutthechallengeswefaceandtheproblemswearetryingtosolve,ourmutualrespectgrows.

    ThismanifestsitselfindevelopersseekinginputfromtheOperationsteamfromtheearlystagesofthesoftwaredevelopmentprocessorinOpstoolsbeingbuilttomeetdevelopers'needs.

    OpswhothinklikeDevs.DevswhothinklikeOps

    --JohnAllspawandPaulHammond,Velocity

    ADevOpsenvironmentisbuiltonsuchrespect.Itisaplacewhereeveryopinionmatters,wherepeoplecananddoopenlyquestiondecisionsintheinterestofthebestsolutiontoaproblem.Thisisapowerfulindicatorofone'scommitmenttowardthecommongoalImentionedearlier.

  • AutomationTodrawanoverlysimplifiedconclusionfromA.Maslow'sTheoryofMotivation,youarelesslikelytothinkaboutpoetrywhenhungry.Inotherwords,ateamwithbasicneedswillbesolvingbasicproblems.

    Automatingroutineandmundanetasksallowsengineerstoconcentrateonthemorecomplex,higher-valueones.Also,peoplegetbored,cutcorners,andmakemistakescomputerstendnottodoso.

  • ReproducibleinfrastructureDescribinginfrastructureascodehasthefollowingadvantages:

    ItcanbekeptunderversioncontrolItiseasilysharedwithotherstore-useorreproduceItservesasaveryusefuldiaryofwhatyoudidandhowexactlyyoudiditProvisioningcloudenvironmentsbecomestrivial(forexample,withTerraformorCloudFormation)ItmakesmodernConfigurationManagementpossible

    Atanyrate,Isuspectanybodymanagingmorethan10serversisalreadycodifyinginfrastructureinsomewayoranother.

  • Metricsandmonitoring

    MeasureAllTheThings!

    --ActualDevOpsslogan

    Storageischeap.Developthehabitofgatheringcopiousamountsofmeasurementsandmakingthoseeasilyaccessibleacrossyourorganization.Themorevisibilityengineershaveintotheperformanceoftheirinfrastructureandapplications,themoreadequatetheirdecisionswillbeincriticalsituations.

    Graphscanconveyagreatdealofinformation,lookrathercoolonbigscreens,andthehumanmindhasbeenprovenexcellentatrecognizingpatterns.

    Anotherimportantroleofmetricsdataisinperformanceoptimization.

    Thetrickiestpartofspeedingupaprogramisnotdoingit,butdecidingwhetherit'sworthdoingatall...Partoftheproblemisthatoptimizationishardtodowell.It'sfrighteninglyeasytodevolveintosuperstitiousritualandrationalization.

    --MatureOptimization,CarlosBueno

    Toavoidfallingpreytoconfirmationbias,youneedanobjectivemethodofassessingyoursystemsbeforeandafterattemptinganyoptimization.Usethosemetrics;itishardtoarguewith(valid)data.

    Onthesubjectofvalidity,pleasedocalibrateyourinstrumentsregularly,sanity-checkoutputandmakesurewhatyouthinkyouareshowingiswhatyourcolleaguesthinktheyareseeing(ref:https://mars.jpl.nasa.gov/msp98/news/mco990930.html).

    https://mars.jpl.nasa.gov/msp98/news/mco990930.html

  • ContinuousIntegration,Delivery,andDeploymentTheObserve,Orient,Decide,andAct(OODA)loopisaconceptdevelopedbyCol.J.Boydthatshowsthevalueinone'sabilitytoadapttoever-changingcircumstances.

    Facedwithunforgiving(andproductive)competition,organizationsshouldbeabletorapidlyreacttodynamicmarketconditions.

    ThisisprobablybestillustratedwiththeoldKodakandNetflixtales.Theformerafterhavingbeenwildlysuccessfulissaidtohavefailedtoadapttothenewtrendsintheirsector,causingthebrandtograduallyfadeaway.Incontrast,Netflixkeepsonskillfullymoldingtheirproducttomatchthenewwaysinwhichweconsumedigitalcontent.Theycompletelytransformedtheirinfrastructure,sharedsomeclever,newandsomewhatcontroversialpracticesplusatonofgreatsoftware.BelikeNetflix.

    ContinuousIntegrationandDeliveryisessentiallyOODAinpractice.Teamscontinuouslyintegraterelativelysmallcodechanges,deliveringreleasesmoreoften,thusgettingfeedbackfromtheirusersmuchquicker.Thetypeoffeedbackneededbyanorganizationtobeabletoadequatelyrespondtoaneverchangingmarket.

    Noneoftheprecedingsuggestshoweverthatoneshouldaimtobecomeareleasehero,rushingthingsintoProduction,settingitonfiretwiceaweek.ACI/CDframeworkstillimpliestheusualstrictcodereviewandtestprocesses,despitehowoftenyoudeploy.Thoughcodereviewsandtestingrequiremuchlesstimeandeffortastypicallythemorefrequentthedeployments,thesmallerthecodechanges.

  • EmbracingfailureNaturally,moreexperimentationislikelytoincreasetheprobabilityoferror.

    Idoubtthisfactcomesasasurprisetoanybody;whatmightsurpriseyou,however,istheadvicetoacceptanadditional,positiveangletofailure.

    Recallthevideonerdsfromtheprevioussection.Well,theydidn'texactlybreezethroughallthatchangewithoutcasualties.IherebyspareyoutheEdisonquotes;however,trialanderrorisindeedavalidformofthescientificmethod,andtheDevOpsprocessesserveasagreatenablertothosewhowouldagree.

    Inotherwords,anorganizationshouldencouragepeopletokeeponchallengingandimprovingthecurrentstateofaffairswhilealsoallowingthemtoopenlytalkaboutthetimeswhenthingswentwrong.

    Butdealingwithexperimentationfailuresispossiblythemoreromanticsideofthestorycomparedtothecold,harshrealityofday-to-dayoperations.

    Systemsfail.Iwouldliketothinkmostofushavecometoacceptthatfactalongwiththechainofthoughtitprovokes:

    wedonotalwaysknowasmuchaswethinkwedo:

    "Knowledgeoftheoutcomemakesitseemthateventsleadingtotheoutcomeshouldhaveappearedmoresalienttopractitionersatthetimethanwasactuallythecase...

    Afteranaccident,practitioneractionsmayberegardedaserrorsorviolations,buttheseevaluationsareheavilybiasedbyhindsightandignoretheotherdrivingforces..."

    --HowComplexSystemsFail,R.I.Cook

    Excelsior!Orhow,inourlong-standingpursuitofsocialdominance,weseemtohavedevelopedtheconvenientbeliefthatfollowinganeventwenotonlyknowexactlywhatandhowithappenedbutalsowhy.ThispeculiarphenomenonhasalreadybeenexplainedratherwellbyD.KahnemaninThinkingFastandSlow;Iwilljustaddthatindeedoneoftenhearsofoverconfidentcharacterswhopointfingersattheircolleaguesbasedonwhatappearstothemasacoherentstoryline.

    Thetruthofthematteristhis:wewerenotthere.Andkeepingthedetailswenowknowandthoseknownatthetimeseparatedisnotaneasytask.Blamingisofzerovalue:

    Etsyandthelikesinourcommunityhavesharedenoughobservationstosuggestthatnegativereinforcementasastrategyforreducinghumanerrorislessthanoptimal.

  • WiththeadoptionofDevOps,weacceptthatpeoplegenerallycometoworkeverydaywiththeintentiontoperformtothebestoftheirabilitiesandintheinterestoftheorganization.Afteranoutage,webeginouranalysiswiththeassumptionthattheoperatorhasactedinthebestpossiblewaygiventhecircumstancesandinformationavailabletothematthetime.Wefocusonwhatcouldhaveledtothemmakingthegivendecisions,theirthoughtprocess,theflowofevents,andwhetheranyofthesecanbeimproved.Resiliencecanbeaccumulated:

    "Whatdoesnotkillus..."mithridatismorNassimTaleb'sconceptofantifragilityareallexpressiveoftheideathatwegetbetteratdealingwithnegativeexperiencesasweencounterthem,andwhat'smore,weshouldlookforthemeverynowandagain.

    Wecantrainourselvesandoursystemstorecoverfromerrorsfasterorevenbettertocontinueoperatingdespitethem.Onewaytoachievethisiswithcontrolled(andwithpractice,lesscontrolled)outages.

    Withtherightmonitoringandauditingtoolsinplace,everyabnormalactivityoffersusamoreintimateviewofourapplicationsandinfrastructure.

    NowthatIhavebestoweduponyou,mydearreader,thesecrettoabetterlifethroughDevOps,letusconcernourselveswiththelatterpartofthetitleofthischapter.

  • ShouldyoucareIfailtoseeareasonwhyoneshouldnot.SomesevenorsoyearshavepassedsincetheinceptionoftheideaofDevOps,andtheamountofevidenceofitseffectivenesshasbeengrowingsteadily.HavingtherespectedAgileframeworkatitsbasefurtheraddstoitscredibilityandperhapshelpsexplainagoodpartofitssuccess.

    Thatisnottosaytherearenotconsiderationstobetakenintoaccounthowever.Thecriticalthinkerwithinyou,wouldwanttoaskaquestionortwopriortoembarkingonsuchaculturalcoupd'tat.

  • Isthistherighttime?DidyoujustfinishadoptingLeanorAgileDevelopment?Whatelsehasbeengoingonintheteam?Isnowthebesttimeforyetanothercryforchange?

    Alteringourhabitsmakesusuneasy;ittakessometimetoadjust.Yourperseveranceislaudable,andpursuingDevOpsasthenextlevelofteamcollaborationisoftentherightchoice.

    Thereisnoneedtogiveitupaltogether;perhapsputitonholdforamoment.

  • Willitwork?Lookaroundyou.Thosefaces,thosedifferentpersonalities,canyoupicturethemalltogethersingingKumbaya?Maybeyes,maybeno,ornotyet.

    Pleasedonote-mailananonymousstaffsurvey.Geteverybodyinaroom,layyourDevOpspropagandaout,andgaugetheirreactions.

    Youwillneedeveryonetofullyunderstandtheconcepts,acknowledgethechallenges,andacceptthesacrificesforthistowork.Therecanneitherbeexceptionsnorambiguity.

    Allofthisrequiresagreatdegreeofculturalchange,whichateamshouldbepreparedfor.

  • Isitworthit?Whatwouldittaketochangethecurrentmentality?Howmuchofadisturbanceyouwouldneedtocause?Whatdegreeofbacklashdoyouexpect?

    WhileIamnotsuggestingthisasanexcusetoputupwiththestatusquo,Ibegyoumaintainapragmaticviewofthesituation.

    Yourtypeoforganizationmightbebettersuitedforaprocessofevolutionratherthanarevolution.

  • Doyouneedit?Howwouldyouscoreyourcurrentprocesses?Wouldyousayyourcross-teamcommunicationissatisfactory?Youregularlymeetbusinessexpectations?Youhavealreadyautomatedmostofyourworkflow?

    Itsoundslikeyouaredoingfineasitis;youmightalreadyhavesomeDevOpsinyourteamwithoutrealizingit.Thepointisthatitcouldbeabetteruseofresourcesifyouweretoconcentrateonoptimizingelsewhere,solvingother,morepressingproblemsatthistime.

    NowthatyouhavebeenthroughayetanotherinterpretationoftheideasbehindDevOps,ifyoufeelthosematchyourwayofthinkingandthefinalfewquestionsdidnotraiseanyconcerns,thenwecansafelytransitiontothemoretechnicaltopicswhereweputprinciplesintopractice.

  • SummaryFirst,weexploredthemainideascontainedintheDevOpsphilosophy,followedbyafewquestionsaimedathelpingyouconstructamoreobjectiveperspectivewhenitcomestoadoptingDevOpswithinyourorganization.

    WehaveseenthatDevOpsisaneffectivecombinationofsomeolder,provenAgileconceptsandothermorerecentlydevelopedones,andthatitteachesushowtobuildbetterteamswhowritebettersoftware,getresultsfaster,andcollaborateeffortlesslyinanenvironmentthatencouragesexperimentationwithoutcompromisingstability.

    Nowthatwehavecoveredthetheory,thenextchaptertakesusintothepracticalapplicationofDevOps.Wearegoingtobeginwithexamplesofdeployinginfrastructureascodeinthecloud.

  • Chapter2.StartTreatingYourInfrastructureasCodeLadiesandgentlemen,putyourhandsintheair,forProgrammableInfrastructureishere!

    PerhapsInfrastructure-as-Code(IaC)isnotanentirelynewconceptconsideringhowlongConfigurationManagementhasbeenaround.Codifyingserver,storage,andnetworkinginfrastructureandtheirrelationships,however,isarelativelyrecenttendencybroughtaboutbytheriseofcloudcomputing.ButletusleaveConfigurationManagementforlaterandfocusourattentiononthatsecondaspectofIaC.

    Youshouldrecallfromthepreviouschaptersomeofthebenefitsofstoringallthethingsascode:

    CodecanbekeptunderversioncontrolCodecanbeshared/collaboratedoneasilyCodedoublesasdocumentationCodeisreproducible

    Thatlastpointwasabigwinformepersonally.Automatedprovisioninghelpedreducethetimeittooktodeployafull-featuredcloudenvironmentfromfourhoursdowntoone,andtheoccurrencesofhumanerrortoalmostzero(oneshallnotbetrustedwithaninputfield).

    Beingabletorapidlyprovisionresourcesbecomesasignificantadvantagewhenateamstartsusingmultipleenvironmentsinparallelandneedsthosebroughtupordownon-demand.Inthischapter,weexamineindetailhowtodescribe(incode)anddeployonesuchenvironmentonAWSwithminimalmanualinteraction.

    ToimplementIaCinthecloud,wewilllookattwotoolsorservices:TerraformandCloudFormation.

    Wewillgothroughexamplesofthefollowing:

    ConfiguringthetoolWritinganIaCtemplateDeployingatemplateDeployingsubsequentchangestothetemplateDeletingatemplateandremovingtheprovisionedinfrastructure

    Forthepurposeoftheseexamples,letusassumeourapplicationrequiresaVirtualPrivateCloud(VPC)thathostsaRelationalDatabaseServices(RDS)backendandacoupleofElasticComputeCloud(EC2)instancesbehindanElasticLoadBalancer(ELB).WewillkeepmostcomponentsbehindNetworkAddressTranslation(NAT),allowingonlytheloadbalancertobeaccessedexternally.

  • IaCusingTerraformOneofthetoolsthatcanhelpdeployinfrastructureonAWSisHashiCorp'sTerraform(https://www.terraform.io).HashiCorpisthatgeniusbunchthatgaveusVagrant,Packer,andConsul.Iwouldrecommendyoulookuptheirwebsiteifyouhavenotalready.

    UsingTerraform(TF),wewillbeabletowriteatemplatedescribinganenvironment,performadryruntoseewhatisabouttohappenandwhetheritisexpected,deploythetemplate,andmakeanylateadjustmentswherenecessary-allofthiswithoutleavingtheshellprompt.

    https://www.terraform.io

  • ConfigurationFirstly,youwillneedtohaveacopyofTF(https://www.terraform.io/downloads.html)onyourmachineandavailableontheCLI.Youshouldbeabletoquerythecurrentlyinstalledversion,whichinmycaseis0.6.15:

    $terraform--versionTerraformv0.6.15

    SinceTFmakesuseoftheAWSAPIs,itrequiresasetofauthenticationkeysandsomelevelofaccesstoyourAWSaccount.InordertodeploytheexamplesinthischapteryoucouldcreateanewIdentityandAccessManagement(IAM)userwiththefollowingpermissions:

    "autoscaling:CreateAutoScalingGroup","autoscaling:CreateLaunchConfiguration","autoscaling:DeleteLaunchConfiguration","autoscaling:Describe*","autoscaling:UpdateAutoScalingGroup","ec2:AllocateAddress","ec2:AssociateAddress","ec2:AssociateRouteTable","ec2:AttachInternetGateway","ec2:AuthorizeSecurityGroupEgress","ec2:AuthorizeSecurityGroupIngress","ec2:CreateInternetGateway","ec2:CreateNatGateway","ec2:CreateRoute","ec2:CreateRouteTable","ec2:CreateSecurityGroup","ec2:CreateSubnet","ec2:CreateTags","ec2:CreateVpc","ec2:Describe*","ec2:ModifySubnetAttribute","ec2:RevokeSecurityGroupEgress","elasticloadbalancing:AddTags","elasticloadbalancing:ApplySecurityGroupsToLoadBalancer","elasticloadbalancing:AttachLoadBalancerToSubnets","elasticloadbalancing:CreateLoadBalancer","elasticloadbalancing:CreateLoadBalancerListeners","elasticloadbalancing:Describe*","elasticloadbalancing:ModifyLoadBalancerAttributes","rds:CreateDBInstance","rds:CreateDBSubnetGroup","rds:Describe*"

    Note

    Pleaserefertothisfileformoreinformation:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/iam_user_policy.json.

    OnewaytomakethecredentialsoftheIAMuseravailabletoTFisbyexportingthefollowingenvironmentvariables:

    https://www.terraform.io/downloads.htmlhttps://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/iam_user_policy.json

  • $exportAWS_ACCESS_KEY_ID='user_access_key'$exportAWS_SECRET_ACCESS_KEY='user_secret_access_key'

    Thisshouldbesufficienttogetusstarted.

    Note

    Downloadingtheexamplecode

    DetailedstepstodownloadthecodebundlearementionedinthePrefaceofthisbook.

    ThecodebundleforthebookisalsohostedonGitHubat:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS.Wealsohaveothercodebundlesfromourrichcatalogofbooksandvideosavailableat:https://github.com/PacktPublishing/.Checkthemout!

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWShttps://github.com/PacktPublishing/

  • TemplatedesignBeforewegettocoding,herearesomeoftherules:

    YoucouldchoosetowriteaTFtemplateasasinglelargefileoracombinationofsmalleronesTemplatescanbewritteninpureJSONorTF'sownformatTFwilllookforfileswith.tfor.tf.jsonextensionsinagivenfolderandloadtheminalphabeticalorderTFtemplatesaredeclarative,hencetheorderinwhichresourcesappearinthemdoesnotaffecttheflowofexecution

    ATFtemplategenerallyconsistsofthreesections:resources,variables,andoutputs.Asmentionedintheprecedingsection,itisamatterofpersonalpreferencehowyouarrangethese;however,forbetterreadabilityIsuggestwemakeuseoftheTFformatandwriteeachsectiontoaseparatefile.Also,whilethefileextensionsareofimportance,thefilenamesareuptoyou.

    Resources

    Inaway,thisfileholdsthemainpartofatemplate,astheresourcesrepresenttheactualcomponentsthatendupbeingprovisioned.Forexample,wewillbeusingtheVPCTerraformresource,RDS,ELBandafewotherstoprovisionwhatroughlylookslikethis:

  • Sincetemplateelementscanbewritteninanyorder,TFdeterminestheflowofexecutionbyexamininganyreferencesthatitfinds(forexample,aVPCshouldexistbeforeanELBthatissaidtobelongtoitiscreated).Alternatively,explicitflowcontrolattributessuchasdepends_onareused,aswewillobserveshortly.

    Tofindoutmore,letusgothroughthecontentsoftheresources.tffile.

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/resources.tf.

    First,wetellTerraformwhatprovidertouseforourinfrastructure:

    #SetaProvider

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/resources.tf

  • provider"aws"{region="${var.aws-region}"}

    Youwillnoticethatnocredentialsarespecified,sincewesetthemasenvironmentvariablesearlier.

    NowwecanaddtheVPCanditsnetworkingcomponents:

    #CreateaVPCresource"aws_vpc""terraform-vpc"{cidr_block="${var.vpc-cidr}"tags{Name="${var.vpc-name}"}}#CreateanInternetGatewayresource"aws_internet_gateway""terraform-igw"{vpc_id="${aws_vpc.terraform-vpc.id}"}#CreateNATresource"aws_eip""nat-eip"{vpc=true}

    Sofar,wehavedeclaredtheVPC,itsInternetandNATgateways,plusasetofpublicandprivatesubnetswithmatchingroutingtables.

    Itwillhelpclarifythesyntaxifweexaminedsomeofthoseresourceblocks,linebyline:

    resource"aws_subnet""public-1"{

    Thefirstargumentisthetypeoftheresourcefollowedbyanarbitraryname:

    vpc_id="${aws_vpc.terraform-vpc.id}"

    Theaws_subnetresourcenamedpublic-1hasavpc_idproperty,whichreferstotheidattributeofadifferentaws_vpcresourcenamedterraform-vpc.Suchreferencestootherresourcesimplicitlydefinetheexecutionflow,thatistosay,theVPCneedstoexistbeforethesubnetcanbecreated:

    cidr_block="${cidrsubnet(var.vpc-cidr,8,1)}"

    Wewilltalkmoreaboutvariablesinamoment,buttheformatisvar.var_nameasshown

  • here.

    Here,weusethecidrsubnetfunctionwiththevpc-cidrvariable,whichreturnsacidr_blocktobeassignedtothepublic-1subnet.PleaserefertotheTFdocumentationforthisandotherusefulfunctions.

    Next,weaddaRDStotheVPC:

    resource"aws_db_instance""terraform"{identifier="${var.rds-identifier}"allocated_storage="${var.rds-storage-size}"storage_type="${var.rds-storage-type}"engine="${var.rds-engine}"engine_version="${var.rds-engine-version}"instance_class="${var.rds-instance-class}"username="${var.rds-username}"password="${var.rds-password}"port="${var.rds-port}"vpc_security_group_ids=["${aws_security_group.terraform-rds.id}"]db_subnet_group_name="${aws_db_subnet_group.rds.id}"}

    Here,wemostlyseereferencestovariableswithafewcallstootherresources.

    FollowingtheRDSisanELB:

    resource"aws_elb""terraform-elb"{name="terraform-elb"security_groups=["${aws_security_group.terraform-elb.id}"]subnets=["${aws_subnet.public-1.id}","${aws_subnet.public-2.id}"]listener{instance_port=80instance_protocol="http"lb_port=80lb_protocol="http"}tags{Name="terraform-elb"}}

    Lastly,wedefinetheEC2AutoScalingGroupandrelatedresourcessuchastheLaunchConfiguration.

    FortheLaunchConfigurationwedefinetheAMIandtypeofinstancetobeused,thenameoftheSSHkeypair,EC2securitygroup(s)andtheUserDatatobeusedtobootstraptheinstances:

  • resource"aws_launch_configuration""terraform-lcfg"{image_id="${var.autoscaling-group-image-id}"instance_type="${var.autoscaling-group-instance-type}"key_name="${var.autoscaling-group-key-name}"security_groups=["${aws_security_group.terraform-ec2.id}"]user_data="#!/bin/bash\nset-euf-opipefail\nexec1>>(logger-s-t$(basename$0))2>&1\nyum-yinstallnginx;chkconfignginxon;servicenginxstart"lifecycle{create_before_destroy=true}

    TheAutoScalingGrouptakestheIDoftheLaunchConfiguration,alistofVPCsubnets,themin/maxnumberofinstancesandthenameoftheELBtoattachprovisionedinstancesto:

    }resource"aws_autoscaling_group""terraform-asg"{name="terraform"launch_configuration="${aws_launch_configuration.terraform-lcfg.id}"vpc_zone_identifier=["${aws_subnet.private-1.id}","${aws_subnet.private-2.id}"]min_size="${var.autoscaling-group-minsize}"max_size="${var.autoscaling-group-maxsize}"load_balancers=["${aws_elb.terraform-elb.name}"]depends_on=["aws_db_instance.terraform"]tag{key="Name"value="terraform"propagate_at_launch=true}}

    Theprecedinguser_datashellscriptwillinstallandstartNGINXontotheEC2node(s).

    Variables

    Wehavemadegreatuseofvariablestodefineourresources,makingthetemplateasre-usableaspossible.Letusnowlookinsidevariables.tftostudythesefurther.

    Similarlytotheresourceslist,westartwiththeVPC:

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/variables.tf.

    variable"aws-region"{type="string"description="AWSregion"}variable"aws-availability-zones"{type="string"description="AWSzones"

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/variables.tf

  • }variable"vpc-cidr"{type="string"description="VPCCIDR"}variable"vpc-name"{type="string"description="VPCname"}

    Thesyntaxisasfollows:

    variable"variable_name"{variableproperties}

    variable_nameisarbitrary,butneedstomatchrelevantvar.var_namereferencesmadeinotherpartsofthetemplate.Forexample,theaws-regionvariablewillsatisfythe${var.aws-region}referencewemadeearlierwhendescribingtheregionoftheproviderawsresource.

    Wewillmostlyusestringvariables,butthereisanotherusefultypecalledmapthatcanholdlookuptables.Mapsarequeriedinasimilarwaytolookingupvaluesinahash/dict(Pleasesee:https://www.terraform.io/docs/configuration/variables.html).

    NextcomesRDS:

    variable"rds-identifier"{type="string"description="RDSinstanceidentifier"}variable"rds-storage-size"{type="string"description="StoragesizeinGB"}variable"rds-storage-type"{type="string"description="Storagetype"}variable"rds-engine"{type="string"description="RDStype"}variable"rds-engine-version"{type="string"description="RDSversion"}variable"rds-instance-class"{type="string"description="RDSinstanceclass"}variable"rds-username"{type="string"description="RDSusername"}

    https://www.terraform.io/docs/configuration/variables.html

  • variable"rds-password"{type="string"description="RDSpassword"}variable"rds-port"{type="string"description="RDSportnumber"}

    Lastly,weaddourEC2relatedvariables:

    variable"autoscaling-group-minsize"{type="string"description="MinsizeoftheASG"}variable"autoscaling-group-maxsize"{type="string"description="MaxsizeoftheASG"}variable"autoscaling-group-image-id"{type="string"description="EC2AMIidentifier"}variable"autoscaling-group-instance-type"{type="string"description="EC2instancetype"}variable"autoscaling-group-key-name"{type="string"description="EC2sshkeyname"}

    Wenowhavethetypeanddescriptionofallourvariablesdefinedinvariables.tf,butnovalueshavebeenassignedtothemyet.

    TFisquiteflexiblewithhowthiscanbedone.Wecoulddoitanyofthefollowingways:

    Assign(default)valuesdirectlyinvariables.tf:variable"aws-region"{type="string"description="AWSregion"default='us-east-1'}Notassignavaluetoavariable,inwhichcaseTFwillpromptforitatruntime*Passa-var'key=value'argumentdirectlytotheTFcommand,likeso:

    -var'aws-region=us-east-1'

    Storekey=valuepairsinafileUseenvironmentvariablesprefixedwithTF_VAR,asinTF_VAR_aws-region

    Usingakey=valuepairsfileprovestobequiteconvenientwithinteams,aseachengineercanhaveaprivatecopy(excludedfromrevisioncontrol).Ifthefileisnamedterraform.tfvarsitwillbereadautomaticallybyTF;alternatively,-var-filecanbeusedonthecommandlinetospecifyadifferentsource.

  • Hereisthecontentofoursampleterraform.tfvarsfile:

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/terraform.tfvars.

    autoscaling-group-image-id="ami-08111162"autoscaling-group-instance-type="t2.nano"autoscaling-group-key-name="terraform"autoscaling-group-maxsize="1"autoscaling-group-minsize="1"aws-availability-zones="us-east-1b,us-east-1c"aws-region="us-east-1"rds-engine="postgres"rds-engine-version="9.5.2"rds-identifier="terraform-rds"rds-instance-class="db.t2.micro"rds-port="5432"rds-storage-size="5"rds-storage-type="gp2"rds-username="dbroot"rds-password="donotusethispassword"vpc-cidr="10.0.0.0/16"vpc-name="Terraform"

    Apointofinterestisaws-availability-zones,asitholdsmultiplevaluesthatweinteractwithusingtheelementandsplitfunctions,asseeninresources.tf.

    Outputs

    Thethird,mostlyinformationalpartofourtemplatecontainstheTFOutputs.Theseallowselectedvaluestobereturnedtotheuserwhentesting,deployingorafteratemplatehasbeendeployed.Theconceptissimilartohowechostatementsarecommonlyusedinshellscriptstodisplayusefulinformationduringexecution.

    Letusaddoutputstoourtemplatebycreatinganoutputs.tffile:

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/outputs.tf.

    output"VPCID"{value="${aws_vpc.terraform-vpc.id}"}output"NATEIP"{value="${aws_nat_gateway.terraform-nat.public_ip}"}output"ELBURI"{value="${aws_elb.terraform-elb.dns_name}"

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/terraform.tfvarshttps://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/Terraform/outputs.tf

  • }output"RDSEndpoint"{value="${aws_db_instance.terraform.endpoint}"}

    Toconfigureanoutput,yousimplyreferenceagivenresourceanditsattribute.Asshowninprecedingcode,wehavechosentheIDoftheVPC,theElasticIPaddressoftheNATgateway,theDNSnameoftheELBandtheendpointaddressoftheRDSinstance.

    Thissectioncompletesthetemplateinthisexample.Youshouldnowhavefourfilesinyourtemplatefolder:resources.tf,variables.tf,terraform.tfvars,andoutputs.tf.

  • OperationsWeshallexaminefivemainTFoperations:

    ValidatingatemplateTesting(dry-run)InitialdeploymentUpdatingadeploymentRemovalofadeployment

    Note

    Inthefollowingcommandlineexamples,Terraformisrunwithinthefolderthatcontainsthetemplatefiles.

    Validation

    Beforegoinganyfurther,abasicsyntaxcheckshouldbedonewiththeterraformvalidatecommand.Afterrenamingoneofthevariablesinresources.tf,validatereturnsanunknownvariableerror:

    $terraformvalidateErrorvalidating:1error(s)occurred:*providerconfig'aws':unknownvariablereferenced:'aws-region-1'.defineitwith'variable'blocks

    Oncethevariablenamehasbeencorrected,re-runningvalidatereturnsnooutput,meaningvalidationhaspassed.

    Dry-run

    Thenextstepistoperformatest/dry-runexecutionwithterraformplan,whichdisplayswhatwouldhappenduringanactualdeployment.Thecommandreturnsacolor-codedlistofresourcesandtheirpropertiesormoreprecisely,asfollows:

    $terraformplanResourcesareshowninalphabeticalorderforquickscanning.Greenresourceswillbecreated(ordestroyedandthencreatedifanexistingresourceexists),yellowresourcesarebeingchangedin-place,andredresourceswillbedestroyed.

    Toliterallygetthepictureofwhattheto-be-deployedinfrastructurelookslike,youcoulduseterraformgraph:

    $terraformgraph>my_graph.dot

    DOTfilescanbemanipulatedwiththeGraphvizopensourcesoftware(Pleaseseehttp://www.graphviz.org)ormanyonlinereaders/converters.Thefollowingdiagramisaportionofalargergraphrepresentingthetemplatewedesignedearlier:

    http://www.graphviz.org

  • Terraformgraph

    Deployment

    Ifyouarehappywiththeplanandgraph,thetemplatecannowbedeployedusingterraformapply:

    $terraformapplyaws_eip.nat-eip:Creating...allocation_id:""=>""association_id:""=>""domain:""=>""instance:""=>""network_interface:""=>""private_ip:""=>""public_ip:""=>""vpc:""=>"1"aws_vpc.terraform-vpc:Creating...cidr_block:""=>"10.0.0.0/16"default_network_acl_id:""=>""default_security_group_id:""=>""dhcp_options_id:""=>""enable_classiclink:""=>""enable_dns_hostnames:""=>""Applycomplete!Resources:22added,0changed,0destroyed.

    Thestateofyourinfrastructurehasbeensavedtothefollowingpath.Thisstateisrequiredtomodifyanddestroyyourinfrastructure,sokeepitsafe.Toinspectthecompletestate,usetheterraformshowcommand.

    Statepath:terraform.tfstateOutputs:ELBURI=terraform-elb-xxxxxx.us-east-1.elb.amazonaws.comNATEIP=x.x.x.xRDSEndpoint=terraform-rds.xxxxxx.us-east-1.rds.amazonaws.com:5432VPCID=vpc-xxxxxx

    Attheendofasuccessfuldeployment,youwillnoticetheOutputsweconfiguredearlierandamessageaboutanotherimportantpartofTF-thestatefile(pleasereferto

  • https://www.terraform.io/docs/state/):

    TFstoresthestateofyourmanagedinfrastructurefromthelasttimeTFwasrun.Bydefault,thisstateisstoredinalocalfilenamedterraform.tfstate,butitcanalsobestoredremotely,whichworksbetterinateamenvironment.

    TFusesthislocalstatetocreateplansandmakechangestoyourinfrastructure.Priortoanyoperation,TFdoesarefreshtoupdatethestatewiththerealinfrastructure.

    Inasense,thestatefilecontainsasnapshotofyourinfrastructureandisusedtocalculateanychangeswhenatemplatehasbeenmodified.Normally,youwouldkeeptheterraform.tfstatefileunderversioncontrolalongsideyourtemplates.Inateamenvironmenthowever,ifyouencountertoomanymergeconflictsyoucanswitchtostoringthestatefile(s)inanalternativelocationsuchasS3(pleasesee:https://www.terraform.io/docs/state/remote/index.html).

    AllowafewminutesfortheEC2nodetofullyinitialize,thentryloadingtheELBURIfromtheprecedingOutputsinyourbrowser.Youshouldbegreetedbynginx,asshowninthefollowingscreenshot:

    Updates

    AsperMurphy'sLaw,assoonaswedeployatemplate,achangetoitwillbecomenecessary.Fortunately,allthatisneededforthisistoupdateandre-deploythegiventemplate.

    Let'ssayweneedtoaddanewruletotheELBsecuritygroup(showninbold):

    1. Updatethe"aws_security_group""terraform-elb"resourceblockinresources.tf:

    resource"aws_security_group""terraform-elb"{name="terraform-elb"description="ELBsecuritygroup"vpc_id="${aws_vpc.terraform-vpc.id}"

    https://www.terraform.io/docs/state/https://www.terraform.io/docs/state/remote/index.html

  • ingress{from_port="80"to_port="80"protocol="tcp"cidr_blocks=["0.0.0.0/0"]}ingress{from_port="443"to_port="443"protocol="tcp"cidr_blocks=["0.0.0.0/0"]}egress{from_port=0to_port=0protocol="-1"cidr_blocks=["0.0.0.0/0"]}}

    2. Verifywhatisabouttochange:

    $terraformplan...~aws_security_group.terraform-elbingress.#:"1"=>"2"ingress.2214680975.cidr_blocks.#:"1"=>"1"ingress.2214680975.cidr_blocks.0:"0.0.0.0/0"=>"0.0.0.0/0"ingress.2214680975.from_port:"80"=>"80"ingress.2214680975.protocol:"tcp"=>"tcp"ingress.2214680975.security_groups.#:"0"=>"0"ingress.2214680975.self:"0"=>"0"ingress.2214680975.to_port:"80"=>"80"ingress.2617001939.cidr_blocks.#:"0"=>"1"ingress.2617001939.cidr_blocks.0:""=>"0.0.0.0/0"ingress.2617001939.from_port:""=>"443"ingress.2617001939.protocol:""=>"tcp"ingress.2617001939.security_groups.#:"0"=>"0"ingress.2617001939.self:""=>"0"ingress.2617001939.to_port:""=>"443"Plan:0toadd,1tochange,0todestroy.

    3. Deploythechange:

    $terraformapply...aws_security_group.terraform-elb:Modifying...ingress.#:"1"=>"2"ingress.2214680975.cidr_blocks.#:"1"=>"1"ingress.2214680975.cidr_blocks.0:"0.0.0.0/0"=>"0.0.0.0/0"ingress.2214680975.from_port:"80"=>"80"ingress.2214680975.protocol:"tcp"=>"tcp"ingress.2214680975.security_groups.#:"0"=>"0"

  • ingress.2214680975.self:"0"=>"0"ingress.2214680975.to_port:"80"=>"80"ingress.2617001939.cidr_blocks.#:"0"=>"1"ingress.2617001939.cidr_blocks.0:""=>"0.0.0.0/0"ingress.2617001939.from_port:""=>"443"ingress.2617001939.protocol:""=>"tcp"ingress.2617001939.security_groups.#:"0"=>"0"ingress.2617001939.self:""=>"0"ingress.2617001939.to_port:""=>"443"aws_security_group.terraform-elb:Modificationscomplete...Applycomplete!Resources:0added,1changed,0destroyed.

    Note

    Someupdateoperationscanbedestructive(pleaserefertohttp://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html).YoushouldalwayschecktheCloudFormationdocumentationontheresourceyouareplanningtomodifytoseewhetherachangeisgoingtocauseaninterruption.TFprovidessomeprotectionviatheprevent_destroylifecycleproperty(pleaserefertohttps://www.terraform.io/docs/configuration/resources.html#prevent_destroy).

    Removal

    ThisisafriendlyremindertoalwaysremoveAWSresourcesafteryouaredoneexperimentingwiththemtoavoidanyunexpectedcharges.

    Beforeperforminganydeleteoperations,wewillneedtograntsuchprivilegestothe(terraform)IAMuserwecreatedinthebeginningofthischapter.Asashortcut,youcouldtemporarilyattachtheAdministratorAccessmanagedpolicytotheuserviatheAWSConsole,asshowninthefollowingfigure:

    ToremovetheVPCandallassociatedresourcesthatwecreatedaspartofthisexample,wewilluseterraformdestroy:

    http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.htmlhttps://www.terraform.io/docs/configuration/resources.html#prevent_destroy

  • $terraformdestroyDoyoureallywanttodestroy?Terraformwilldeleteallyourmanagedinfrastructure.Thereisnoundo.Only'yes'willbeacceptedtoconfirm.Enteravalue:yes

    Terraformasksforaconfirmationthenproceedstodestroyresources,endingwiththefollowing:

    Applycomplete!Resources:0added,0changed,22destroyed.

    Next,weremovethetemporaryadminaccesswegrantedtotheIAMuserbydetachingtheAdministratorAccessmanagedpolicy,asshowninthefollowingscreenshot:

    Then,verifythattheVPCisnolongervisibleintheAWSConsole.

  • IaCusingCloudFormationCloudFormationisanAWSservicefordeployinginfrastructureascode.Asbefore,wearegoingtodescribeourinfrastructureviatemplatescontainingparameters(variables),resources,andoutputs.

    CloudFormationcallseachdeployedtemplateaStack.Creating,listing,updating,anddeletingstacksispossibleviatheAWSConsole,CLI,orAPI.Inasmallsetup,youwouldprobablydeployeachofyourstacksindividually,butasyourarchitecturebecomesmorecomplex,youcanstartnestingstacks.Youwouldhaveatop-leveloraparentstack(template)thatinvokesanumberofsub-stacks.Nestedstacksallowyoutopassvariablesbetweenthemand,ofcourse,saveyouthetimeofhavingtodeployeachoneindividually.

  • ConfigurationCloudFormationprovidesaGUIviatheAWSConsole;wehowever,aregoingtofocusontheAWSCLIsinceitismostsuitableforautomatingtasksinthefuture.

    DependingontheOSyourun,youcoulddownloadaninstallerfromhttps://aws.amazon.com/cli/orusePythonPIP:

    $pipinstallawscli$aws--versionaws-cli/1.10.34...

    WewillneedasetofAPIkeys,solet'screateanewIAMusercalledcloudformationwiththefollowingprivileges:

    "cloudformation:CancelUpdateStack","cloudformation:ContinueUpdateRollback","cloudformation:Create*","cloudformation:Describe*","cloudformation:EstimateTemplateCost","cloudformation:ExecuteChangeSet","cloudformation:Get*","cloudformation:List*","cloudformation:PreviewStackUpdate","cloudformation:SetStackPolicy","cloudformation:SignalResource","cloudformation:UpdateStack","cloudformation:ValidateTemplate","autoscaling:CreateAutoScalingGroup","autoscaling:CreateLaunchConfiguration","autoscaling:DeleteLaunchConfiguration","autoscaling:Describe*","autoscaling:UpdateAutoScalingGroup","ec2:AllocateAddress","ec2:AssociateAddress","ec2:AssociateRouteTable","ec2:AttachInternetGateway","ec2:AuthorizeSecurityGroupEgress","ec2:AuthorizeSecurityGroupIngress","ec2:CreateInternetGateway","ec2:CreateNatGateway","ec2:CreateRoute","ec2:CreateRouteTable","ec2:CreateSecurityGroup","ec2:CreateSubnet","ec2:CreateTags","ec2:CreateVpc","ec2:Describe*","ec2:Modify*","ec2:RevokeSecurityGroupEgress","elasticloadbalancing:CreateLoadBalancer","elasticloadbalancing:CreateLoadBalancerListeners","elasticloadbalancing:Describe*","elasticloadbalancing:ModifyLoadBalancerAttributes",

    https://aws.amazon.com/cli/

  • "elasticloadbalancing:SetLoadBalancerPoliciesOfListener","rds:CreateDBInstance","rds:CreateDBSubnetGroup","rds:Describe*"

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/iam_user_policy.json.

    Youhavethechoiceofusingawsconfigure,whichwillpromptyoufortheAPIcredentials,orifyouprefernottostorethempermanently,youcoulduseanenvironmentvariable:

    $exportAWS_ACCESS_KEY_ID='user_access_key'$exportAWS_SECRET_ACCESS_KEY='user_secret_access_key'

    CloudFormationtemplatesdonotstoreanyAWSregioninformation,sotoavoidspecifyingitonthecommandlineeachtime.Itcanbeexportedaswell:

    $exportAWS_DEFAULT_REGION='us-east-1'

    Withthoseenvironmentvariablesinplace,awsclishouldbereadyforuse.

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/iam_user_policy.json

  • TemplatedesignCloudFormationtemplatesarewritteninJSONandusuallycontainatleastthreesections(inanyorder):parameters,resourcesandoutputs.

    Unfortunatelyitisnotpossibletostoretheseintoseparatefiles(withtheexceptionofparametervalues),sointhisexamplewewillworkwithasingletemplatefilenamedmain.json.

    Templatescanbeusedlocallyorimportedfromaremotelocation(anS3bucketisacommonchoice).

    Parameters

    ParametersaddflexibilityandportabilitytoourStackbylettinguspassvariablestoitsuchasinstancetypes,AMIids,SSHkeypairnamesandsimilarvalueswhichitisbestnottohard-code.

    Eachparametertakesanarbitrarylogicalname(alphanumeric,uniquewithinthetemplate),description,type,andanoptionaldefaultvalue.TheavailabletypesareString,Number,CommaDelimitedList,andthemorespecialAWS-specifictype,suchasAWS::EC2::KeyPair::KeyName,asseenintheprecedingcode.

    Thelatterisusefulforvalidation,asCloudFormationwillcheckwhetherakeypairwiththegivennameactuallyexistsinyourAWSaccount.

    ParameterscanalsohavepropertiessuchasAllowedValues,Min/MaxLength,Min/MaxValue,NoEchoandother(pleaseseehttp://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html).

    Thereisalimitof60parameterspertemplate.

    Letusexaminetheparametersfoundatthetopofourtemplate:

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/main.json.

    "Parameters":{"vpcCidr":{"Description":"VPCCIDR","Type":"String"},"vpcName":{"Description":"VPCname","Type":"String"},"awsAvailabilityZones":{

    http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.htmlhttps://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/main.json

  • "Description":"ListofAZs","Type":"CommaDelimitedList"},"publicCidr":{"Description":"ListofpublicsubnetCIDRs","Type":"CommaDelimitedList"},..."rdsInstanceClass":{"Description":"RDSinstanceclass","Type":"String","AllowedValues":["db.t2.micro","db.t2.small","db.t2.medium"]},"rdsUsername":{"Description":"RDSusername","Type":"String"},"rdsPassword":{"Description":"RDSpassword","Type":"String","NoEcho":"true"},..."autoscalingGroupKeyname":{"Description":"EC2sshkeyname","Type":"AWS::EC2::KeyPair::KeyName"},"autoscalingGroupImageId":{"Description":"EC2AMIID","Type":"AWS::EC2::Image::Id"}}

    Wehaveusedthefollowing:

    CommaDelimitedList,whichwewillconvenientlyquerylaterwithaspecialfunctionAllowedValuesandMinValuetoenforceconstraintsNoEchoforpasswordsorothersensitivedataSomeAWS-specifictypestohaveCloudFormationfurthervalidateinput

    Youwillnoticethattherearenovaluesassignedtoanyoftheprecedingparameters.

    Tomaintainareusabletemplate,wewillstorevaluesinaseparatefile(parameters.json):

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/parameters.json.

    [{"ParameterKey":"vpcCidr","ParameterValue":"10.0.0.0/16"},{

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/parameters.json

  • "ParameterKey":"vpcName","ParameterValue":"CloudFormation"},{"ParameterKey":"awsAvailabilityZones","ParameterValue":"us-east-1b,us-east-1c"},{"ParameterKey":"publicCidr","ParameterValue":"10.0.1.0/24,10.0.3.0/24"},{"ParameterKey":"privateCidr","ParameterValue":"10.0.2.0/24,10.0.4.0/24"},{"ParameterKey":"rdsIdentifier","ParameterValue":"cloudformation"},{"ParameterKey":"rdsStorageSize","ParameterValue":"5"},{"ParameterKey":"rdsStorageType","ParameterValue":"gp2"},{"ParameterKey":"rdsEngine","ParameterValue":"postgres"},...

    Resources

    Youarealreadyfamiliarwiththeconceptofresourcesandhowtheyareusedtodescribedifferentpiecesofinfrastructure.

    Regardlessofhowresourcesappearinatemplate,CloudFormationwillfollowitsinternallogictodecidetheorderinwhichthesegetprovisioned.

    Thesyntaxfordeclaringaresourceisasfollows:

    "LogicalID":{"Type":"","Properties":{}}

    IDsneedtobealphanumericanduniquewithinthetemplate.

    ThelistofCloudFormationresourcetypesandtheirpropertiescanbefoundhere:http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html

    http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html

  • Themaxnumberofresourcesatemplatecanhaveis200.Reachingthatlimit,youwillneedtosplitatemplateintosmalleronesandpossiblylookintonestedstacks.

    Backtoourexample,aspertraditionwestartbycreatingaVPCanditssupportingelementssuchassubnets,InternetgatewayandNATgateway:

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/main.json.

    "Resources":{"vpc":{"Type":"AWS::EC2::VPC","Properties":{"CidrBlock":{"Ref":"vpcCidr"},"EnableDnsSupport":"true","EnableDnsHostnames":"true","Tags":[{"Key":"Name","Value":{"Ref":"vpcName"}}]}},"publicSubnet1":{"Type":"AWS::EC2::Subnet","Properties":{"AvailabilityZone":{"Fn::Select":["0",{"Ref":"awsAvailabilityZones"}]},"CidrBlock":{"Fn::Select":["0",{"Ref":"publicCidr"}]},"MapPublicIpOnLaunch":"true","Tags":[{"Key":"Name","Value":"Public"}],"VpcId":{"Ref":"vpc"}}},..."internetGateway":{"Type":"AWS::EC2::InternetGateway","Properties":{"Tags":[{"Key":"Name","Value":{"Fn::Join":["-",[{"Ref":"vpcName"},"IGW"]]}}]}},"internetGatewayAttachment":{"Type":"AWS::EC2::VPCGatewayAttachment","Properties":{"InternetGatewayId":{"Ref":"internetGateway"},"VpcId":{"Ref":"vpc"}}},"natEip":{"Type":"AWS::EC2::EIP","Properties":{"Domain":"vpc"}},"natGateway":{"Type":"AWS::EC2::NatGateway",

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/main.json

  • "Properties":{"AllocationId":{"Fn::GetAtt":["natEip","AllocationId"]},"SubnetId":{"Ref":"publicSubnet1"}},"DependsOn":"internetGatewayAttachment"},

    NotesomeoftheCloudFormationfunctionsusedintheprecedingcode:

    "Fn::Select"in"CidrBlock":{"Fn::Select":["0",{"Ref":"publicCidr"}]},whichallowsustoquerytheCommaDelimitedListtypeparameterswesetearlier"Fn::Join",forconcatenatingstrings"Fn::GetAtt",forretrievingresourceattributes

    Also,theDependsOnpropertyofthenatGatewayresourceallowsustosetexplicitconditionsontheorderofexecution.Inthiscase,wearesayingthattheInternetGatewayresourceneedstobeready(attachedtotheVPC)beforetheNATGatewayisprovisioned.

    AftertheVPC,let'saddRDS:

    "rdsInstance":{"Type":"AWS::RDS::DBInstance","Properties":{"DBInstanceIdentifier":{"Ref":"rdsIdentifier"},"DBInstanceClass":{"Ref":"rdsInstanceClass"},"DBSubnetGroupName":{"Ref":"rdsSubnetGroup"},"Engine":{"Ref":"rdsEngine"},"EngineVersion":{"Ref":"rdsEngineVersion"},"MasterUserPassword":{"Ref":"rdsPassword"},"MasterUsername":{"Ref":"rdsUsername"},"StorageType":{"Ref":"rdsStorageType"},"AllocatedStorage":{"Ref":"rdsStorageSize"},"VPCSecurityGroups":[{"Ref":"rdsSecurityGroup"}],"Tags":[{"Key":"Name","Value":{"Ref":"rdsIdentifier"}}]}}

    ThenaddtheELB:

    ..."elbInstance":{"Type":"AWS::ElasticLoadBalancing::LoadBalancer","Properties":{"LoadBalancerName":"cloudformation-elb","Listeners":[{"InstancePort":"80","InstanceProtocol":"HTTP","LoadBalancerPort":"80","Protocol":"HTTP"}],"SecurityGroups":[{"Ref":"elbSecurityGroup"}],"Subnets":[{"Ref":"publicSubnet1"},{"Ref":"publicSubnet2"}],"Tags":[{"Key":"Name","Value":"cloudformation-elb"}]}}

    AndaddtheEC2resources:

  • ..."launchConfiguration":{"Type":"AWS::AutoScaling::LaunchConfiguration","Properties":{"ImageId":{"Ref":"autoscalingGroupImageId"},"InstanceType":{"Ref":"autoscalingGroupInstanceType"},"KeyName":{"Ref":"autoscalingGroupKeyname"},"SecurityGroups":[{"Ref":"ec2SecurityGroup"}]

    WestilluseaUserDatashellscripttoinstalltheNGINXpackage;however,thepresentationisslightlydifferentthistime.CloudFormationisgoingtoconcatenatethelinesusinganewlinecharacterasadelimiterthenencodetheresultinBase64:

    "UserData":{"Fn::Base64":{"Fn::Join":["\n",["#!/bin/bash","set-euf-opipefail","exec1>>(logger-s-t$(basename$0))2>&1","yum-yinstallnginx;chkconfignginxon;servicenginxstart"]]}}}}

    WeuseDependsOntoensuretheRDSinstancegoesinbeforeautoScalingGroup:

    "autoScalingGroup":{"Type":"AWS::AutoScaling::AutoScalingGroup","Properties":{"LaunchConfigurationName":{"Ref":"launchConfiguration"},"DesiredCapacity":"1","MinSize":"1","MaxSize":"1","LoadBalancerNames":[{"Ref":"elbInstance"}],"VPCZoneIdentifier":[{"Ref":"privateSubnet1"},{"Ref":"privateSubnet2"}],"Tags":[{"Key":"Name","Value":"cloudformation-asg","PropagateAtLaunch":"true"}]},"DependsOn":"rdsInstance"}

    Outputs

    Again,wewillusethesetohighlightsomeresourceattributesfollowingasuccessfuldeployment.AnotherimportantfeatureofOutputs,however,isthattheycanbeusedasinputparametersforothertemplates(stacks).Thisbecomesveryusefulwithnestedstacks.

    Note

  • Oncedeclared,Outputscannotbesubsequentlyupdatedontheirown.YouwillneedtomodifyatleastoneresourceinordertotriggeranOutputupdate.

    WeaddtheVPCID,NATIPaddressandELBDNSnameasOutputs:

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/main.json.

    "Outputs":{"vpcId":{"Description":"VPCID","Value":{"Ref":"vpc"}},"natEip":{"Description":"NATIPaddress","Value":{"Ref":"natEip"}},"elbDns":{"Description":"ELBDNS","Value":{"Fn::GetAtt":["elbInstance","DNSName"]}}}

    Currently,atemplatecanhavenomorethan60Outputs.

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/blob/master/5585_02_CodeFiles/CloudFormation/main.json

  • OperationsIfyouhavebeenfollowingalong,youshouldnowhaveamain.jsonandaparameters.jsoninyourcurrentfolder.Itistimetoputthemtouse,sohereareafewoperationswearegoingtoperform:

    ValidateatemplateDeployastackUpdateastackDeleteastack

    Templatevalidation

    Firstthingsfirst,abasiccheckofourJSONtemplatewithvalidate-template:

    $awscloudformationvalidate-template--template-bodyfile://main.json{"Description":"ProvisionsEC2,ELB,ASGandRDSresources","Parameters":[{"NoEcho":false,"Description":"EC2AMIID","ParameterKey":"autoscalingGroupImageId"}

    Ifthere'snoerrors,theCLIreturnstheparsedtemplate.Notethatwecouldhavejustaseasilypointedtoaremotelocationusing--template-urlinsteadof-template-body.

    DeployingaStack

    Todeployourtemplate(stack),wewillusecreate-stack.Ittakesanarbitraryname,thelocationofthetemplate,andthefilecontainingparametervalues:

    $awscloudformationcreate-stack--stack-namecfn-test--template-bodyfile://main.json--parametersfile://parameters.json{"StackId":"arn:aws:cloudformation:us-east-1:xxxxxx:stack/cfn-test/xxxxxx"}

    CloudFormationstartscreatingthestackandnofurtheroutputisreturned.TogetprogressinformationontheCLI,usedescribe-stacks:

    $awscloudformationdescribe-stacks--stack-namecfn-test{"Stacks":[{"StackId":"arn:aws:cloudformation:us-east-xxxxxx:stack/cfn-test/xxxxxx"..."CreationTime":"2016-05-29T20:07:17.813Z","StackName":"cfn-test","NotificationARNs":[],"StackStatus":"CREATE_IN_PROGRESS",

  • "DisableRollback":false}]}

    Andforevenmoredetails,usedescribe-stack-events.

    Afterafewminutes(basedonoursmalltemplate)StackStatuschangesfromCREATE_IN_PROGRESStoCREATE_COMPLETEandweareprovidedtherequestedOutputs:

    $awscloudformationdescribe-stacks--stack-namecfn-test"Outputs":[{"Description":"VPCID","OutputKey":"vpcId","OutputValue":"vpc-xxxxxx"},{"Description":"NATIPaddress","OutputKey":"natEip","OutputValue":"x.x.x.x"},{"Description":"ELBDNS","OutputKey":"elbDns","OutputValue":"cloudformation-elb-xxxxxx.us-east-1.elb.amazonaws.com"}],"CreationTime":"2016-05-29T20:07:17.813Z","StackName":"cfn-test","NotificationARNs":[],"StackStatus":"CREATE_COMPLETE","DisableRollback":false

    Atthispoint,theelbDNSURLshouldreturnthenginxwelcomepage,asshownhere:

    Ifnot,youmightneedtoallowsomemoretimefortheEC2nodetofullyinitialize.

  • Updatingastack

    CloudFormationofferstwowaysofupdatingadeployedstack.

    Note

    Someupdateoperationscanbedestructive(pleaserefertohttp://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html).YoushouldalwayschecktheCloudFormationdocumentationontheresourceyouareplanningtomodifytoseewhetherachangeisgoingtocauseanyinterruption.

    Ifyouwouldliketoquicklydeployaminorchange,thenallyouneedtodoismodifythetemplatefileanddeployitdirectlywithupdate-stack:

    $awscloudformationupdate-stack--stack-namecfn-test--template-bodyfile://main.json--parametersfile://parameters.json

    Otherwise,agoodpracticewouldbetouseChangeSetstopreviewstackchangesbeforedeployingthem.Forexample,letusupdatetherulesintheELBsecuritygroupaswedidbefore:

    1. Modifythemain.jsontemplate(addanotherruletoelbSecurityGroup):

    "elbSecurityGroup":{"Type":"AWS::EC2::SecurityGroup","Properties":{"SecurityGroupIngress":[{"ToPort":"80","FromPort":"80","IpProtocol":"tcp","CidrIp":"0.0.0.0/0"},{"ToPort":"443","FromPort":"443","IpProtocol":"tcp","CidrIp":"0.0.0.0/0"}]

    2. CreateaChangeSet:

    $awscloudformationcreate-change-set--change-set-nameupdatingElbSecGroup--stack-namecfn-test--template-bodyfile://main.json--parametersfile://parameters.json

    3. PreviewtheChangeSet:

    $awscloudformationdescribe-change-set--change-set-nameupdatingSecGroup--stack-namecfn-test

    4. ExecutetheChangeSet:

    $awscloudformationexecute-change-set--change-set-nameupdatingSecGroup--stack-namecfn-test

    Tip

    http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html

  • WhetherviaaChangeSetorupdatingdirectly,ifyouaresimplymodifyingparametervalues(parameters.json)youcanskipre-uploadingthetemplate(main.json)with--use-previous-template.

    Deletingastack

    Inordertotidyupafterourexperiments,wewillneedtogranttemporaryAdminprivilegestotheCloudFormationIAMuser(thesameprocedureasintheearlierTFsection);rundelete-stack:

    $awscloudformationdelete-stack--stack-namecfn-test

    ThenrevoketheAdminprivileges.

  • SummaryInthischapter,welookedattheimportanceandusefulnessofInfrastructureasCodeandwaystoimplementitusingTerraformorAWSCloudFormation.

    WeexaminedthestructureandindividualcomponentsofbothaTFandaCFtemplatethenpracticeddeployingthoseontoAWSusingtheCLI.Itrustthattheexampleswewentthroughhavedemonstratedthebenefitsandimmediategainsfromthepracticeofdeployinginfrastructureascode.

    Sofar,however,wehaveonlydonehalfthejob.Withtheprovisioningstagecompleted,youwillnaturallywanttostartconfiguringyourinfrastructure,andthatiswhatwearegoingtodointhenextchapteronConfigurationManagement.

  • Chapter3.BringingYourInfrastructureUnderConfigurationManagementAshintedattheendofthepreviouschapter,thereissomemoreworktobedonebeforewecanclaimtohavefullyimplementedIaC.

    Thefirststepwastodescribethehardwaresideofourinfrastructureincode;nowitistimetolookatthesoftwareorconfigurationaspectofit.

    LetussaywehaveprovisionedafewEC2nodesandwouldliketohavecertainpackagesinstalledonthem,andrelevantconfigurationfilesupdated.PriortoConfigurationManagement(CM)toolsgainingpopularity,suchtaskswouldhavebeenperformedmanuallybyanengineereitherfollowingachecklist,runningacollectionofshellscripts,orboth.Asyoucanimagine,suchmethodsdonotscalewellastheygenerallyimplyoneengineersettinguponeserveratatime.

    Inaddition,checklistsorscripts:

    ArehardtowritewhenitcomestoconfiguringahostplusafullapplicationstackrunningonitAreusuallytargetedatagivenhostorapplicationandarenotveryportableGetprogressivelyhardertocomprehendthefurtheryougetfromthepersonwhooriginallywrotethemBuildscriptstendtogetexecutedonlyonce,usuallyatthetimeahostisprovisioned,thusconfigurationstartstodriftfromthatmomenton

    Fortunately,notmanypeopleusethesenowadays,asConfigurationManagementhasbecomeacommonpractice.Letusexaminesomeofthebenefits:

    CMallowsustodeclarethedesiredstateofamachineonceandthenreproducethatstateasoftenasnecessaryPowerfulabstractiontakescareofspecificssuchasenvironment,hardware,andOStype,allowingustowritereusableCMcodeThedeclaredmachinestatecodeiseasytoread,comprehend,andcollaborateon.ACMdeploymentcanbeperformedontens,hundreds,orthousandsofmachinessimultaneously

    InthisageofDevOps,thereareavarietyofCMtoolstochoosefrom.YoumighthavealreadyheardofPuppet,Chef,Ansible,OpsWorks,ortheonewearegoingtouse-SaltStack(theSaltOpenproject).

    Allofthesearewelldeveloped,sophisticatedCMsolutionswithactivecommunitiesbehindthem.Ifindithardtojustifyanyreportedclaimsofonebeingbetterthantherestastheyalldothejobprettywell,eachwithitsownsetofstrengthsandweaknesses.Sowhichoneyouuse,asis

  • oftenthecase,isuptopersonalpreference.

    Regardlessofthetoolyouendupusing,Iwouldliketostresstheimportanceoftwopoints:namingconventionsandcodereusability.

    Followingnamingconventionswhenwritingcodeisanobviouswinasitguaranteesotherpeoplewillbeabletounderstandyourworkwithlesseffort.Inadditiontowritingcodehowever,CMinvolvesexecutingitagainstyournodesandthisiswherenamingalsobecomesimportant.Imagineyouhadfourservers:leonardo,donatello,michelangelo,andraphael.Twoofthoseareyourfrontendlayerandtwothebackend,soyousitdownandwriteyourConfigurationManagementmanifestsrespectively:webserver-nodeanddatabase-node.Sofar,sogood,giventhenumberofhostsyoucanlaunchyourCMtoolandeasilytellittoruntherelevantmanifestagainsteachofthem.

    Nowimagine50,then100hosts,withinasimilarflat-namingschema,andyoustarttoseetheproblem.Asthesizeandcomplexityofyourinfrastructuregrows,youwillneedahost-namingconventionthatnaturallyformsahierarchy.Hostnamessuchaswebserver-{0..10},db-{0..5}andcache-{0..5}canbefurthergroupedintofrontendandbackendandthenrepresentedinastructured,hierarchicalway.SuchawayofgroupingnodesbasedonroleorotherpropertiesisextremelyusefulwhenapplyingConfigurationManagement.

    CodereusabilityshouldalreadybeonyourmindwhenyoustartwritingCMcode(manifests).Youwillfindthattherearegenerallytwowaysofapproachingthistask.Youcouldwritealarge,say,webserverpiecewhichcontainsinstructionsonhowtosetupthefirewall,someCLItools,NGINX,andPHPonanode,oryoucouldbreakitdownintosmallerpartslikeiptables,utils,NGINX,PHP,andsoon.

    Inmyopinion,thelatterdesignaddssomeoverheadwhenwritingthemanifests,butthebenefitofreusabilityissubstantial.Insteadofwritinglargesetsofdeclarationsdedicatedtoeachservertype,youmaintainacollectionofgeneric,smallonesandcherry-pickfromthemtosuitthemachineinquestion.

    Toillustrate:

    manifests:everything_a_websrv_needs,everything_for_a_db,cache_mainnodes:web01,db01,cache01CM_execution:web01=(everything_a_websrv_needs),db01=(everything_for_a_db),cache01=(cache_main)

    Orbetter:

    manifests:iptables,utils,nginx,postgresql,redis,phpnodes:web01,db01,cache01CM_execution:web01=(iptables,utils,nginx,php),db01=(iptables,utils,postgresql),cache01=(iptables,utils,redis)

  • IntroductiontoSaltStackSaltStack(seehttps://saltstack.com/),firstreleasedin2011,isanautomationsuitewhichoffersConfigurationManagementplusstandardand/orevent-drivenorchestration.Itiscommonlyusedinamaster-minionsetup,whereamasternodeprovidescentralizedcontrolacrossacomputeestate.Itisknownforitsspeedandscalabilitythankstothefastandlightweightmessagebus(ZeroMQ)usedforcommunicationbetweenthesalt-masterandminions.Itcanalsobeusedinanagentlessfashion,wheretheminionsarecontrolledoverSSH,similarlytohowAnsibleoperates.

    SaltStackiswritteninPythonandiseasilyextensible.Youcanwriteyourownmodulesforit,attachlong-runningprocessestoitseventbus,andinjectrawPythoncodeinunusualplaces.

    Themaster-minionmodelisquitepowerful,offersalotofflexibility,andistherecommendedapproachifyouarelookingafteranythingmorethanafewdevnodesandwanttotakeadvantageofallthefeaturesSaltStackhastooffer.

    Note

    Moreonhowtogetasalt-masterupandrunningcanbefoundhere:https://docs.saltstack.com/en/latest/topics/configuration/index.html

    Inourcase,wearegoingtoexplorethepowerofConfigurationManagementusingSaltStackinastandaloneormasterlessmode.WewillreusepartsoftheTerraformtemplatefromthepreviouschaptertolaunchasetofEC2resources,bootstrapaSaltStackminionandhaveitconfigureitselftoserveawebapplication.

    Providedallgoeswell,weshouldendupwithafullyconfiguredwebserver(EC2node)behindaload-balancer(EC2ELB).

    Hereisourtask-list:

    1. PrepareourSaltStackdevelopmentenvironment.2. WritetheconfigurationthatwewouldlikeSaltStacktoapplytoournode(s).3. ComposetheTerraformtemplatedescribingourinfrastructure.4. DeploytheinfrastructureviaTerraformandletSaltStackconfigureit.

    https://saltstack.com/https://docs.saltstack.com/en/latest/topics/configuration/index.html

  • PreparationSaltStackConfigurationManagementisperformedusingthefollowingmaincomponents:

    Statesarethefileswhichdescribethedesiredstateofamachine.Herewewriteinstructionsforinstallingpackages,modifyingfiles,updatingpermissions,andsoon.PillarsarethefilesinwhichwedefinevariablestohelpmakeStatesmoreportableandflexible.Grainsarepiecesofinformationgatheredontheminionhostitself.TheseincludedetailsabouttheOS,environment,thehardwareplatform,andothers.TheSaltFileServerstoresanyfiles,scripts,orotherartifactswhichmaybereferencedintheStates.TheSaltTopfile(s)areusedtomapStatesand/orPillarstominions.

    Inamaster-minionsetup,allofthesecomponentsexcepttheGrainswouldbehostedonandmadeavailabletotheminionsbythesalt-master(otherbackendsarealsosupported).

    WeareplanningtorunSaltinmasterlessmodehowever,meaningthatwewillneedawaytotransferanyStates,Pillars,andrelatedfilesfromourlocalenvironmenttotheminion.Git?Goodidea.WewillwriteallSaltcodelocally,pushittoaGitrepository,andthenhaveitcheckedoutontoeachminionatboottime.

    AsforchoosingaGithostingsolution,GithuborBitbucketareexcellentservices,butgivingourminionEC2nodesaccesstothesewillinvolvesomekeyhandling.Incomparison,CodeCommit(theAWSGitsolution)offersamuchsmootherintegrationwithEC2instancesviaIAMRoles.

    LetusstartbycreatinganewIAMuserandaCodeCommitGitrepository.Wewillbeusingtheuser'saccesskeystocreatetherepositoryandaSSHkeytocloneandworkwithit:

    1. IntheAWSConsole,createanIAMuser(writedownthegeneratedaccesskeys)andattachtheAWSCodeCommitFullAccessbuilt-in/ManagedIAMpolicytoitasshowninthefollowingscreenshot:

    2. Onthesamepage,switchtotheSecurityCredentialstabandclickontheUploadSSH

  • publickeyasshowninthefollowingscreenshot:

    3. Configureawscli:

    $exportAWS_ACCESS_KEY_ID='AKIAHNPFB9EXAMPLEKEY'$exportAWS_SECRET_ACCESS_KEY='rLdrfHJvfJUHY/B7GRFTY/VYSRwezaEXAMPLEKEY'$exportAWS_DEFAULT_REGION='us-east-1'

    4. Createarepository:

    $awscodecommitcreate-repository--repository-namesalt--repository-description"SaltStackrepo"{"repositoryMetadata":{"repositoryName":"salt","cloneUrlSsh":"ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/salt","lastModifiedDate":1465728037.589,"repositoryDescription":"SaltStackrepo","cloneUrlHttp":"https://git-codecommit.us-east-1.amazonaws.com/v1/repos/salt","creationDate":1465728037.589,"repositoryId":"d0628373-d9a8-44ab-942a-xxxxxx","Arn":"arn:aws:codecommit:us-east-1:xxxxxx:salt","accountId":"xxxxxx"}}

    5. Clonethenewrepositorylocally:

    $gitclonessh://[email protected]/v1/repos/saltCloninginto'salt'...warning:Youappeartohaveclonedanemptyrepository.

  • Checkingconnectivity...done.

    Here,SSH_KEY_IDistheonewesawafteruploadingapublickeyinstep2.

    Note

    FormoreoptionsonconnectingtoCodeCommitseehttp://docs.aws.amazon.com/codecommit/latest/userguide/setting-up.html

    Wearereadytostartpopulatingourempty,newSaltrepository.

    http://docs.aws.amazon.com/codecommit/latest/userguide/setting-up.html

  • WritingConfigurationManagementcodeForSaltStacktohelpusconfigureournodeasawebserver,weneedtotellitwhatoneofthoseshouldlooklike.InConfigurationManagementterms,weneedtodescribethedesiredstateofthemachine.

    Inourexample,wewillbeusingacombinationofSaltStackStates,Pillars,Grains,andTopfilestodescribetheprocessesof:

    CreatingLinuxuseraccountsInstallingservices(NGINXandPHP-FPM)Configuringandrunningtheinstalledservices

  • StatesAStatecontainsasetofinstructionswhichwewouldliketobeappliedtoourEC2minion(s).Wewilluse/srv/salt/statesontheminionastherootoftheSaltStatetree.Statescanbestoredinthereintheformofasinglefile,forexample/srv/salt/states/mystate.sls,ororganizedintofolderslikeso/srv/salt/states/mystate/init.sls.Lateron,whenwerequestthatmystateisexecuted,Saltwilllookforeitherastate_name.slsorastate_name/init.slsintherootoftheStateTree.Ifindthesecondapproachtidierasitallowsforotherstate-relatedfilestobekeptintherelevantfolder.

    WebegintheConfigurationManagementofourwebservernodewithastateformanagingLinuxuseraccounts.InsideourSaltGitrepository,wecreatestates/users/init.sls:

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/tree/master/5585_03_CodeFiles/CodeCommit/salt/states/users.

    veselin:user.present:-fullname:VeselinKantsev-uid:{{salt['pillar.get']('users:veselin:uid')}}-password:{{salt['pillar.get']('users:veselin:password')}}-groups:-wheelssh_auth.present:-user:veselin-source:salt://users/files/veselin.pub-require:-user:veselinsudoers:file.managed:-name:/etc/sudoers.d/wheel-contents:'%wheelALL=(ALL)ALL'

    WewilluseYAMLtowritemostSaltconfiguration.Youwillnoticethreedifferentstatemodulesusedintheprecedingsection:

    user.present:Thismoduleensuresthatagivenuseraccountexistsonthesystemorcreatesoneifnecessaryssh_auth.present:AmoduleformanagingtheSSHauthorized_keysfileofauserfile.managed:Amoduleforcreating/modifyingfiles

    Note

    SaltStack'sstatemodulesofferrichfunctionality.Forfulldetailsofeachmoduleseehttps://docs.saltstack.com/en/latest/ref/states/all/

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/tree/master/5585_03_CodeFiles/CodeCommit/salt/states/usershttps://docs.saltstack.com/en/latest/ref/states/all/

  • Toavoidhardcodingcertainvaluesunderuser.present,wemakeuseoftheSaltStackPillarssystem.Wewillexamineapillarfileshortly,butfornowjustnotethesyntaxofreferencingpillarvaluesinsideourstate.

    Twootherpointsofinterestherearethesourceofourkeyfileandtherequireproperty.Inthisexample,asalt://formattedsourceaddressreferstotheSaltFileServerwhichbydefaultservesfilesfromtheStateTree(forsupportedbackends,pleaseseehttps://docs.saltstack.com/en/latest/ref/file_server/).Therequirestatementenforcesanorderofexecution,ensuringthattheuseraccountispresentbeforetryingtocreateanauthorized_keysfileforit.

    Note

    SaltStackfollowsanimperativeexecutionmodeluntilsuchcustomorderingisenforced,invokingadeclarativemode(seehttps://docs.saltstack.com/en/latest/ref/states/ordering.html).

    ThankstothereadabilityofYAML,onecaneasilytellwhatisgoingonhere:

    1. WecreateanewLinuxuser.2. Weapplydesiredattributes(uid,password,group,andsoon).3. WedeployanSSHauthorized_keysfileforit.4. Weenablesudoforthewheelgroupofwhichtheuserisamember.

    Perhapsyoucouldtryeditthisstateandaddauserforyourself?Itwillbeusefullaterafterwedeploy.

    WewillnowmoveontoanNGINXinstallationviastates/nginx/init.sls.

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/tree/master/5585_03_CodeFiles/CodeCommit/salt/states/nginx.

    WeinstallNGINXusingthepkg.installedmodule:

    pkg.installed:[]

    Settheservicetostartonboot(enable:True),enablereloadinginsteadofrestartingwhenpossible(reload:True),ensuretheNGINXpkghasbeeninstalled(require:)beforerunningtheservice(service.running:)

    nginx:service.running:-enable:True-reload:True-require:-pkg:nginx

    https://docs.saltstack.com/en/latest/ref/file_server/https://docs.saltstack.com/en/latest/ref/states/ordering.htmlhttps://github.com/PacktPublishing/Implementing-DevOps-on-AWS/tree/master/5585_03_CodeFiles/CodeCommit/salt/states/nginx

  • Thenputaconfigfileinplace(file.managed:),ensuringtheservicewaitsforthistohappen(require_in:)andalsoreloadseachtimethefileisupdated(watch_in:):

    /etc/nginx/conf.d/default.conf:file.managed:-source:salt://nginx/files/default.conf-require:-pkg:nginx-require_in:-service:nginx-watch_in:-service:nginx

    Notetherequire/require_in,watch/watch_inpairs.Thedifferencebetweeneachoftheserequisitesandits_incounterpartliesinthedirectioninwhichtheyact.

    Forexample:

    nginx:service.running:-watch:-file:nginx_confignginx_config:file.managed:-name:/etc/nginx/nginx.conf-source:salt://...

    Hasthesameeffectas:

    nginx:service.running:[]nginx_config:file.managed:-name:/etc/nginx/nginx.conf-source:salt://...-watch_in:-service:nginx

    Inbothcases,theNGINXservicerestartsonconfigfilechanges;however,youcanseehowthesecondformatcanbepotentiallyquiteusefulthefurtheryougetfromtheserviceblock-sayinadifferentfile,aswewillseeinthenextstate.

    AddinsomePHP(states/php-fpm/init.sls):

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/tree/master/5585_03_CodeFiles/CodeCommit/salt/states/php-fpm.

    include:-nginxphp-fpm:

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/tree/master/5585_03_CodeFiles/CodeCommit/salt/states/php-fpm

  • pkg.installed:-name:php-fpm-require:-pkg:nginxservice.running:-name:php-fpm-enable:True-reload:True-require_in:-service:nginx...

    Hereyoucanbetterseetheusefulnessofan_inrequisite.Afterweincludethenginxstateatthetop,ourrequire_inmakessurethatnginxdoesnotstartbeforephp-fpmdoes.

    WithNGINXandPHP-FPMnowconfigured,letusaddaquicktestpage(states/phptest/init.sls).

    Note

    Pleasereferto:https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/tree/master/5585_03_CodeFiles/CodeCommit/salt/states/phptest.

    WesetafewvariablespulledfromGrains(moreonthoseshortly):

    {%setpublqic_ipv4=salt['cmd.shell']('ec2-metadata--public-ipv4|awk'{print$2}'')%}{%setgrains_ipv4=salt['grains.get']('ipv4:0')%}{%setgrains_os=salt['grains.get']('os')%}{%setgrains_osmajorrelease=salt['grains.get']('osmajorrelease')%}{%setgrains_num_cpus=salt['grains.get']('num_cpus')%}{%setgrains_cpu_model=salt['grains.get']('cpu_model')%}{%setgrains_mem_total=salt['grains.get']('mem_total')%}

    Thenwedeploythetestpageandaddcontentstoitdirectly:

    phptest:file.managed:-name:/var/www/html/index.php-makedirs:True-contents:|

    Wewillusethispagepost-deploymenttocheckwhetherbothNGINXandPHP-FPMareoperational.

    https://github.com/PacktPublishing/Implementing-DevOps-on-AWS/tree/master/5585_03_CodeFiles/CodeCommit/salt/states/phptest

  • PillarsNowletuslookatthemainmechanismforstoringvariablesinSalt-thePillars.Theseare:

    YAMLtree-likedatastructuresDefined/renderedonthesalt-master,unlessrunningmasterlessinwhichcasetheyliveontheminionUsefulforstoringvariablesinacentralplacetobesharedbytheminions(unlesstheyaremasterless)HelpfulforkeepingStatesportableAppropriateforsensitivedata(theycanalsobeGPGencrypted;seehttps://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.gpg.html)

    Wewillbeusing/srv/salt/pillarsastherootofourPillartreeontheminion.Letusgobacktotheusersstateandexaminethefollowinglines:

    -uid:{{salt['pillar.get']('users:veselin:uid')}}-password:{{salt['pillar.get']('users:veselin:password')}}

    Theuidandpasswordattributesaresettobesourcedfromapillarnamedusers.AndifwecheckourPillarTree,wefinda/srv/salt/pillars/users.slsfilecontaining:

    users:veselin:uid:5001password:'$1$wZ0gQOOo$HEN/gDGS85dEZM7QZVlFz/'

    Itisnoweasytoseehowtheusers:veselin:passwordreferenceinsidethestatefilematchesagainstthispillar'sstructure.

    Note

    Formoredetailsandexamplesonpillarusage,see:https://docs.saltstack.com/en/latest/topics/tutorials/pillar.html

    https://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.gpg.htmlhttps://docs.saltstack.com/en/latest/topics/tutorials/pillar.html

  • GrainsUnlikePillars,Grainsareconsideredstaticdata:

    Theygetgeneratedminion-sideandarenotsharedbetweendifferentminionsTheycontainfactsabouttheminionitselfTypicalexamplesareCPU,OS,networkinterfaces,memory,andkernelsItispossibletoaddcustomGrainstoaminion

    WehavealreadymadegooduseofGrainswithinourprecedingtestpage(s


Related Documents
avanti.in...ABHINAV GOUD BINGI ABHINAV GUPTA ABHINAV SHARMA ABHINAV SHARMA ABHISHEK AGARWAL ABHISHEK BANSAL ABHISHEK JAIN ABHISHEK KUMAR SHUKLA ABHISHEK MITTAL ABHISHEK P KHANNUR ...

avanti.in...ABHINAV GOUD BINGI ABHINAV GUPTA ABHINAV SHARMA....

Category: Documents
Music Theory Through Improvisation - pdf.ebook777.compdf.ebook777.com/058/9780415997256.pdf · Modal Etude 1 52 Two-player ... Multiple Configurations of Extended Voicings:A and B

Music Theory Through Improvisation -...

Category: Documents
pdf.ebook777.compdf.ebook777.com/066/8496492389.pdfestudio del Palo japonés, el Bo. Se trata de un libro multidisciplinar, pues el Bo es un arma que comparte tradición tanto en el

pdf.ebook777.compdf.ebook777.com/066/8496492389.pdfestudio.....

Category: Documents
Chess Openings for Kids - pdf.ebook777.compdf.ebook777.com/054/1906454264.pdf · Introduction Let's face it: it's more fun to win a game of chess than lose, and no fun at all to lose

Chess Openings for Kids -...

Category: Documents
Love among the Ruins - pdf.ebook777.compdf.ebook777.com/066/9780691095226.pdf · Love among the Ruins THE EROTICS OF DEMOCRACY IN CLASSICAL ATHENS Victoria Wohl PRINCETON UNIVERSITY

Love among the Ruins -...

Category: Documents
The creation of Wing Chun - pdf.ebook777.compdf.ebook777.com/065/9781438456942.pdf · For all he has taught the world about Wing Chun, for his enthusiasm and never-ending curiosity,

The creation of Wing Chun -...

Category: Documents
Easy Guitar Scales and Modes - pdf.ebook777.compdf.ebook777.com/071/B06XTBCTSB.pdf · Guitar scales are groups of notes that outline a specific key center, mode, or chord color. Scales

Easy Guitar Scales and Modes -...

Category: Documents
Tiger Tank: Panzers on The Eastern Front - pdf.ebook777.compdf.ebook777.com/057/B01NB1XTS7.pdf · Panzers on the Eastern Front ... tank was not put out of action. ... remains finally

Tiger Tank: Panzers on The Eastern Front -...

Category: Documents
pdf.ebook777.compdf.ebook777.com/059/9781576235133.pdf · Hand-Foot Patterns BUILDING THE ... large degree by Pete DePoe's King Kong Beat the layered sticking concept ... Gary Chaffee

pdf.ebook777.compdf.ebook777.com/059/9781576235133.pdf ·.....

Category: Documents
Animating 2D Characters for - pdf.ebook777.compdf.ebook777.com/069/9781305501843.pdf · with office locations around the globe, including Singapore, the United ... Nemesis of Ragnarok.

Animating 2D Characters for -...

Category: Documents
pdf.ebook777.compdf.ebook777.com/066/9780821421260.pdf · Union of Communist Struggle. ... August 4, 1983, takeover, 50—53 and CNR, 53—54 ideology, 15, ... education, 72, 77,

pdf.ebook777.compdf.ebook777.com/066/9780821421260.pdf ·.....

Category: Documents
Abhishek K-Abhishek A

Abhishek K-Abhishek A

Category: Documents