SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Copyright©2016Splunk Inc.

KevinDaliankdalian@ford.com

GlenUpretiGlen.Upreti@Sierra-Cedar.com

Splunk GoneWild!– InnovatingALargeSplunk SolutionAtTheSpeedOfManagement

Disclaimer

2

Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose

containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor

functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.

Introductions

KevinDalianTeamLeadServerHostingTools

FordMotorCompanyNothingInterestingAbout Kevin,he’saboringwork-aholic.

GlenUpretiDirectorEnterpriseandCloudTechnologies

Sierra-CedarTerribleatJenga

Agenda

4

WhereWeCameFromWhereWePlannedtoGoWhereWeEndedUpInstallationOnBoardingDataWhatWe’reuptoNowQ&A

WhereWe Started2SplunkEnvironments– NetworkandServerOperationsServerOps– 4StandaloneSearchHead/Indexers– 3DeploymentServers– 20Gblicense– +11,500UniversalForwarders

WhereWePlannedtoGo

6

MajornewFordInitiative– FordPass/Connected-X– PCF– PivotalCloudFoundry– MicrosoftAzureCloud– Mixtureofinternalandexternalapplicationsanddata– 100Gb/day

WhereWePlannedtoGo(cont’d)

PCF

Azure

PublicInternet

PeerIndexer1

SearchHead1

DeploymentServer/DMC/LicenseMaster

SearchHead1 SearchHead3

PeerIndexer2 PeerIndexer MasterIndexer

SearchHeadMaster

ServerInfrastructure

SplunkForwarders

Syslogfirehose SyslogRelay

SearchHead/Indexer

JMXRESTSQL

SyslogReceiver(w/SplunkUF)

SyslogRelay

Syslog

Syslog

DMZ Intranet

UniversalForwarder

WhereWeEndedUp

8

MobileSearchHead2

Syslog

PCF-Prod

AzureNA2

SyslogReceiver(w/SplunkUF)

DMZ

Intranet

PublicInternet

PeerIndexer1

SearchHead1

DeploymentServer/BatchProcessor/

Archive/DMC/LicenseMaster

SearchHead2 SearchHead3

PeerIndexer2 PeerIndexer MasterIndexer

SearchHeadMaster

ServerInfrastructure

SplunkForwarders

FMCHeavyForwarder

Syslogfirehose

JMXRESTSQL

UniversalForwarder

SSL

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

PCF-Prod

AzureNA1SSL

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

TCP

SyslogRelay

SyslogRelay

SyslogRelay

Syslog

Syslog

UniversalForwarder

TCP

SSL

SSL

SearchHead/Indexer

SearchHead/IndexerSyslogfirehose

JMXRESTSQL

Syslogfirehose

JMXRESTSQL

PCFFMC-Prod

JMXRESTSQL

PCFFMC-

PreprodECCHeavyForwarder

Syslogfirehose

JMXRESTSQL

PCFECC-

Preprod

SyslogfirehoseJMXRESTSQL

PCF-Dev

PCF-Dev

MobileSearchHead1

PCF-Prod

21VCN1

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

SyslogRelay

SearchHead/IndexerSyslogfirehose

JMXRESTSQLPCF-Dev

SSL

SSL

syslog

Http

Android

FMCSearchHead/Indexer

ECCSearchHead/Indexer

PCF-Prod

21VCN2

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

SyslogRelay

SearchHead/IndexerSyslogfirehose

JMXRESTSQLPCF-Dev

syslog

SSL

SSL

Https

Https

Https

Https

Apple

Syslogfirehose

PCFECC-Prod

PCF-Prod

AzureEU1Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

PCF-Prod

AzureEU2

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

SyslogRelay

SyslogRelay

SearchHead/Indexer

SearchHead/IndexerSyslogfirehose

JMXRESTSQL

Syslogfirehose

JMXRESTSQL

PCF-Dev

PCF-Dev

SSL

SSL

SSL

SSL

syslog

syslog

InstallingGlenandKevinmeetandplanforinstallationtasksHaveaPOCenvironmentinAzureAzuretoon-premise,HOW???Startedwithtemporarystandaloneinstance

Installing(Surprises)Hardwareshowsup&everythingfallsapart– Hardwarearrivedpiecemeal– NotenoughCPUs– AzureVMs,wewerethefirsttemplateinstall– Serversin‘Public’DMZweren’tpubliclyaccessible

EvenwithissuesSHClusterandIDXClusterallinstalledwithindays!

OnBoarding DataOnboardeddatafrom– Pivotal CloudFoundry– MicrosoftAzurePAASviaDBConnect– Thirdpartyandcustomdevelopedinputs

Onboarding DataWhen onboardingalwaysset– TIME_PREFIX– TIME_FORMAT– MAX_TIMESTAMP_LOOKAHEAD– SHOULD_LINEMERGE– LINE_BREAKER– TRUNCATE

Onboarding Data(Surprises)‘Ohbytheway…’– Newinputs– Newregions– Newenvironments(pre-production)– Newteams– NewSplunkLicense– SensitiveData- Needforobfuscation

GotData,NowWhat?PrototypedDashboardwrapupquick.WhatDoesThisMean?– Engagedevelopers andusercommunities

KeepCreating– Alwaysbemovingforward

Alerts– Alertingisaniterativeprocess– Bepreparedforalotofnoiseatfirst– Refine,refine,refine

GotData,NowWhat?(Surprises)Surprises:– MassiveDashboards– NewUsersandRoles– DataSecurity– RetentionTimes

Ohyeah,andmobile…

MobileMadnessSplunkAdd-onforMobileAccess– CrazyEasy!InitialPOCinAzureworkedlikeachampPlanned,preparedandmovedtoDMZNotificationsdon’twork– Newmanagementsurprise…thekindyoudon’twantBacktothedrawingboard

MobileMadness(TempSolution)

PCF

Azure

PublicInternet

PeerIndexer1

SearchHead1

DeploymentServer/DMC/LicenseMaster

SearchHead1 SearchHead3

PeerIndexer2 PeerIndexer MasterIndexer

SearchHeadMaster

Syslogfirehose SyslogRelay

SearchHead/Indexer

JMXRESTSQL

DMZ Intranet

UniversalForwarder

Http

AndroidApple

MobileMadness(Eventually)

WhereAreWeNow?PlanningfortheFuture/ScalingRefininganddocumentingMigratingdata/appsfromoriginalenvironmentExpandingthecustomerbaseStillrefiningdashboardsRe-sourcetypingPreparingformoremanagementshenanigans

AdviceMovingForward1. Insistnon-productionenvironment2. Workwiththecustomertofurtherunderstandingofdata3. Define/DocumentallCustomerrequirementsandgetsignoff4. Avoidthedatagraveyard5. Splunkisveryflexible,keepanopenmindandstaycalm!

AndRemember…

21

“Fallseventimesandstandupeight.”

- JapaneseProverb

Q&A

Questions?

THANKYOU