Top Banner
Copyright © 2016 Splunk Inc. Kevin Dalian [email protected] Glen Upreti [email protected] Splunk Gone Wild! – Innovating A Large Splunk Solution At The Speed Of Management
23

SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Aug 23, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Copyright©2016Splunk Inc.

[email protected]

[email protected]

Splunk GoneWild!– InnovatingALargeSplunk SolutionAtTheSpeedOfManagement

Page 2: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Disclaimer

2

Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose

containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor

functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.

Page 3: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Introductions

KevinDalianTeamLeadServerHostingTools

FordMotorCompanyNothingInterestingAbout Kevin,he’saboringwork-aholic.

GlenUpretiDirectorEnterpriseandCloudTechnologies

Sierra-CedarTerribleatJenga

Page 4: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Agenda

4

WhereWeCameFromWhereWePlannedtoGoWhereWeEndedUpInstallationOnBoardingDataWhatWe’reuptoNowQ&A

Page 5: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

WhereWe Started2SplunkEnvironments– NetworkandServerOperationsServerOps– 4StandaloneSearchHead/Indexers– 3DeploymentServers– 20Gblicense– +11,500UniversalForwarders

Page 6: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

WhereWePlannedtoGo

6

MajornewFordInitiative– FordPass/Connected-X– PCF– PivotalCloudFoundry– MicrosoftAzureCloud– Mixtureofinternalandexternalapplicationsanddata– 100Gb/day

Page 7: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

WhereWePlannedtoGo(cont’d)

PCF

Azure

PublicInternet

PeerIndexer1

SearchHead1

DeploymentServer/DMC/LicenseMaster

SearchHead1 SearchHead3

PeerIndexer2 PeerIndexer MasterIndexer

SearchHeadMaster

ServerInfrastructure

SplunkForwarders

Syslogfirehose SyslogRelay

SearchHead/Indexer

JMXRESTSQL

SyslogReceiver(w/SplunkUF)

SyslogRelay

Syslog

Syslog

DMZ Intranet

UniversalForwarder

Page 8: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

WhereWeEndedUp

8

MobileSearchHead2

Syslog

PCF-Prod

AzureNA2

SyslogReceiver(w/SplunkUF)

DMZ

Intranet

PublicInternet

PeerIndexer1

SearchHead1

DeploymentServer/BatchProcessor/

Archive/DMC/LicenseMaster

SearchHead2 SearchHead3

PeerIndexer2 PeerIndexer MasterIndexer

SearchHeadMaster

ServerInfrastructure

SplunkForwarders

FMCHeavyForwarder

Syslogfirehose

JMXRESTSQL

UniversalForwarder

SSL

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

PCF-Prod

AzureNA1SSL

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

TCP

SyslogRelay

SyslogRelay

SyslogRelay

Syslog

Syslog

UniversalForwarder

TCP

SSL

SSL

SearchHead/Indexer

SearchHead/IndexerSyslogfirehose

JMXRESTSQL

Syslogfirehose

JMXRESTSQL

PCFFMC-Prod

JMXRESTSQL

PCFFMC-

PreprodECCHeavyForwarder

Syslogfirehose

JMXRESTSQL

PCFECC-

Preprod

SyslogfirehoseJMXRESTSQL

PCF-Dev

PCF-Dev

MobileSearchHead1

PCF-Prod

21VCN1

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

SyslogRelay

SearchHead/IndexerSyslogfirehose

JMXRESTSQLPCF-Dev

SSL

SSL

syslog

Http

Android

FMCSearchHead/Indexer

ECCSearchHead/Indexer

PCF-Prod

21VCN2

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

SyslogRelay

SearchHead/IndexerSyslogfirehose

JMXRESTSQLPCF-Dev

syslog

SSL

SSL

Https

Https

Https

Https

Apple

Syslogfirehose

PCFECC-Prod

PCF-Prod

AzureEU1Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

PCF-Prod

AzureEU2

Syslogfirehose

HeavyForwarders

JMXRESTSQL

JMXRESTSQL

SyslogRelay

SyslogRelay

SearchHead/Indexer

SearchHead/IndexerSyslogfirehose

JMXRESTSQL

Syslogfirehose

JMXRESTSQL

PCF-Dev

PCF-Dev

SSL

SSL

SSL

SSL

syslog

syslog

Page 9: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

InstallingGlenandKevinmeetandplanforinstallationtasksHaveaPOCenvironmentinAzureAzuretoon-premise,HOW???Startedwithtemporarystandaloneinstance

Page 10: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Installing(Surprises)Hardwareshowsup&everythingfallsapart– Hardwarearrivedpiecemeal– NotenoughCPUs– AzureVMs,wewerethefirsttemplateinstall– Serversin‘Public’DMZweren’tpubliclyaccessible

EvenwithissuesSHClusterandIDXClusterallinstalledwithindays!

Page 11: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

OnBoarding DataOnboardeddatafrom– Pivotal CloudFoundry– MicrosoftAzurePAASviaDBConnect– Thirdpartyandcustomdevelopedinputs

Page 12: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Onboarding DataWhen onboardingalwaysset– TIME_PREFIX– TIME_FORMAT– MAX_TIMESTAMP_LOOKAHEAD– SHOULD_LINEMERGE– LINE_BREAKER– TRUNCATE

Page 13: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Onboarding Data(Surprises)‘Ohbytheway…’– Newinputs– Newregions– Newenvironments(pre-production)– Newteams– NewSplunkLicense– SensitiveData- Needforobfuscation

Page 14: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

GotData,NowWhat?PrototypedDashboardwrapupquick.WhatDoesThisMean?– Engagedevelopers andusercommunities

KeepCreating– Alwaysbemovingforward

Alerts– Alertingisaniterativeprocess– Bepreparedforalotofnoiseatfirst– Refine,refine,refine

Page 15: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

GotData,NowWhat?(Surprises)Surprises:– MassiveDashboards– NewUsersandRoles– DataSecurity– RetentionTimes

Ohyeah,andmobile…

Page 16: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

MobileMadnessSplunkAdd-onforMobileAccess– CrazyEasy!InitialPOCinAzureworkedlikeachampPlanned,preparedandmovedtoDMZNotificationsdon’twork– Newmanagementsurprise…thekindyoudon’twantBacktothedrawingboard

Page 17: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

MobileMadness(TempSolution)

PCF

Azure

PublicInternet

PeerIndexer1

SearchHead1

DeploymentServer/DMC/LicenseMaster

SearchHead1 SearchHead3

PeerIndexer2 PeerIndexer MasterIndexer

SearchHeadMaster

Syslogfirehose SyslogRelay

SearchHead/Indexer

JMXRESTSQL

DMZ Intranet

UniversalForwarder

Http

AndroidApple

Page 18: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

MobileMadness(Eventually)

Page 19: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

WhereAreWeNow?PlanningfortheFuture/ScalingRefininganddocumentingMigratingdata/appsfromoriginalenvironmentExpandingthecustomerbaseStillrefiningdashboardsRe-sourcetypingPreparingformoremanagementshenanigans

Page 20: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

AdviceMovingForward1. Insistnon-productionenvironment2. Workwiththecustomertofurtherunderstandingofdata3. Define/DocumentallCustomerrequirementsandgetsignoff4. Avoidthedatagraveyard5. Splunkisveryflexible,keepanopenmindandstaycalm!

Page 21: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

AndRemember…

21

“Fallseventimesandstandupeight.”

- JapaneseProverb

Page 22: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

Q&A

Questions?

Page 23: SplunkGone Wild! –Innovating A Large SplunkSolution At The …€¦ · Server Ops – 4 Standalone Search Head / Indexers – 3 Deployment Servers – 20 Gb license – +11,500

THANKYOU


Related Documents
Más de 11,500 Hermanos Maristas fallecidos (1825-2009) Más de 11,500 Hermanos Maristas fallecidos (1825-2009) Tras las huellas de Marcelino Champagnat.

Más de 11,500 Hermanos Maristas fallecidos (1825-2009) Más...

Category: Documents
APPROXIMATELY 11,500 RSF COMBINED Mount Vernon Triangle

APPROXIMATELY 11,500 RSF COMBINED Mount Vernon Triangle

Category: Documents
GLENELLERSLIE HEREFORD STUD - Login€¦ · 2012 GlenellerslieKenny $11,500 Janine Cooper Wulgulmerang, Glenellerslie Koala 2 $11,500 Greg O’Brien, Maringo Hereford Stud Merrijig.

GLENELLERSLIE HEREFORD STUD - Login€¦ · 2012...

Category: Documents
24 - Innovating Food, Innovating the Law - Sven Bostyn

24 - Innovating Food, Innovating the Law - Sven Bostyn

Category: Education
Social Networking for Indexers – Panel Discussion – with ... · Overview This panel session will discuss the electronic systems through which indexers communicate ‘en masse’.

Social Networking for Indexers – Panel Discussion – with...

Category: Documents
04 - Innovating Food, Innovating the Law - Iacopo Berti

04 - Innovating Food, Innovating the Law - Iacopo Berti

Category: Education
MODELS: PCL352 and PCL353 INDEXERS and DPF723352 and … · 2020. 11. 24. · DPF73352 PCL353 MODELS: PCL352 and PCL353 INDEXERS and DPF723352 and DPF73353 DRIVER PACK S ANAHEIM AUTOMATION

MODELS: PCL352 and PCL353 INDEXERS and DPF723352 and … ·...

Category: Documents
19 - Innovating Food, Innovating the Law - Alberto Alemanno

19 - Innovating Food, Innovating the Law - Alberto Alemanno

Category: Education
1 Jenkins Ñ Innovating Game Development CS134: Innovating ...

1 Jenkins Ñ Innovating Game Development CS134: Innovating.....

Category: Documents
Diving in OOPS Indexers Part8

Diving in OOPS Indexers Part8

Category: Documents
Some new results on strong integer additive set-indexers of graphs

Some new results on strong integer additive set-indexers of....

Category: Documents
23 - Innovating Food, Innovating the Law - VINOD KUMAR GUPTA

23 - Innovating Food, Innovating the Law - VINOD KUMAR...

Category: Education