Splunk ITSI Sandbox Guidebook

1

SplunkITSISandboxGuidebookSTARTHERE....................................................................................................................................................................................................................................21-FlyOvertheProduct..............................................................................................................................................................................................................32-PreparefortheJourney:CoreConcepts.......................................................................................................................................................................43-TourtheGlassTables.........................................................................................................................................................................................................134-TroubleshootingTourwithGlassTablesandDeepDives.................................................................................................................................165-DiveintoDeepDive...........................................................................................................................................................................................................257-TourMulti-KPIAlerts.........................................................................................................................................................................................................286-DiveintotheNotableEventsReview.........................................................................................................................................................................347-DiveintotheServiceAnalyzer......................................................................................................................................................................................368-SideTriptoOSHostDetails............................................................................................................................................................................................377b-AndBacktoServiceAnalyzer......................................................................................................................................................................................37

DocumentRevisionHistoryDate Notes2016Apr18 AddedMulti-KPIAlerts,tweakedvariouschapters,editedforconsistency..dmillis2016Apr08 Updatedfor2.2..lsnow2015Dec07 Completed"tour"chapters..jlebaugh,dmillis2015Dec03 Filledoutfirst4chapters..dmillis2015Nov29 Initialversion..dmillis

2

STARTHEREWelcometotheITSISandboxPlaybook!Itisintendedasatravelguidetohelpyouexplorethefeatures,capabilitiesandpossibilitiesofITServiceIntelligence,usingyournewSplunkITSIOnlineSandbox.IfyoudonotalreadyhaveanITSISandbox,gototheITSIHomepage(http://www.splunk.com/itsi)andclickthegreen"FreeOnlineSandbox"button.Itonlytakesafewminutes!Theplaybookcontainsaseriesofchapters,orexercises,tofacilitatetheexplorationofITSIandillustratehowitcouldbeusefulinactual"realworld"environments.ThestudentshouldalreadyhaveabasicunderstandingofcoreSplunk,especiallyhowtocreatesearchesandreports.Thisplaybookshouldnotbeconsidered"realtraining";pleaseseeSplunkEducation(http://www.splunk.com/view/education/SP-CAAAAH9)forin-depthcoursesonITSIandothertopics."Fly-Over"and"Tour"chaptersshowfeaturesandcapabilities,inlessdetailandmoredetail,respectively."Divein"chaptersgointothemostdetailabouthowtosetupandconfigure.Otherchapterscoverhowtocreatenewcomponents,howtouseITSItotroubleshootproblemsquickly,andhowtomock-upvisualizationsforyourownhigh-valueservices.AlthoughtheITSISandboxisnotsetuptoallowoutsidemachine-datatobebroughtin,itdoescontainaneventgeneratortosimulatetheeventswhichmightbeseeninatypicalITenvironment,includingfailurescenarios.Italsocontainsanumberofpre-builtKPIs,services,GlassTablesandothergoodiestomakethejourneymoreinteresting.Generally,thechaptersarelaidoutwiththemorebasicconceptsandexercisesfirst,andmoreadvancedtopicslater.Studentscanskipchaptersandjumparoundastheycareto;eachchapterliststherecommendedpre-requisitechapters.Ultimately,thepurposeofthisplaybookisallowstudentstoworkwithandunderstandthefullcapabilitiesofITServiceIntelligence,andexplorehowITSIcouldhelpsolveactual,useful,high-valuechallengesintheirownITenvironments.

3

1-FlyOvertheProductForthetravellerwhoisinahurry,whowantsthe30,000-footview,thisisthesectionforyou!Itisalsothebestplacetobegin,forthestudentwhoislargelyunfamiliarwithITServiceIntelligence.

Instructions1. AfterloggingintoSplunk,clickon"ProductTour"

2. Clickthroughtheslidestopreviewservices,entities,KPIs,thresholding,DeepDives,Multi-KPIAlerts,NotableEvents

andtheServiceAnalyzer3. Thesetopics,andmore,arecoveredinmoredetailinthefollowingchapters

4

2-PreparefortheJourney:CoreConceptsBeforewebeginthejourney,itishelpfultounderstandafewcoreconceptsofITServiceIntelligence.

ITSICoreConcepts–Services

DNS RequestsResponses

TechnicalServices

CustomerTransac6ons

RequestsResponses

BusinessServices

AuthRequestsResponses

WebRequestsResponses

SupportDesk RequestsResponses

Conceptually,aServiceisa“blackbox”whichwesendrequestsandexpectresponses.Includestechnical(lower-level)andbusiness(higher-level)

8

5

ITSICoreConcepts–Services

PacketNetwork

HypervisorandHosts

RDBMSs

StorageTier

APIServices

WebServices

CustomerTransac4ons

MobileAPI/

Middlew

are

PartnerPortal

DNS

9

ServicescanencompassmulCpleCersoftheITdomain.Servicesmayalsodependuponotherservices

6

ITSICoreConcepts–KPIs&HealthScores

DNS RequestsResponses

KPI:Numberofrequests

KPI:Errorrate

KPI:Averageresponse9me

KPI:ServerCPUload

KPI:ServernetworkI/Ferrors

CustomerTransac:ons

RequestsResponses

KPI:Numberoftransac9ons

KPI:Errorrate

KPI:Averageresponse9me

KPI:CountofIncidentTickets

KPI:Synthe9cTransxHealth

AKeyPerformanceIndicator(KPI)isaSplunksavedsearchthatproducesametriclikeCPU%,AvgResponseTime,ErrorRate,etc.KPIsarecontainedwithinServices.AHealthScoreisascorefrom0-100thathelpsdeterminethehealthofaservice.Itiscalculatedbasedonimportanceandstatus

(e.g.,green,orange,red)ofallKPIs,onceeveryminute

12

7

ITSICoreConcepts–ServiceAnalyzer

ServiceAnalyzerisanauto-generated,filterable,8ledviewofServicesandKPIs.ItisalaunchingpointforexploringServiceandEn8tyHealthindetail,aswellas

crea8ngad-hocDeepDives

13

8

ITServiceIntelligence–CoreConcepts

27

AGlassTableisacustomizablefreeformdrawingdashboardstoviewHealthscoresand

KPIsofchoicewithvisualtoolstocreatecontextwithlivewidgets

GoDeepertoaDeepDiveView

9


28

DeepDive–SwimlaneanalysisdashboardtoshowKPIindicators

over:meforinves:ga:ons

10


29

Mul5KPIAlerts–Visualtooltocreatecorrela0onsearchesbasedonKPIs

11

12

ITSIrepresentsanewwayofdealingwithITServicechallenges:

• Data-drivenapproachusesALLITData-events,metrics,logs,structured,

unstructured,from-the-device,from-the-wire,etc.

• Service-awarenessprovidesactionableinsightsintohigh-visibilityservices

• Customizedcontextualvisualizationscanbetailoredforanypersonorgroup:

highlytechnicaltobusiness-oriented

• Mitigateproblemsbeforetheyimpactcustomers

13

3-TourtheGlassTablesGlassTablesareanewtypeofdashboard,whichallowITSIservices,KPIsandhealthscorestobevisualizedinhighlycustomizableways.GlassTablescanbetailoredtoshowverydetailedtechnicalviews,orhigher-levelbusinessviewswithcustomer/revenue-relevantmetrics.Fromthetechnical"soldiersinthetrenches"toexecutivemanagement,GlassTablescanbecraftedtoshowservices,servicerelationships,transactionflows,healthscores,keybusinessmetricsandothercontentwhicharerelevanttotheusers.Andthey'realotoffuntobuild,too!ThissectionshowsanumberofexampleGlassTables.

Instructions1. NavigatetotheGlassTablelistbyclickingon'GlassTables'inthetopmenubar

2. FromthelistofGlassTables,clickonaTitletoviewthatGlassTable

14

3. SelectButtercupGamesBusinessProcess

ThisGlassTableshowsthehigh-levelbusinessprocessstatusButtercupGames.Itcouldbeusedbyserviceowners,executivemanagementorotherswhoneedtoquicklyunderstandthe"bigpicture".

4. SelectOnLineTransactionService

ThisGlassTableshowsadetailedviewofacustomer-facingservice,includingtransactionflow,componentrelationshipsanddependencies,andcriticalhealthscoresandmetricsofkeyservicepointsalongtheway.Itmakesexcellentuseofapre-existingdrawing,withliveITSI"widgets"placedstrategicallyontop.ThisGlassTablewouldhelpfulforNOC,Tier1&2andsimilarsupportpersonnelwhoneedtounderstandthecomplexrelationshipsofalltheservicecomponentssupportinganimportantbusinessservice.

5. SelectButtercupGamesOnlineStore

ThisGlassTableshowsastreamlinedviewofButtercupGames'customer-facingservice--the"onlinestore"summarizedinthe"ButtercupGamesBusinessProcess"GlassTable.Thisviewprovidesmoredetailoftheunderlyingtechnicalservices,theirdependencies,andtheoveralltransactionflow.ItusesnativeGlassTabledrawingtools,aswellasserviceandKPIwidgets,whichdisplayhealthandmetricvalueslive(updatingovertime).Thesewidgetshaveconfigurabledrill-downcapabilities,includingtheabilitytonavigatetoother,even-more-detailedGlassTables.Forexample,ifyouclickonthewidgetnexttoWebTier,youwillnavigateto...

6. WebTier

ThisGlassTablerepresentsamoredetailedvisualizationoftheKPIs,overallWebTierhealthscore,andthehealthscoreofitsdependentservice,Middleware.SuchGlassTablesallowtechnicalpersonneltoquicklytroubleshootproblemsbybeingabletodrilldowntothedetailedtechnicalmetricswhichmatter.

7. SelectButtercupGamesOnlineStore(again)

Severaldrill-downoptionsareavailablewhenawidgetisclicked.ClickonthewidgetnexttoDatabase;thiswillnavigatetoaDeepDive.

15

GlassTablesallowservices,dependencies,healthscores,KPIsandothercriticalinformationtobevisualizedinacontextualwaythatistrulymeaningfultothetargetedaudience.Thisallowsuserstoquicklysize-upservicedeliveryhealthandwhennecessary,efficientlyisolateproblems.

16

4-TroubleshootingTourwithGlassTablesandDeepDivesThissectiondescribesapossibleproblemscenario,andhowITSIcouldbeusedtoefficientlytroubleshoottofindrootcause.ThiswouldtypicallybedrivenbyaNOCorTier1orTier2supportperson.We'regoingto"setup"thefailurescenarioandfirstseehowGlassTablescanacceleratethetroubleshootingprocess,thencontinueisolatingrootcausewithDeepDives.

Pre-RequisitesYoushouldalreadybefamiliarwith:

• CoreConcepts(Ch.2)• GlassTables(Ch.3)

Abouttheeventgenerator...InordertomaketheITSISandboxmoreinterestingtoplayin,aneventgeneratorisincludedwhichcontinuouslygeneratesasimulatedstreamofrealisticmachineevents,includingwebaccess,database,Linuxmetrics(fromthe*nixTechnologyAdd-on)andothers.Includedinthisstreamofeventsaretwofailurescenarios,showingasequenceoffailuresandresultingservicedegradations,eachscenariorepeatinghourly.Typically,theinitialfailuresforeachscenariooccuratthetopofthehour,andresetbackto"OK"aroundthetopofthenexthour.However,theeventgenerator(eventgen)timingmaynotbeprecise.Thefailurescenariosmayoccuratslightlydifferenttimesfromhourtohour,andmayvaryfromsandboxtosandbox.Thus,withintheSandbox,itisimpossibletopredictexactlyhowthehealthscoresandKPIswillappear,duringanyspecifichour.Thismakesitdifficulttosetupa"clean"failuresimulation.Pleasepardonanyeventgeninconsistencies.WedecidedtoputmostofoureffortintodevelopingITSI--notaneventgenerator.

17

Instructions1. NavigatetotheGlassTablecalled,ButtercupGamesOnlineStore:

a. ClickonGlassTablesintheuppermenubartonavigatetothepage,SavedGlassTablesb. ClickonButtercupGamesOnlineStoretonavigatetothisGlassTable

2. Modifytheviewtimebyclickingonthetimepickerintheupperrightcorner.Inthepop-upwindow,typeinanexplicittimefromthepast,suchasXX:15.0fromtheprevioushour(orthehourbeforethat,etc).BesuretousethecorrectHH:MM:SS.sssformat(example:"10:15:00.0")

18

3. Inafewseconds,thecolorsofthewidgetswillchange,toindicatetheirstatesatthatparticulartimeinthepast.Asnoted

earlier,thetwodifferentfailurescenariostoggleeachhour.Trydifferenteven&oddhoursinthepasttoseethis.4. Forthepurposesofthistroubleshootingexercise,imaginethatyourGlassTablelookslikethefollowing:

19

5. Thescenario:CustomerCarehasinformedusthatcustomersarecallingtocomplainwhentheytrytopurchasethroughtheOnlineStore;theyareseeingslowresponseandoccasionalerrors.Theproblemsseemtobeaffectingbothweb-basedandmobile-basedcustomers.

6. Basedonjustthereportsthatthecustomer-facingweb-basedserviceishavingproblems,mostsupportpersonswouldbegintroubleshooting"fromthetop"--thewebandmobiletiersinthiscase.Ifnoobviousproblemswerefound,theywouldproceeddowntheservicedependencytree--tothemiddlewaretier,etc.

7. ButusingaGlassTablesuchas"ButtercupGamesOnlineStore"providesinstantandcontext-relevantvisibilityintoservicehealthscoresandimportantKPIs,allinoneplace.Intheaboveexample,whichsupportingtierseemstobeindistress?(Database)Bybeingabletovisualizetherelevantservicesandtheirhealthscores,wehavetheabilitytoimmediatelyfocusourtroubleshootingontheareasthataredegraded.Thiscansavehugeamountsoftimeandgreatlyreducethetimerequiredtofindrootcause.

8. OnyourSandboxGlassTable,clickonthewidgetbeneathDatabasetodrilldownintotheDatabasetiertocontinuethetroubleshootingexercise.(SelectLeaveThisPageifprompted)

20

(NowinDBDeepDive)9. Clickonthe>nexttoFocustocollapsetheservicetreenavigatorpanelontherightside;wewillexplorethisfeaturelater.10. ChangethePrimaryTimeRangetoLast2Hoursbyclickingonthetimepickerinthelowerleftcorner:

a. IntheRelativesection,typein"2"andselectHoursAgob. ClickApply

21

(TheDeepDiveshouldnowdisplayssome"mostlygreen"andsome"mostlyred"–yourscreenmaynotlookexactlylikethebelow,butyoushouldseeapointwherethingsgofromgreentoredandredtogreen)

11. WearelookingattheAggregatedHealthScore(topswimlane)andKPIsfortheDBService,acrossatimerangewhich

showstheservicemovingfrom"healthy"to"nothealthy".12. Slowlymouseovertheswimlanestocomparevaluesatvariouspointsintime.13. Clickthecheckboxintheupperlefttoselectallswimlanes,andusethe“BulkActions”menuto“ShowStateThresholds”or

"HideThresholds",togglingtocomparetheswimlaneswithandwithoutthethresholdcolors/statesoverlaid.14. NotethattheServiceHealthScoreinthetopswimlaneisanaggregationoftheservice'sKPIsanddependentservices,

rangingfrom100-0.Whendidthehealthscorebegintodeteriorate,andwhichKPI(s)mayhavebeenpartoftherootcause?

22

15. Clickonthename-boxforStorageFreeSpace:%System,thendragitupwardstorepositionthisswimlane.16. Afewoftheswimlanesarecontinuouslygreen,indicatingthattheyarenotparticularlyhelpfulinourtroubleshooting

exercise("CPUUtilization","MemoryFree",etc).Clickonthecheckboxintheupperleftcornertounselectallswimlanes,thenselectthecheckboxforCPUUtilization:%UserandMemoryFree.SelectBulkActions->Deleteto(temporarily)removethisswimlanefromourDeepDive.

17. ClickonthedarkerbluetilewithintheDBServiceErrorsswimlanetoreveal"rawerrors"fromtheunderlyingSplunk

search.ClickonHideEventstodismiss.

23

18. MouseovertheStorageFreeSpace:%Systemswimlane,intheplacewhereitgoesfromgreentored.Notethehigh&lowmetricvaluesshownfortheswimlane,andthatthismetrichasgoneto0%,indicatingthatafilesystemisfull.

19. ClickanywherewithintheStorageFreeSpace:%Systemswimlanetorevealanoptionspopup.SelectAddOverlayasLane.

(Threenewswimlanesareaddedatthebottom,representingtheseparateKPIvaluesfortheindividualentities(hosts)whichcomprisethisKPI)20. Whichhost/serverissufferingfromafilesystem-fullcondition?(mysql-02)

24

OverallservicehealthcanbeeffectivelyandefficientlyvisualizedinGlassTables,allowingsupportpersonneltoquicklyfindlikelyhotspots.TheycandrilldowntomoredetailedlayoutsandultimatelycompareandcorrelateKPI&ServicetrendsinparallelswimlaneswithinDeepDive.Fasterrootcauseanalysis(RCA)leadstosubstantialreductionsinMeanTimeToRepair(MTTR).

25

5-DiveintoDeepDiveDeepDivesallowKPImetricsandhealthscorestobecomparedinside-by-sideswimlanes,whichallowstrendsandcorrelationstobemoreeasilyandquicklydiscovered.ThischapterexploresDeepDivesandhowtheycanbeused.


• CoreConcepts(Ch.2)• Troubleshooting(Ch.4)alsogoesintoDeepDives

Instructions1. NavigatetotheDeepDivecalled,DBDeepDive:

a. ClickonDeepDivesintheuppermenubartonavigatetothepage,SavedDeepDivesb. ClickonDBDeepDivetonavigatetothisDeepDive

2. Clickonthe>nexttoFocustocollapsetheservicetreenavigatorpanelontherightside;wewillexplorethisfeaturelater.3. SelectanarbitrarytimerangebyclickingonthePrimaryTimeRangemenuoptionatthebottomright;itfunctionslikea

standardSplunksearchbartimepicker4. Zoomintoatightertimerangeinthecurrentviewbyclick-holdinganywhereintheswimlanes,thendragginghorizontally

toselecttherange.5. Togglethethresholdhealthscorecolorsbyclickingonthecheckboxintheupperleftcornertoselectallswimlanes,then

BulkActions->ShowStateThresholds/ShowLevelThresholds/HideThresholds.6. Clickonthe>nexttoFocustoopentheservicetreenavigatorpanelontherightside.

a. Clickonaservicenodetonavigateupanddownthedependencytreeofservicesb. Afterclickingonaservicenode,notethatthoseservice'sKPIsarelistedbelow.c. Clickonthe+onalistedKPItoaddittothecurrentswimlanesd. Clickonthe>nexttoFocustocollapsetheservicetreenavigatorpanelontherightside

7. Mouse-overthename-boxforanyswimlanetorevealthe"optionswheel",thenselectittoviewavailableoptions:

26

8. Thestudentisencouragedtoexploretheseoptions,whicharecoveredinmoredetailat

http://docs.splunk.com/Documentation/ITSI/latest/User/DeepDives9. Click-holdonthename-boxforanyswimlane,andthendragitverticallytorepositionthisswimlane.10. ClickonthedarkerbluetilewithintheDBErrors(orany"event"-style)swimlanetoreveal"rawerrors"fromthe

underlyingSplunksearch.ClickonHideEventstodismiss.11. TosaveaDeepDiveaftermodifyingthelayoutand/orvisualizationoptions,clickontheEditmenuoptionintheupper

rightcorner,thenselectSave12. Tocomparethecurrenttimerangeagainstadifferenttimerange,clickonCompareto...inthelowerleftcorner,then

selectacomparisontimerange.ThiscauseseachKPItodisplaytwinswimlanes:primarytimerangeabovecomparisontimerange.Notethatwhenmousingovertheswimlanes,thetimedisplayatthetopnowshowsbothtimes.

13. Todismissthe"twin"lanesdisplay,deselectthecheckboxnexttoCompareto...inthelowerleftcorner

27

DeepDiveallowsanyKPIsandServicestobecomparedandcorrelatedinaside-by-sidefashion,acrossmultipletimeranges,usingavarietyofvisualizations.Itisintendedtogreatlyenhanceandstreamlinethetroubleshootingprocessforfindingrootcause,significantlydecreasingMeanTimeToRepair(MTTR).

28

7-TourMulti-KPIAlerts

Multi-KPIAlertsareCorrelationSearcheswhichcancombineanyKPIstocreatemeaningful,actionablealerts,usingmultiplecorrelationfactorssuchKPIthresholdindications,lengthoftimeinthisstate,time-of-day,andothers.Multi-KPIalertscanfindnotjust"failures",butearly"canaryinthecoalmine"indicationsthattheserviceisbecomingunstable;itispossibletofindproblemsBEFOREtheyimpactcustomer-facingservices.WhenaMulti-KPIAlertfires,itcreatesaNotableEvent;itcouldalsoexecuteascriptand/orsendemail.


• CoreConcepts(Ch.2)• TroubleshootwithGlassTablesandDeepDives(Ch.4)

Instructions1. NavigatetotheDeepDivecalled,DBDeepDive:

a. ClickonDeepDivesintheuppermenubartonavigatetothepage,SavedDeepDivesb. ClickonDBDeepDivetonavigatetothisDeepDive

2. Clickonthe>nexttoFocustocollapsetheservicetreenavigatorpanelontherightside.3. ChangethePrimaryTimeRangetoLast2Hoursbyclickingonthetimepickerinthelowerleftcorner:

a. IntheRelativesection,typein"2"andselectHoursAgob. ClickonApply

29

(TheDeepDiveshouldnowdisplayssome"mostlygreen"andsome"mostlyred"–yourscreenmaynotlookexactlylikethebelow,butyoushouldseeapointwherethingsgofromgreentoredandredtogreen)

30

4. WearelookingattheAggregatedHealthScore(topswimlane)andKPIsfortheDBService,acrossarangeoftimewhich

showstheservicemovingfrom"healthy"to"nothealthy".5. Click/dragacrossanarrowerrangeoftimewhentheservicetransitionsfromgreentoyellow/orange.6. Clickonthecheckboxintheupperlefttounselectallswimlanes,thenselectthecheckboxesnexttotheKPIswimlanes

whichwereinvolvedinthisoutage(turnedred)duringthisperiod,suchasStorageFreeSpace,DBServiceQueries&DBServiceResponseTime.

7. Intheupperleft,selectBulkActions->CreateMultiKPIAlert

31

(ThiswillopentheMultiKPIAlertconfigurationworkflowpage)

32

ITSIprovidesasophisticatedarrayofoptionsforsettingupMulti-KPIAlerts,alsoknownasCorrelationSearches.ThegoalistoallowthecreationofusefulalertsbasedoncorrelationsofseveralKPIs--fewer"noise"alerts,moreactionablealerts.Herearesomeofthefeaturesandcapabilities:

• ControltherangeoftimetocorrelatetheKPIsacross(time-pickerintheupperrightcorner)• AddKPIsfromanyservice• CreateaKPIbasedontheaggregatehealthscoreoftheKPIs,oronStatusovertime(upperrightcorner)• Re-weighttheKPIsusingtheImportancesliders(lowerrightcorner)

33

• Controlalertactions,suppression,andotherdetails(laterinthecreationworkflow,afterhitting'Save'inthelowerrightcorner)

OneofthemostimportantthingswhichhappenswhenaMulti-KPIAlertfires,isthecreationofaNotableEvent.NotableEventsareexploredinalaterchapter.ExistingsampleMulti-KPIalertscanbeexaminedbyclickingonConfigure->CorrelationSearches,thenselectingacorrelationsearchfromthelist.Moredetailsareavailablehere:http://docs.splunk.com/Documentation/ITSI/latest/User/CreateMulti-KPIAlertsMulti-KPIAlertscancombineanyKPIstocreateuseful,actionablealerts(lessalert"noise")."Canaryinthecoalmine"problemscanbedetectedearly,potentiallybeforetheyaffectcustomers,revenueorSLAs.

34

6-DiveintotheNotableEventsReviewWhenaMulti-KPIAlertfires,itcreatesaNotableEvent.TheNotableEventsReviewisSplunk'snext-generationeventmanagementconsole.NotableEventsReviewprovidesaquickwayview,siftandorganizeevents,allowingustotriage,manageandstreamlineworkflowmoreeffectively.IthastheabilityfilterNotableEventsandeventsfromothereventmanagementsources,basedonvariouscriteria,suchasSeverity,Status,Serviceandothers.Italsoallowseventstobemodified,tochangeOwner,Severity,Status,and/oraddcomments.Eventscanalsohaveworkflowactionsassociatedwiththem,toallowanoperatortheabilitytoquicklyhittroubleshootingoptions,executemitigationscripts,oropena"real"IncidentManagementtrouble-ticket.


• CoreConcepts(Ch.2)• TourMulti-KPIAlerts(Ch.7)

Instructions1. NavigatetotheNotableEventsReviewbyclickingonNotableEventsReviewintheuppermenubar2. ClickonShowTimelinetorevealthetimeline3. Seedetailsforanevent:ClickonanyeventtoopentheDetailspanelontheright.

DetailsincludewhichKPIscontributed,andwhichservicesmightbeaffected,aswellastheabilitytoexaminetheseinmoredetailinaDeepDive.Severity,StatusandAssignmentcanalsobechangeddirectly.

4. ModifySeverityforanevent:ClickontheSeveritydropdownatupperleftoftheDetailspanel,chooseadifferentSeverity5. Chooseaworkflowaction:Clickon</>iconinupperrightcornerofDetailspaneltorevealtheworkflowoptions

CustomworkflowactionscanbecreatedforeachtypeofNotableEvent,tostreamlineworkflowactions.Thesecanbeadditionaltroubleshootingormitigationscripts,orsomethingasbasicasopeninga'real'incidentticket.

6. DismissDetails:ClickontheXintheupperrightcornerofDetailspaneltodismiss7. FiltertheNotableEventsbySeverity:Clickonthe"gearwheel"intheupperrightcorner,thenchooseEditFilterSettings.

ClickAddFilter,andthenSeverity.ClickintheSeverityboxtoseeandchoosefromalistoftheavailableseveritylevels.

35

8. FilterbyStatus,Owner,Service,TimeRange,Name("Title")orfreeformsearchcriteriabyaddingotherfilterstoyourfiltersettings.

9. ClickDoneto(re)applysearchfiltercriteria10. Changeviewoptions:Clickonthe"gearwheel"intheupperrightcorner,thenchooseEditViewSettings.SelectViewing

Option->ProminentandDeduplication->On,thenDoneAnEventCountcolumnhasnowbeenaddedfordeduplicatedevents,andSeveritycolorisnowmore'prominent'

11. Add,removeorre-ordercolumns:Clickonthe"gearwheel"intheupperrightcorner,thenchooseEditViewSettings.InColumnsShown,clickXtoremoveacolumn,click+AddColumntoaddacolumn,orclick/dragacolumntore-orderhowitisviewed.

12. Tosorttheeventrows:ClickontheVchevronnexttoSortBy:(leftside,aboverows),thenselectacolumntosortby.Togglethesortorder(ascending/descending)byclickingontheverticalarrownexttoSortBy:

Moredetailsareavailablehere:http://docs.splunk.com/Documentation/ITSI/latest/User/NotableEventsReviewTheNotableEventsReviewallowsanoperatorto:• Quicklyandeffectivelyfind,deduplicateandmanagejusttheeventstheywant• Tieworkflowactionstoevents,tostreamlineoperations• ManageITSINotableEventsandeventsfromothersources

36

7-DiveintotheServiceAnalyzerTheServiceAnalyzerisa"BigPicture"viewofallservices,andthe"mostinteresting"KPIs(i.e.,KPIswithdegradedhealthscores).Itis"nofrills",designedforNOCs,Tier1or2support,andotherswhoneedahighlevelviewofallservices/KPIs,orasubset.ItalsoprovidesalaunchingpointforexploringServices,KPIsandEntitiesinmoredetail.


• CoreConcepts(Ch.2)

Instructions1. NavigatetotheServiceAnalyzerbyclickingonServiceAnalyzerintheuppermenubar,thenchoosingDefaultService

Analyzer.2. ClickonMiddlewareServicetonavigatetoitsservicehealthpage.Hereyoucanseetheservicetreeontheleft,theKPIs

inthecenter,andtheentitiesassociatedwithaselectedKPIontheright.3. ClickonDBServiceintheleftservicetreepaneltonavigatetothisservice4. FromServiceHealth,youcanalsonavigatetoadeepdivecontainingtheKPIsforthatserviceusingthelinkatthetopof

theKPItableinthecenterofthepage.a. Noticethedeepdivehasbeenbuiltforyouonthefly,containingalltheKPI’sassociatedwiththatservice

5. ClickonStorageFreeSpace:%SystemandnoticethatyounowhaveatableontherightthatshowstheentitiesassociatedwiththisKPI.

6. Clickonmysql-02intheentitylisttonavigatetoitsEntityHealthpage.a. Thisisanentity-centricview,showinginformationaboutaspecificentity,includingwhichservicesandKPIsit

supports.b. Clickingonaservicenamewillnavigatetothatservicehealthpage

37

8-SideTriptoOSHostDetails7. IfyouareusingoneormoreITSImodules,relevantmoduledashboardsforthisentitywillshowupintheleft-sideModules

panel.Inthiscase,"OSHostDetails"islisted.Moredetailsaboutmodulesareavailablehere:http://docs.splunk.com/Documentation/ITSI/latest/IModules/AboutITSIModules

8. ClickonOSHostDetailstonavigatetothispage.a. TheOSHostDetailssectionoffersseveraldashboardswithdetailedstatus,performanceandeventreports.b. OSHostDetailscanalsobeaccessedinDeepDive.c. MoredetailsabouttheOperatingSystem(OS)Moduleareavailablehere:

http://docs.splunk.com/Documentation/ITSI/latest/IModules/AbouttheOperatingSystemModule

7b-AndBacktoServiceAnalyzer9. NavigatebacktoServiceAnalyzer10. ClickintheSelectservice(s)tomonitorboxtoselect&showonlycertainservices11. Clickonthe"OptionWheel"nexttoTop...Servicestocontrolhowmanyservicesareshown12. Clickonthe"OptionWheel"nexttoTop...KPIstocontrolhowmanyKPIsareshown,andtoselectwhichKPIsareshown13. Tocreateanad-hocDeepDive:

a. MouseoveroneormoreServiceorKPItiles,thenselectthecheckboxintheupperrightcornerofthetileb. ClickDrilldowntoDeepDive

ServiceAnalyzerprovidesa"BigPicture"viewofallservicesandthe"mostinteresting"(notgreen)KPIs.ItisalsoalaunchingpointforexploringServices,KPIsandEntitiesinmoredetail,aswellasforcreatingad-hocDeepDiveswithselectedKPIs.