Top Banner
Copyright © 2014 Splunk Inc. Machine Data 101: Turning Data into Insight Audience Version
96

Machine Data 101 workshop audienceversion

Apr 13, 2017

Download

Technology

Splunk
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Machine Data 101 workshop audienceversion

Copyright©2014SplunkInc.

MachineData101:TurningDataintoInsight

AudienceVersion

Page 2: Machine Data 101 workshop audienceversion

Agenda

§ Non-TraditionalDataSources

§ DataEnrichment

§ LevelUponSearchandReportingCommands

§ DataModelsandPivot

§ AdvancedVisualizationsandtheWebFramework

2

Page 3: Machine Data 101 workshop audienceversion

Non-TraditionalDataSources

Page 4: Machine Data 101 workshop audienceversion

Non-TraditionalDataSources

§ NetworkInputs

§ HTTPEventCollector

§ LogEventAlertAction

§ SplunkAppforStream

§ ScriptedInputs

§ DatabaseInputs

§ SplunkODBCDriver

§ ModularInputs

§ zLinux Forwarder

§ MINT

§ Non-SplunkDatastores

4

Page 5: Machine Data 101 workshop audienceversion

TraditionalDataSources§ Captureseventsfromlogfilesinrealtime

§ Runsscriptstogathersystemmetrics,connecttoAPIsanddatabases

§ Listenstosyslog andgathersWindowsevents

§ Universallyindexesanydataformatsoitdoesn’tneedadapters

5

Windows• Registry• Eventlogs• Filesystem• sysinternals

Linux/Unix• Configurations• Syslog• Filesystem• Ps,iostat,top

Virtualization• Hypervisor• GuestOS• GuestApps

Applications• Weblogs• Log4J,JMS,JMX• .NETevents• Codeandscripts

Databases• Configurations• Audit/querylogs• Tables• Schemas

Network• Configurations• syslog• SNMP• netflow

Page 6: Machine Data 101 workshop audienceversion

NetworkInputs

§ CollectdataoveranyUDPorTCPport§ Somedevicesonlysenddataoveranetworkport

§ BestPractice:usesyslog-ng orrsyslog§ Offerspersistence§ Categorizesdatabyhost

6

Page 7: Machine Data 101 workshop audienceversion

HTTPEventCollector(HEC)

§ CollectdataoverHTTPorHTTPSdirectlytoSplunk§ ApplicationDeveloperfocus– fewlinesofcodeinapp

tosenddata§ HECFeaturesInclude:

§ Token-based,notcredentialbased§ IndexerAcknowledgements– guaranteesdataindexing§ RawandJSONformattedeventpayloads§ SSL,CORS(CrossOrigion access),andNetworkRestrictions

7

Page 8: Machine Data 101 workshop audienceversion

LogEventAlertAction

§ UseSplunkalertingtoindexacustomlogevent§ Splunksearchableindexofcustomalertevents

§ ConfigurableFeaturesInclude:§ Host§ Source§ Sourcetype§ Index§ Eventtext– constructtheexactsyntaxofthelogevent,

includinganytext,tokens,orotherinformation

8

Page 9: Machine Data 101 workshop audienceversion

TheSplunkAppforStream

WireDataEnhancesthePlatformforOperationalIntelligence

Efficient,Cloud-readyWireDataCollection

SimpleDeploymentSupportsFastTimetoValue

9

Page 10: Machine Data 101 workshop audienceversion

Stream=BetterInsightsfor*

SolutionArea ContextualData WireData Enriched View

ApplicationManagement

applicationlogs,monitoringdata,metrics,events

protocolconversationsondatabaseperformance,DNSlookups,clientdata,businesstransactionpaths…

Measureapplicationresponsetimes,deeperinsightsforroot-causediagnostics,tracetxpaths,establishbaselines…

IT Operations applicationlogs,monitoringdata,metrics,events

payloaddataincludingprocesstimes,errors,transactiontraces,ICAlatency,SQLstatements,DNSrecords…

Analyzetrafficvolume,speedandpacketstoidentifyinfrastructureperformanceissues,capacityconstraints,changes;establishbaselines…

10

Page 11: Machine Data 101 workshop audienceversion

Stream=BetterInsightsfor*SolutionArea ContextualData WireData Enriched View

Security app+infralogs,monitoringdata,events

protocolidentification,protocolheaders,contentandpayloadinformation,flowrecords

Buildanalyticsandcontextforincidentresponse,threatdetection,monitoringandcompliance

DigitalIntelligence

websiteactivity,clickstreamdata,metrics

browser-levelcustomerinteractions

CustomerExperience – analyzewebsiteandapplicationbottleneckstoimprovecustomerexperienceandonlinerevenues

CustomerSupport(online,callcenter)– fasterrootcauseanalysisandresolutionofcustomerissueswithwebsiteorapps

11

Page 12: Machine Data 101 workshop audienceversion

ScriptedInputs

12

§ SenddatatoSplunkviaacustomscript§ Splunkindexesanythingwrittentostdout§ Splunkhandlesscheduling§ Supportsshell,Pythonscripts,WINbatch,PowerShell§ Anyotherutilitythatcanformatandstreamdata

StreamingMode§ Splunkexecutesscriptandindexesstdout

§ Checksforanyrunninginstances

WritetoFileMode§ Splunklaunchesscriptwhichproducesoutputfile,noneedforexternalscheduler

§ Splunkmonitorsoutputfile

Page 13: Machine Data 101 workshop audienceversion

UseCasesforScriptedInputs

13

§ Alternativetofile-baseornetwork-basedinputs§ Streamdatafromcommand-linetools,suchasvmstat andiostat§ Pollawebservice,APIordatabaseandprocesstheresults§ Reformatcomplexorbinarydataforeasierparsingintoeventsandfields§ Maintaindatasourceswithsloworresource-intensivestartup

procedures§ Providespecialorcomplexhandlingfortransientorunstableinputs§ Scriptsthatmanagepasswordsandcredentials§ Wrapperscriptsforcommandlineinputsthatcontainspecialcharacters

Page 14: Machine Data 101 workshop audienceversion

DatabaseInputs

§ Createvaluewithstructureddata§ Enrichsearchresultswithadditionalbusinesscontext

§ Easilyimportdatafordeeperanalysis§ IntegratemultipleDBsconcurrently§ Simpleset-up,non-invasiveandsecure

DBConnectprovidesreliable,scalable,real-timeintegrationbetweenSplunkandtraditionalrelationaldatabases

14

Page 15: Machine Data 101 workshop audienceversion

ConfigureDatabaseInputs

15

§ DBConnectApp§ Real-time,scalableintegrationwithrelationalDBs§ Browseandnavigateschemasandtablesbeforedataimport§ Reliablescheduledimport§ SeamlessinstallationandUIconfiguration§ Supportsconnectionpoolingandcaching

§ “Tail”tablesorimportentiretables§ Detectandimportnew/updatedrowsusingtimestampsoruniqueIDs

§ SupportsmanyRDBMSflavors§ AWSRDSAurora,AWSRedShift,IBMDB2forLinux,Informix,MemSQL,MSSQL,MySQL,

Oracle,PostgreSQL,SAPSQLAnywhere(akaSybaseSA),SybaseASEandIQ,Teradata

Page 16: Machine Data 101 workshop audienceversion

SplunkODBCDriver

16

§ Interactwith,manipulateandvisualizemachinedatainSplunkEnterpriseusingbusinesssoftwaretools

§ LeverageanalyticsfromSplunkalongsideMicrosoftExcel,TableauDesktoporMicrostrategy AnalyticsDesktop

§ Industry-standardconnectivitytoSplunkEnterprise§ Empowersbusinessuserswithdirectandsecureaccesstomachinedata

§ Combinemachinedatawithstructureddataforbetteroperationalcontext

Page 17: Machine Data 101 workshop audienceversion

ODBC:HowitWorks

17

Page 18: Machine Data 101 workshop audienceversion

ModularInputs

18

§ Createyourowncustominputs§ Scriptedinputwithstructureandintelligence§ FirstclasscitizenintheSplunkmanagementinterface§ AppearsunderSettings>DataInputs

§ Benefitsoversimplescriptedinput§ Instancecontrol:launchasingleormultipleinstances§ Inputvalidation§ Supportmultipleplatforms§ StreamdataastextorXML§ SecureaccesstomodinputscriptsviaRESTendpoints

Page 19: Machine Data 101 workshop audienceversion

ExampleModularInputs

19

Twitter§ StreamJSONdatafromaTwittersourcetoSplunkusingTweepy

AmazonS3OnlineStorage§ IndexdatafromtheAmazonS3onlinestoragewebservice

JavaMessagingService(JMS)§ PollmessagequeuesandtopicsthroughJMSMessagingAPI§ Talkstomultipleproviders:MQSeries (Websphere MQ),ActiveMQ,TibcoEMS,HornetQ,RabbitMQ,NativeJMS,WebLogic JMS,SonicMQ

SplunkWindowsInputs§ RetrieveWINeventlogs,registrykeys,perfmon counters

Page 20: Machine Data 101 workshop audienceversion

MoreModularInputs

20

Page 21: Machine Data 101 workshop audienceversion

zLinux Forwarder

21

§ EasilycollectandindexdataonIBMmainframes

§ Collectapplicationandplatformdata

§ DownloadasnewForwarderdistributionfors390xLinux

Page 22: Machine Data 101 workshop audienceversion

ExtendOperationalIntelligencetoMobileApps

22

DeliverBetterPerforming,MoreReliableApps

DeliverReal-TimeOmni-Channel

Analytics

End-to-EndPerformanceandCapacityInsights

Page 23: Machine Data 101 workshop audienceversion

MonitorAppUsageandPerformance

• Improveuserretentionbyquicklyidentifyingcrashesandperformanceissues

• Establishwhetherissuesarecausedbyanapporthenetwork(s)

• Correlateapp,OSanddevicetypetodiagnosecrashandnetworkperformanceissues

23

Page 24: Machine Data 101 workshop audienceversion

IntegratedAnalyticsPlatformforDiverseDataStoresFull-featured,IntegratedProduct

FastInsightsforEveryone

WorkswithWhatYouHaveToday

Explore Visualize Dashboards

ShareAnalyze

HadoopClusters NoSQLandOtherDataStores

Hadoop ClientLibraries StreamingResourceLibraries

Bi-directionalIntegrationwithHadoop

Page 25: Machine Data 101 workshop audienceversion

ConnecttoNoSQLandOtherDataStores

• Buildcustomstreamingresourcelibraries

• SearchandanalyzedatafromotherdatastoresinHunk

• InpartnershipwithleadingNoSQLvendors

• UseinconjunctionwithDBConnectforrelationaldatabaselookups

Page 26: Machine Data 101 workshop audienceversion

VirtualIndexes

§ EnablesseamlessuseofalmosttheentireSplunkstackondata

§ AutomaticallyhandlesMapReduce

§ Technologyispatentpending

Page 27: Machine Data 101 workshop audienceversion

DataEnrichment

Page 28: Machine Data 101 workshop audienceversion

Agenda

§ Tags – categorizeandaddmeaningtodata

§ FieldAliases – simplifysearchandcorrelation

§ CalculatedFields – shortcutcomplex/repetitivecomputations

§ EventTypes – groupcommoneventsandshareknowledge

§ Lookups – augmentdatawithadditionalexternalfields

28

Page 29: Machine Data 101 workshop audienceversion

§ Addsinlinemeaning/context/specificitytorawdata

§ Usedtonormalizemetadataorrawdata

§ Simplifiescorrelationofmultipledatasources

§ CreatedinSplunk

§ Transferredfromexternalsources

WhatisDataEnrichment?

29

Page 30: Machine Data 101 workshop audienceversion

§ Addmeaning/context/specificitytorawdata

§ Labelsdescribingteam,category,platform,geography

§ Appliedtofield-valuecombination

§ Multipletagscanbeappliedforeachfield-value

§ Casesensitive

Tags

30

Page 31: Machine Data 101 workshop audienceversion

CreateTags

31

Page 32: Machine Data 101 workshop audienceversion

§ Searcheventswithtaginanyfield

§ Searcheventswithtaginaspecificfield

§ Searcheventswithtagusingwildcards

FindtheWebServersTagsinAction

32

tag=webserver

tag::host=webserver

tag=web*

§ Tagthehostaswebserver

§ Tagthesourcetypeasweb

1

2

3

4

5

Page 33: Machine Data 101 workshop audienceversion

§ Normalizefieldlabelstosimplifysearchandcorrelation§ Applymultiplealiasestoasinglefield

§ Example:Username|cs_username |Userà user§ Example:c_ip |client|client_ipà clientip

§ Processedafterfieldextractions+beforelookups

§ Canapplytolookups

§ Aliasesappearalongsideoriginalfields

FieldAliases

33

Page 34: Machine Data 101 workshop audienceversion

Re-LabelFieldtoIntuitiveNameCreateFieldAlias

34

1

2

3

Page 35: Machine Data 101 workshop audienceversion

§ Createfieldaliasofclientip=customer

§ Searcheventsinlast15minutes,findcustomerfield

§ Fieldalias(customer)andoriginalfield(clientip)arebothdisplayed

SearchusinganIntuitiveFieldNameFieldAliasinAction

35

1

3

2

sourcetype=access_combined

Page 36: Machine Data 101 workshop audienceversion

§ Shortcutforperformingrepetitive/long/complextransformationsusingevalcommand

§ Basedonextractedordiscoveredfieldsonly

§ Donotapplytolookuporgeneratedfields

CalculatedFields

36

Page 37: Machine Data 101 workshop audienceversion

ComputeKilobytesfromBytesCreateCalculatedField

37

1

21

2

3

Page 38: Machine Data 101 workshop audienceversion

§ Createkilobytes=bytes/1024

§ Searcheventsinlast15minutesforkilobytesandbytes

SearchUsingKilobytesinsteadofBytesCalculatedFieldsinAction

38

1

2

sourcetype=access_combined

Page 39: Machine Data 101 workshop audienceversion

§ Classifyandgroupcommonevents

§ Captureandshareknowledge

§ Basedonsearch

§ Useincombinationwithfieldsandtagstodefineeventtopography

EventTypes

39

Page 40: Machine Data 101 workshop audienceversion

§ BestPractice:Usepunctfield§ Defaultmetadatafielddescribingeventstructure§ Builtoninterestingcharacters:",;-#$%&+./:=?@\\'|*\n\r\"(){}<>[]^! »§ Canusewildcards

CreateEventTypes

40

event punct

####<Jun3,20145:38:22PMMDT><Notice><WebLogicServer><bea03><asiAdminServer><WrapperStartStopAppMain><>WLSKernel<><><BEA-000360><ServerstartedinRUNNINGmode>

####<_,__::__>_<>_<>_<>_<>_<>_

172.26.34.223- - [01/Jul/2005:12:05:27-0700]"GET/trade/app?action=logoutHTTP/1.1"2002953

..._-_-_[:::_-]_\"_?=_/.\"__

Page 41: Machine Data 101 workshop audienceversion

§ Showpunctforsourcetype=access_combined

§ Pickapunct,thenwildcarditafterthetimestamp

§ AddNOTstatus=200

§ Saveas“bad”eventtype+Color:red+Priority:1(shiftreloadinbrowsertoshowcoloring)

ClassifyEventsasKnownBadCreateEventType

41

eventtype=bad

sourcetype="access_combined" punct="..._-_-_[//_:::]*" NOT status=200

1

2

3

4

Page 42: Machine Data 101 workshop audienceversion

LookupstoEnrichRawData

LDAPAD

WatchLists

CRM/ERP

CMDB

ExternalDataSources

Insightcomesout

DatagoesinCreateadditionalfieldsfromtherawdatawithalookuptoanexternaldatasource

Page 43: Machine Data 101 workshop audienceversion

§ Augmentraweventswithadditionalfields§ Providecontextorsupportingdetails

§ Translatefieldvaluestomoredescriptivedata§ Example:addtextdescriptionsforerrorcodes,IDs§ Example:addcontactdetailstousernamesorIDs§ Example:adddescriptionstoHTTPstatuscodes

§ File-basedorscriptedlookups

Lookups

43

Page 44: Machine Data 101 workshop audienceversion

44

1.Upload/createtable

2.Assigntabletolookupobject

3.Maplookuptodataset

Convert a Code into a DescriptionConfigure a Static Lookup

Page 45: Machine Data 101 workshop audienceversion

§ GetthelookupfromtheSplunkWiki(saveto.csv file)http://wiki.splunk.com/Http_status.csv

§ Lookuptablefiles>Addnew§ Name:http_status.csv (musthave.csv fileextension)§ Upload:<pathto.csv>

§ Verifylookupwascreatedsuccessfully

1.CreateHTTPStatusTable

45

| inputlookup http_status.csv

1

2

3

Page 46: Machine Data 101 workshop audienceversion

§ Lookupdefinitions>Addnew§ Name:http_status§ Type:File-based§ Lookupfile:http_status.csv

§ Invokethelookupmanually

2.AddLookupDefinition

46

1

2

sourcetype=access_combined | lookup http_status status OUTPUT status_description

Page 47: Machine Data 101 workshop audienceversion

§ Automaticlookups>Addnew§ Name:http_status (cannothavespaces)§ Lookuptable:http_status§ Applyto:sourcetype=access_combined§ Lookupinputfield:status§ Lookupoutputfield:status_description

§ Verifylookupisinvokedautomatically

3.ConfigureAutomaticLookup

47

1

2

sourcetype=access_combined

Page 48: Machine Data 101 workshop audienceversion

§ Temporallookupsfortime-basedlookups§ Example:IdentifyusersonyournetworkbasedontheirIPaddress

andthetimestampinDHCPlogs

§ Usesearchresultstopopulatealookuptable§ … | outputlookup <tablename|filename>

§ Callanexternalcommandorscript§ Pythonscriptsonly§ Example:DNSlookupforIPßà Host

§ Createalookuptableusingarelationaldatabase§ ReviewmatchesagainstadatabasecolumnorSQLquery

FancyLookups

48

Page 49: Machine Data 101 workshop audienceversion

§ CreatingandManagingAlerts(JobInspector)

§ Macros

§ WorkflowActions

MoreDataEnrichment

49

Page 50: Machine Data 101 workshop audienceversion

LevelUponSearch&ReportingCommands

Page 51: Machine Data 101 workshop audienceversion

Agenda

§ Doingmorewithbasicsearchcommands

§ Advancedsearchcommands

§ Doingmorewithbasicreportingcommands

51

Page 52: Machine Data 101 workshop audienceversion

SearchSyntaxComponents

52

Page 53: Machine Data 101 workshop audienceversion

AnatomyofaSearch

53

Disk

Page 54: Machine Data 101 workshop audienceversion

§ top– limit§ rare– sameoptionsastop§ timechart– parameters§ stats– functions(sum,avg,list,values,sparkline)§ sort– inlineascendingordescending§ addcoltotals§ addtotals

DoingMorewithBasicSearchCommands

54

Page 55: Machine Data 101 workshop audienceversion

§ Commandshaveparametersorqualifiers

§ topandrarehavesimilarsyntax

§ Eachsearchcommandhasitsownsyntax– showinlinehelp

FindMostandLeastActiveCustomersUsingthetop+rareCommands

... | top limit=20 clientip

... | rare limit=20 clientip

IPswiththemostvisits

IPswiththeleastvisits

Page 56: Machine Data 101 workshop audienceversion

§ Sortinlinedescendingorascending

56

... | stats count by clientip | sort - count

... | stats count by clientip | sort + count

Numberofrequestsbycustomer- descending

Numberofrequestsbycustomer- ascending

SorttheNumberofCustomerRequestsUsingthesortCommand

Page 57: Machine Data 101 workshop audienceversion

§ ShowSearchCommandReferenceDocs§ Functionsforeval+where§ Functionsforstats+chartandtimechart

§ Invokeafunction

§ Renameinline

57

... | stats sum(bytes) by clientip | sort - sum(bytes)

... | stats sum(bytes) as totalbytes by clientip | sort - totalbytes

Totalpayloadbycustomer- descending

Totalpayloadbycustomer- ascending

DetermineTotalCustomerPayloadUsingfunctions+renamecommand

Page 58: Machine Data 101 workshop audienceversion

§ Listallvaluesofafield

§ Listonlydistinctvaluesofafield

58

... | stats values(action) by clientip

... | stats list(action) by clientip

Activitybycustomer

Distinctactionsbycustomer

ObserveCustomerActivityUsingthelist+valuesFunctions

Page 59: Machine Data 101 workshop audienceversion

§ Showdistinctactionsandcardinalityofeachaction

59

sourcetype=access_combined| stats count(action) as value by clientip, action| eval pair=action + " (" + value + ")"| stats list(pair) as values by clientip

AnalyzeCustomerActivityCombinelist+valuesFunctions

Page 60: Machine Data 101 workshop audienceversion

§ Addcolumns

§ Sumspecificcolumns

60

... | stats count by clientip, action

2cols:clientip +action

... | stats sum(bytes) as totalbytes, avg(bytes) as avgbytes, count as totalevents by clientip | addcoltotals totalbytes, totalevents

Sumtotalbytesandtotaleventscolums

BuildingaTableofCustomerActivityAddColumnsandSumColumns

Page 61: Machine Data 101 workshop audienceversion

61

... | stats sum(bytes) as totalbytes, sum(other) as totalother by clientip | addtotals fieldname=totalstuff

Foreachrow,addtotalbytes+totalother

Abetterexample:physicalmemory+virtualmemory=

totalmemory

BuildingaTableofCustomerActivitySumAcrossRows

Page 62: Machine Data 101 workshop audienceversion

62

... | stats sparkline(count) as trendline by clientip

Incontextoflargereventset

... | stats sparkline(count) as trendline sum(bytes) by clientip

Inlineintables

TrendIndividualCustomerActivitySparklinesinAction

Page 63: Machine Data 101 workshop audienceversion

AdvancedSearchCommandsCommand ShortDescription Hints

transaction Groupeventsbyacommonfieldvalue. Convenient,but resourceintensive.cluster Clustersimilareventstogether. Canbeusedon_raworfield.associate Identifiescorrelationsbetweenfields. Calculatesentropybtn fieldvalues.correlate Calculatesthecorrelationbetween

differentfields.Evaluatesrelationshipof allfieldsinaresultset.

contingency Buildsacontingencytablefortwofields. Computesco-occurrence,or%twofieldsexistinsameevents.

anomalies Computesanunexpectednessscoreforanevent.

Computessimilarityofevent(X)toasetofpreviousevents(P).

anomalousvalue Findsandsummarizesirregular,oruncommon,searchresults.

Considers frequencyofoccurrenceornumberofstdev fromthemean

Page 64: Machine Data 101 workshop audienceversion

§ Seweventstogether+createsduration+eventcount

§ Sparklinesinlineintables

64

... | transaction JSESSIONID | table JSESSIONID, action, product_id

GroupbyJSESSIONID

ViewCustomerActivitybySessionUsingthetransactionCommand

Page 65: Machine Data 101 workshop audienceversion

§ Intelligentgroup(createscluster_countandcluster_label)

65

... | cluster showcount=1 | table _raw, cluster_count, cluster_label

AutomaticallyGroupCustomerActivityUsingtheclusterCommand

Page 66: Machine Data 101 workshop audienceversion

§ Predictovertime

§ ChartOverlaywithandwithoutstreamstats

§ Mapswithiplocation+geostats

§ Singlevalue

§ Meteredvisualswithgauge

DoMorewithBasicReportingCommands

66

Page 67: Machine Data 101 workshop audienceversion

§ Predictfuturevaluesusinglower/upperbounds– singleandmultipleseries

67

... | timechart count as traffic | predict traffic

PredictWebsiteTrafficUsingthepredictCommand

Page 68: Machine Data 101 workshop audienceversion

68

sourcetype=access_combined (action=view OR action=purchase)| timechart span=10m count(eval(action="view")) as Viewed,

count(eval(action="purchase")) as Purchased

CompareBrowsingvs.BuyingActivitySimpleChartOverlay

Page 69: Machine Data 101 workshop audienceversion

69

... | iplocation clientip | geostats count by clientip

CombineIPlookupwithgeomapping

MapCustomerActivity GeographicallyGeolocation inAction

Page 70: Machine Data 101 workshop audienceversion

70

... | stats count

DisplayaSimpleCountofEventsSingleValueinAction

Page 71: Machine Data 101 workshop audienceversion

DisplayCountsUsingGaugesSingleValue,RadialandFillerGaugesinAction

71

... | stats count | gauge count 10000 20000 30000 40000 50000

Page 72: Machine Data 101 workshop audienceversion

DataModelandPivot

Page 73: Machine Data 101 workshop audienceversion

Agenda

§ Whatisadatamodel?

§ Buildadatamodel

§ PivotInterface

§ Accelerateadatamodel

73

Page 74: Machine Data 101 workshop audienceversion

PowerfulAnalyticsAnyoneCanUse

Enablesnon-technicaluserstobuildcomplexreportswithoutthesearchlanguage

Providesmoremeaningfulrepresentationofunderlyingrawmachinedata

Accelerationtechnologydeliversupto1000xfasteranalyticsoverSplunk5

74

Pivot

DataModel

AnalyticsStore

Page 75: Machine Data 101 workshop audienceversion

DefineRelationshipsinMachineDataDataModel• Describeshowunderlyingmachinedataisrepresentedandaccessed

• Definesmeaningfulrelationshipsinthedata

• Enablessingleauthoritativeviewofunderlyingrawdata

Hierarchicalobjectviewofunderlyingdata

Addconstraintstofilteroutevents

Page 76: Machine Data 101 workshop audienceversion

TransparentAcceleration

• Automaticallycollected– Handlestimingissues,

backfill…• Automaticallymaintained– Usesaccelerationwindow

• Storedontheindexers– Peertothebuckets

• Faulttolerantcollection

Timewindowofdatathatisaccelerated

Checktoenableaccelerationofdatamodel

HighPerformanceAnalyticsStore

Page 77: Machine Data 101 workshop audienceversion

Easy-to-UseAnalytics

• Drag-and-dropinterfaceenablesanyusertoanalyzedata

• Createcomplexqueriesandreportswithoutlearningsearchlanguage

• Clicktovisualizeanycharttype;reportsdynamicallyupdatewhenfieldschange

Selectfieldsfromdatamodel

Timewindow

Allcharttypesavailableinthecharttoolbox

Savereporttoshare

Pivot

Page 78: Machine Data 101 workshop audienceversion

§ Definesleastcommondenominatorforadatadomain

§ Standardmethodtoparse,categorize,normalizedata

§ Setoffieldnamesandtagsbydomain§ PackagedasaDataModelsinaSplunkApp

§ Domains:security,web,inventory,JVM,performance,networksessions,andmore

§ MinimalsetuptousePivotinterface

CommonInformationModel(CIM)App

78

Page 79: Machine Data 101 workshop audienceversion

§ Apps>FindMoreApps>

§ Search:“CommonInformationModel”

§ Installfree

§ Showfieldsforweb+WebDataModel

DownloadCIMApp

79

1

2

3

4

Page 80: Machine Data 101 workshop audienceversion

DataModel&PivotTutorial

http://docs.splunk.com/Documentation/Splunk/latest/PivotTuto

rial/WelcometothePivotTutorial

80

Page 81: Machine Data 101 workshop audienceversion

CustomVisualizationsandtheWebFrameworkToolkit

Page 82: Machine Data 101 workshop audienceversion

Agenda

§ DeveloperPlatform

§ WebFrameworkToolkit(WFT)

§ RESTAPIandSDKs

§ GetaFlyingStart

82

Page 83: Machine Data 101 workshop audienceversion

OptimizingtheAnalyticsProcess

83

Focusonthedata– intuitivetoolstoenabletheanalyst

Nosinglevisualizationexiststohandlealldatasets.

Neverlosesightoftherawdata

SplunkAnalytics

Explore

Context

Visualize

Algorithms

Page 84: Machine Data 101 workshop audienceversion

6.0+6.1:Simple,Interactive,andExtensible

84

VISUALIZATIONEXPLORATION

CUSTOMIZABLEFRAMEWORK

POWERFULANALYTICS

PivotDataModels

InteractiveFormsContextualDrilldown

DashboardEditorWebFramework

Page 85: Machine Data 101 workshop audienceversion

TheSplunkEnterprisePlatform

Collection

Indexing

SearchProcessingLanguage

CoreFunctions

Inputs,Apps,OtherContent

SDKContent

CoreEngine

UserandDeveloperInterfaces

WebFramework

RESTAPI

Page 86: Machine Data 101 workshop audienceversion

What’sPossiblewiththeSplunkEnterprisePlatform?

PowerMobileApps

LogDirectly

ExtractData

CustomerDashboards

IntegrateBITools

IntegratePlatformServices

Developer Platform

Page 87: Machine Data 101 workshop audienceversion

PowerfulPlatformforEnterpriseDevelopersDevelopersCanCustomizeandExtend

RESTAPI

BuildSplunkApps ExtendandIntegrateSplunk

SimpleXML

JavaScript

HTML5

WebFramework

JavaJavaScriptPython

RubyC#PHP

DataModels

SearchExtensibility

ModularInputs

SDKs

Page 88: Machine Data 101 workshop audienceversion

SplunkSoftwareforDevelopers

GainApplicationIntelligence

BuildSplunkApps

IntegrateandExtendSplunk

Page 89: Machine Data 101 workshop audienceversion

AWealthofSplunk AppsOver1,100appsavailableontheSplunkappssite

APISDKs UI

Server, Storage, Network

Server Virtualization

Operating Systems

Custom Applications

Business Applications

Cloud Services

App Performance MonitoringTicketing/ and

Other

WebIntelligence

Mobile Applications

Stream

Page 90: Machine Data 101 workshop audienceversion

§ Interactive,cut/pasteexamplesfrompopularsourcerepositories:D3,GitHub,jQuery

§ Splunk6.xDashboardExamplesApphttps://apps.splunk.com/app/1603

§ CustomSimpleXML ExtensionsApphttps://apps.splunk.com/app/1772

§ SplunkWebFrameworkToolkitApphttps://apps.splunk.com/app/1613

ExampleAdvancedVisualizations

90

Page 91: Machine Data 101 workshop audienceversion

91

http://www.d3js.org

Page 92: Machine Data 101 workshop audienceversion

AddaD3BubbleChart

92

1. GotoFindMoreAppsandInstalltheSplunk6.xDashboardExamplesApp

2. EntertheApp3. GotoExamples>CustomVisualizations>

D3BubbleChart4. Copyautodiscover.js (file)+components/bubblechart (dir)

from:$SH/etc/apps/simple_xml_examples/appserver/staticto:$SH/apps/search/appserver/static

5. CopyandpastesimpleXMLtonewdashboard

Page 93: Machine Data 101 workshop audienceversion

Resources

Page 94: Machine Data 101 workshop audienceversion

SplunkDocumentation

94

• http://docs.splunk.com• OfficialProductDocs• Wikiandcommunitytopics• Updateddaily• Canbeprintedto.PDF

Page 95: Machine Data 101 workshop audienceversion

SplunkAnswers

95

• http://answers.splunk.com• Communitydriven• Splunksupported• Knowledgeexchange• Q&A

Page 96: Machine Data 101 workshop audienceversion

SplunkEducation

96

• RecommendedforUsers– UsingSplunk– Searching&Reporting

• RecommendedforUI/DashboardDevelopers– DevelopingApps

• Instructor-LedCourses– Web– Onsite