Page 1
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
GAINING APPLICATION LIFECYCLE
INTELLIGENCE WITH SPLUNK
By Damien Dallimore , Dev Evangelist @ Splunk
Page 2
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Who am I ?
2
Page 3
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/3
From Middle Earth
Make things
JVM background
Page 4
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/4
Page 5
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/5
apps.splunk.com
github.com/damiendallimore
Page 6
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Agenda
6
Page 7
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/7
Overview of Splunk and build a simple app
How Splunk can help in the Application Development Lifecycle
Various ways to get data into Splunk and demos
Page 8
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Data Data Everywhere
8
VOLUME
VARIETY
VERACITY
VELOCITY
Page 9
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
How can Splunk help ?
Page 10
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Spelunking
10
Page 11
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Platform for machine data
11
Splunk storage Other Big Data stores
DeveloperPlatform
Data collection
and indexing
Report and
analyze
Custom dashboards
Monitor and alert
Ad hoc search
Page 12
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
DeveloperPlatform
Report and
analyze
Custom dashboards
Monitor and alert
Ad hoc search
Platform for machine data
12
Splunk storage Other Big Data stores
Data
collection
and indexing
Any amount, any location, any source.Schema at read time, not write time
Data in any format
No RDBMS
Very Extensible
Page 13
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
What Does Machine Data Look Like?
13
Sources
Twitter
Care IVR
Middleware Error
Order Processing
Page 14
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Machine Data Contains Critical Insights
14
Customer ID Order ID
Customer’s Tweet
Time Waiting On Hold
Twitter ID
Product ID
Company’s Twitter ID
Sources
Twitter
Care IVR
Middleware Error
Order Processing
Customer IDOrder ID
Customer ID
Page 15
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Machine Data Contains Critical Insights
15
Order ID
Customer’s Tweet
Time Waiting On Hold
Product ID
Company’s Twitter ID
Sources
Twitter
Care IVR
Middleware Error
Order Processing
Order ID
Customer ID
Twitter ID
Customer ID
Customer ID
Page 16
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/16
How are we best
going wrangle this
data ?
Page 17
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Release the Developers
17
Page 18
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Very Extensible Platform for Developers
18
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
Django
Web
FrameworkJava
JavaScript
Python
Ruby
C#
PHP
Data Models
Search
Extensibility
Modular Inputs
SDKs
Page 19
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Lets build something simple
19
Page 20
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Simple Swarm App (ex Foursquare)
20
Get my actual checkin data in via REST
Search over this data
Visualize
Page 21
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Application Lifecycle Data
21
Page 22
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
BuildUnit Testing
Code
Check-inIntegration
Testing Deploy
Staging
22
Application Development Challenges
Lack of visibility across the product
development lifecycle
Pressure to increase velocity and
agility with DevOps
Limited insights into behavior and
performance from application logs
Page 23
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Quickly trace and identify errors anywhere
in the codebase with real-time search
and monitoring
Instrument your app logs to gain application intelligence
Break down dev tool silos with real-time
insights from machine data
GAIN END-TO-END VISIBILITY
ACROSS THE DEV TOOL CHAIN
FIND AND FIX
ISSUES FASTER
PUSH BETTER
CODE USING
ANALYTICS
Splunk for Application Lifecycle Intelligence
23
Page 24
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Real-time dashboards show error
rate in production and impact of
pushing new builds
Developers can search and
visualize web logs, Java logs—
without production access
Alerts notify developers as soon as
a problem arises
24
Find and Fix Issues Faster
Page 25
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Gain end-to-end visibility to make
informed decisions
Analytics insights without the need
for additional analytics tools
Ask questions while exploring and
collecting data
void submitPurchase(purchaseId){
log.info("action=submitPurchaseSt
art, purchaseId=%d", purchaseId)
//these calls throw an exception on error
submitToCreditCard(...)
generateInvoice(...)
generateFullfillmentOrder(...)
log.info("action=submitPurchaseCo
mpleted, purchaseId=%d", purchaseId)
}
25
Push Better Code Using Analytics
Page 26
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
End-To-End Visibility Across The Dev Tool Chain
26
CI / Build
Servers
Project and
Issue Tracking
Code
Repository
QA / Testing
ToolsDeployment
Servers
Page 27
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
App Development Lifecycle Demo
27
Page 28
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Getting your data into Splunk
28
Page 29
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/29
Log DataLog Files
Splunk Logging Appenders
CodingSplunk Java SDK
Splunk Spring Integration Adaptors
JMX
MessagingJMS
AMQP w/Rabbit
Page 30
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Log Data
30
Page 31
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Standard Log Files
31
Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1153 ms
Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Application logs that are part of the product
Developer logs for any code that was deployed
Written to local disk or network storage
Page 32
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Structured and Unstructured Data
32
Ideally events are in a best practice semantic format
key=value format , JSON
You can perform index time and search time extractions in Splunk
Page 33
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Logging best practices
33
Clearly timestamp every event , human readable at beginning of line
Log in text , binary needs decoding
Categorize – Use INFO, WARN, ERROR, DEBUG, Event type etc...
Log unique identifiers
Log anything that can add value when aggregated, charted or further
2012-08-07 15:54:06:644+1200 name="Failed Login" event_id="someID" app="myapp" user="jane" somefieldname="foobar"
Page 34
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
SplunkJavaLogging
34
Sometimes you can’t write to file
Appenders for Java Util Logging , Log4J , Logback
Simply add a logging appender to your logging configuration
file
Page 35
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
LogBack Appender Example
35
Page 36
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Code
36
Page 37
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Better Exception Logging
37
Page 38
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Easier to work with in Splunk
38
Page 39
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Coding
39
Page 40
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Splunk SDK for Java
40
Use the SDK from any JVM Language , Java / Groovy / Scala etc….
Send log events via REST , UDP or TCP directly to Splunk from your
code
Search over data in Splunk
SDK available from dev.splunk.com
Page 41
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/41
Page 42
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Spring Integration Adaptors
42
Inbound Adapter
Used to execute Splunk searches and get data out
Outbound Adapter
Write data to Splunk via REST, TCP , UDP
Write to a named index, submit a REST request, write to a data input bound
to a server TCP port
Get the code on Github
Page 43
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Inbound Adaptor
43
Page 44
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Outbound Adaptor
44
Page 45
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
JMX
45
Page 46
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
What is JMX
46
JMX = Java Management Extensions
Monitor JVM via MBean attributes , operations and
notifications
JVM MBeans
Vendor MBeans
Custom Coded MBeans
Page 47
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Getting this data into Splunk
47
Runs on all supported Splunk platforms
Works with all main JVM variants
100% Free and Open Source
Page 48
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Simple to Configure
48
Page 49
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Many Connectivity Options
49
Page 50
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Messaging
50
Page 51
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
JMS
51
Not a messaging protocol , but a programming interface to many
different underlying message providers
WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc…
Page 52
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
AMQP
52
Built with Rabbit Java Client Library
AMQP 0.9.1, 0.9, 0.8
Page 53
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
But wait , there’s more…..
53
Page 54
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/54
Poll data from any REST API
Pull data directly off the wire
Capture output from executing any commands
Page 55
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Closer look at capturing command output
55
Let’s see what the host Operating System can tell us : top
External programs that provide additional JVM insights : jstat
Index this data in Splunk and correlate
Page 56
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
top
56
Page 57
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/57
Page 58
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
jstat
58
Page 59
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/59
Page 60
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/60
Page 61
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Splunk options galore
61
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.100% uptime.
Splunk> AMI : BYOL versions for Amazon AWS Cloud.
Splunk> Sandbox : Spin up a cloud instance in minutes.Load in data.
Hunk> : Splunk for data in Hadoop HDFS , MongoDB
Page 62
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
More Info
62
Splunk Docs , Downloads , Vids : http://www.splunk.com
Download Splunk Apps : http://apps.splunk.com
Ask : http://answers.splunk.com
Watch the App Dev / Devops Video : http://www.splunk.com/goto/appdev
Splunk Developer Platform : http://dev.splunk.com
Splunk on Github : https://github.com/splunk
Page 63
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Thankyou.
Page 64
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Questions ?@damiendallimore
[email protected]