Gaining Application Lifecycle Intelligence
Post on 14-Jun-2015
624 Views
Preview:
DESCRIPTION
Transcript
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
GAINING APPLICATION LIFECYCLE
INTELLIGENCE WITH SPLUNK
By Damien Dallimore , Dev Evangelist @ Splunk
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Who am I ?
2
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/3
From Middle Earth
Make things
JVM background
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/4
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/5
apps.splunk.com
github.com/damiendallimore
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Agenda
6
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/7
Overview of Splunk and build a simple app
How Splunk can help in the Application Development Lifecycle
Various ways to get data into Splunk and demos
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Data Data Everywhere
8
VOLUME
VARIETY
VERACITY
VELOCITY
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
How can Splunk help ?
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Spelunking
10
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Platform for machine data
11
Splunk storage Other Big Data stores
DeveloperPlatform
Data collection
and indexing
Report and
analyze
Custom dashboards
Monitor and alert
Ad hoc search
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
DeveloperPlatform
Report and
analyze
Custom dashboards
Monitor and alert
Ad hoc search
Platform for machine data
12
Splunk storage Other Big Data stores
Data
collection
and indexing
Any amount, any location, any source.Schema at read time, not write time
Data in any format
No RDBMS
Very Extensible
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
What Does Machine Data Look Like?
13
Sources
Care IVR
Middleware Error
Order Processing
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Machine Data Contains Critical Insights
14
Customer ID Order ID
Customer’s Tweet
Time Waiting On Hold
Twitter ID
Product ID
Company’s Twitter ID
Sources
Care IVR
Middleware Error
Order Processing
Customer IDOrder ID
Customer ID
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Machine Data Contains Critical Insights
15
Order ID
Customer’s Tweet
Time Waiting On Hold
Product ID
Company’s Twitter ID
Sources
Care IVR
Middleware Error
Order Processing
Order ID
Customer ID
Twitter ID
Customer ID
Customer ID
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/16
How are we best
going wrangle this
data ?
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Release the Developers
17
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Very Extensible Platform for Developers
18
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
Django
Web
FrameworkJava
JavaScript
Python
Ruby
C#
PHP
Data Models
Search
Extensibility
Modular Inputs
SDKs
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Lets build something simple
19
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Simple Swarm App (ex Foursquare)
20
Get my actual checkin data in via REST
Search over this data
Visualize
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Application Lifecycle Data
21
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
BuildUnit Testing
Code
Check-inIntegration
Testing Deploy
Staging
22
Application Development Challenges
Lack of visibility across the product
development lifecycle
Pressure to increase velocity and
agility with DevOps
Limited insights into behavior and
performance from application logs
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Quickly trace and identify errors anywhere
in the codebase with real-time search
and monitoring
Instrument your app logs to gain application intelligence
Break down dev tool silos with real-time
insights from machine data
GAIN END-TO-END VISIBILITY
ACROSS THE DEV TOOL CHAIN
FIND AND FIX
ISSUES FASTER
PUSH BETTER
CODE USING
ANALYTICS
Splunk for Application Lifecycle Intelligence
23
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Real-time dashboards show error
rate in production and impact of
pushing new builds
Developers can search and
visualize web logs, Java logs—
without production access
Alerts notify developers as soon as
a problem arises
24
Find and Fix Issues Faster
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Gain end-to-end visibility to make
informed decisions
Analytics insights without the need
for additional analytics tools
Ask questions while exploring and
collecting data
void submitPurchase(purchaseId){
log.info("action=submitPurchaseSt
art, purchaseId=%d", purchaseId)
//these calls throw an exception on error
submitToCreditCard(...)
generateInvoice(...)
generateFullfillmentOrder(...)
log.info("action=submitPurchaseCo
mpleted, purchaseId=%d", purchaseId)
}
25
Push Better Code Using Analytics
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
End-To-End Visibility Across The Dev Tool Chain
26
CI / Build
Servers
Project and
Issue Tracking
Code
Repository
QA / Testing
ToolsDeployment
Servers
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
App Development Lifecycle Demo
27
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Getting your data into Splunk
28
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/29
Log DataLog Files
Splunk Logging Appenders
CodingSplunk Java SDK
Splunk Spring Integration Adaptors
JMX
MessagingJMS
AMQP w/Rabbit
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Log Data
30
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Standard Log Files
31
Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1153 ms
Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Application logs that are part of the product
Developer logs for any code that was deployed
Written to local disk or network storage
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Structured and Unstructured Data
32
Ideally events are in a best practice semantic format
key=value format , JSON
You can perform index time and search time extractions in Splunk
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Logging best practices
33
Clearly timestamp every event , human readable at beginning of line
Log in text , binary needs decoding
Categorize – Use INFO, WARN, ERROR, DEBUG, Event type etc...
Log unique identifiers
Log anything that can add value when aggregated, charted or further
2012-08-07 15:54:06:644+1200 name="Failed Login" event_id="someID" app="myapp" user="jane" somefieldname="foobar"
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
SplunkJavaLogging
34
Sometimes you can’t write to file
Appenders for Java Util Logging , Log4J , Logback
Simply add a logging appender to your logging configuration
file
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
LogBack Appender Example
35
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Code
36
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Better Exception Logging
37
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Easier to work with in Splunk
38
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Coding
39
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Splunk SDK for Java
40
Use the SDK from any JVM Language , Java / Groovy / Scala etc….
Send log events via REST , UDP or TCP directly to Splunk from your
code
Search over data in Splunk
SDK available from dev.splunk.com
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/41
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Spring Integration Adaptors
42
Inbound Adapter
Used to execute Splunk searches and get data out
Outbound Adapter
Write data to Splunk via REST, TCP , UDP
Write to a named index, submit a REST request, write to a data input bound
to a server TCP port
Get the code on Github
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Inbound Adaptor
43
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Outbound Adaptor
44
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
JMX
45
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
What is JMX
46
JMX = Java Management Extensions
Monitor JVM via MBean attributes , operations and
notifications
JVM MBeans
Vendor MBeans
Custom Coded MBeans
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Getting this data into Splunk
47
Runs on all supported Splunk platforms
Works with all main JVM variants
100% Free and Open Source
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Simple to Configure
48
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Many Connectivity Options
49
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Messaging
50
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
JMS
51
Not a messaging protocol , but a programming interface to many
different underlying message providers
WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc…
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
AMQP
52
Built with Rabbit Java Client Library
AMQP 0.9.1, 0.9, 0.8
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
But wait , there’s more…..
53
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/54
Poll data from any REST API
Pull data directly off the wire
Capture output from executing any commands
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Closer look at capturing command output
55
Let’s see what the host Operating System can tell us : top
External programs that provide additional JVM insights : jstat
Index this data in Splunk and correlate
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
top
56
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/57
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
jstat
58
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/59
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/60
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Splunk options galore
61
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.100% uptime.
Splunk> AMI : BYOL versions for Amazon AWS Cloud.
Splunk> Sandbox : Spin up a cloud instance in minutes.Load in data.
Hunk> : Splunk for data in Hadoop HDFS , MongoDB
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
More Info
62
Splunk Docs , Downloads , Vids : http://www.splunk.com
Download Splunk Apps : http://apps.splunk.com
Ask : http://answers.splunk.com
Watch the App Dev / Devops Video : http://www.splunk.com/goto/appdev
Splunk Developer Platform : http://dev.splunk.com
Splunk on Github : https://github.com/splunk
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Thankyou.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/
Questions ?@damiendallimore
ddallimore@splunk.com
top related