Top Banner
Grigori Melnik, Principal Product Manager – Splunk Developer Platform Copyright © 2015 Splun Inc. Splunk for Developers Grigori Melnik Principal Product Manager Developer Platform @gmelnik Seattle
34
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Copyright © 2015 Splunk Inc.

Splunk for DevelopersGrigori Melnik

Principal Product ManagerDeveloper Platform

@gmelnik

Seattle

Page 2: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform2

EMPOWERING DEVELOPERS

Gain Application Intelligence

Build Splunk Apps

Integrate & Extend Splunk

Page 3: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform3 3

www.splunk.com/apptitude

Page 4: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Copyright © 2015 Splunk Inc.

Splunk for Application Development

Page 5: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

BuildUnit Testing

Code

Check-in IntegrationTesting Deploy

Staging

Application Development Challenges

5

Page 6: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

BuildUnit Testing

Code

Check-in IntegrationTesting Deploy

Staging

Lack of visibility across the product development lifecycle

Pressure to increase velocity and agility with DevOps

Limited insights into behavior and performance from application logs

Application Development Challenges

6

Page 7: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Quickly trace and identify errors anywhere in the codebase with real-time search and monitoring

Instrument your app logs to gain application intelligence

Break down dev tool silos with real-time insights from machine data

GAIN END-TO-END VISIBILITY ACROSS THE DEV TOOL CHAIN

FIND AND FIX ISSUES FASTER

PUSH BETTER CODE USING ANALYTICS

Splunk for Application Lifecycle Intelligence

7

Page 8: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Real-time dashboards show error rate in production and impact of pushing

new builds

Developers can search and visualize web logs, Java logs, eventlogs etc;

trace tx without complex instrumentation

Alerts notify developers as soon as a problem arises

Find and Fix Issues Faster

8

Page 9: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Gain end-to-end visibility to make informed decisions

Analytics insights without the need for additional analytics tools

Ask questions while exploring and collecting data

Push Better Code Using Analytics

9

Page 10: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform10 10

CI / Build Servers

Project and Issue Tracking

Code Repository

QA / Testing Tools

End-To-End Visibility Across The Dev Tool Chain

Deployment Servers / Automation

Page 11: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform11

Page 12: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform12

CI / Build Server

Code Review

Task Tracking

What Data Can You Splunk?

Logs – Which code has already been reviewed for this release/sprint? Who has completed the most code reviews? What code has NOT been reviewed?

Logs/API – Who is changing files? What kinds of files are being changed? What branches are most active? What types of activities are occurring for a branch?

Version Control

Logs/API – How many builds completed today/this week/this month? Which check-in kicked off this build? Which tests ran against this failed build?

Logs – Which tasks are assigned to which developers? What progress is being made to complete assigned tasks? What tasks remain for this release/sprint?

12

Page 13: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform13

Key Benefits of Application Lifecycle IntelligenceReduced Time

to Market

Shrink the time it takes to get code through dev/test to market

through faster issue identification and

resolution

Increased Agility

With real-time visibility into processes like code

check-ins, builds and tests to support

DevOps practices like continuous integration

“Our devs are now able to find and fix issues five to ten

times faster.”

“We can monitor all the automation and handoffs it takes to deploy 5-10 times

a day”

Application Insights

Instrument customer application logs to

capture critical business events and

user behavior

“My code isn’t ready until it’s Splunk-ready”

13

Page 14: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Copyright © 2015 Splunk Inc.

Demo:ADLC

Page 15: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Copyright © 2015 Splunk Inc.

Touring the Splunk Development Platform

Page 16: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform16

Evolving the Splunk Platform

Collection

Indexing

Search Processing Language

Core Functions

Inputs, Apps, Other Content

SDKs & plug-ins

Operational Intelligence Platform

Content

Core Engine

User and Developer Interfaces

Web Framework

REST API

Page 17: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform17

Powerful Platform for Enterprise Developers

17

REST API

Build Splunk Apps Extend and Integrate Splunk

Simple XML

JavaScript/CSS Extensions C#JavaScriptPython

RubyJavaPHP

Data Models

Search Extensibility

Modular Inputs

SDKs

KV Store

Page 18: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Log directly to Splunk via TCP,

UDP, HTTP

Integrate search results with other applications using

custom visualizations

Create and run searches from

other applications

The REST API and SDKs

18

VisualizeSearch Manage

Add/Delete Users

Manage Inputs

Index

Page 19: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform19

The Splunk REST APIExposes an API method for every feature in the product– Whatever you can do in the UI – you can do through the API– Index, Search, Visualize, Manage

API is RESTful– Endpoints are served by splunkd– Requests are GET, POST, and DELETE HTTP methods– Responses are Atom XML & JSON – Versioning as of Splunk 5.0– Search results can be output in CSV/JSON/XML

19

Page 20: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform20

SDKs Overview

20

Stay true to the semantics of the particular language• E.g. Keep Python “pythonic”• E.g. C#: Fully async , PCL, support for Rx

Provide implementation that feels natural to the developer

• E.g. Project, build, IDE (where applicable) support

Cover REST API endpoints based on use cases of languageNamespaces• owner: splunk username (defaults to current user)• app: app context (defaults to default app)• sharing: user | app | global | system

Page 21: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

A Developer’s Smörgåsbord Data ingestion

InputScripted inputs Modular inputsCustom (trained) source

types Custom sources

Data ingestion pipelineField extractions Field transformations

IndexingCustom indexes

SearchingSearch authoring

Custom search commandsMacros (basic,

parametrized)Saved searches

Data classificationEvent typesTransactions

Data enrichmentLookupsKV store collectionsWorkflow actions

Data normalizationTagsAliases

Data mining cluster & dedupanomalousvaluekmeanspredict commands …

Processing & reporting

Search-time mappingData models

CIM extensionsCustom UI/visualizations

Pages, views & dashboardsJS ExtensionsCSS ExtensionsCustom setup screens

Scheduled processingScheduled reports

AlertingScripted alerts

Branding & navigationCustom app navigation &

brandingManageability

Custom splunkweb controllers

Custom splunkd endpoints

Page 22: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Copyright © 2015 Splunk Inc.

Building Splunk Apps

Page 23: SplunkLive! Seattle - Splunk for Developers

Splunk Developer Guidance

Splunk Reference AppsComplete, working real-world Splunk solutions built together with partners (Conducive; Auth0)̶Z 2 (pseudo-) production releases̶Z entire code & test repos on GitHub̶Z under Apache 2.0

Associated GuidanceI. Start-to-Finish Journey Documentary

II. Essentials

dev.splunk.com/goto/devguide

Page 24: SplunkLive! Seattle - Splunk for Developers

1. Started with a Questions BacklogArchitecture– What does a typical Splunk application reference architecture look like?– What common paradigms are applicable to Splunk app development?– What are the typical deployment topologies? Why should I choose a specific one? What are the confounding factors on

the choice of my topology?– How do I partition my Splunk solutions?– What are the tradeoffs of various types of inputs?– How do I architect my Splunk solution and deployment for a very large scale?– How do I architect my Splunk solution for the cloud? What are specific considerations for deploying to AWS or Azure?– What’s the landscape of Splunk extension points?– How do I integrate data from Splunk into existing applications and systems?– How do I plan and design a robust alerting and monitoring subsystem on top of Splunk?– What should I consider for my sizing requirements?– What are recommended configurations of Splunk deployment to meet my sizing requirements?– Should I architect my solution to index my data in local data center (zone) or centrally?– What are things we can automatically degrade so we can make sure our core experience is working?– When something happens, how do I effectively propagate the info and react to it?– How are other solutions on Splunk built? What were the challenges? How have they been addressed?Packaging and Deployment– How do I piece together various parts of a Splunk app (custom search commands, mod inputs etc.)?– How do I package a Splunk solution with a single install that automatically rolls out all the necessary dependencies?– How do I manage my Splunk solution versioning, backward and future compat?– What's the best way to split up custom apps for deployment? Development– How should I set up my development environment to be productive with Splunk?– What are different ways of how I develop my Splunk app ? Pros and cons of using specific SDK vs REST APIs?

Pros and cons of using SimpleXML vs Advanced XML vs Web Framework …– How do I analyze a data source for a TA?– What are the different ways of enriching the data in Splunk? What are their tradeoffs?– When should I use event types and transactions for data classification?– How do I extend Splunk to define a custom input capability? – When should I use modular inputs vs scripted inputs vs..?– What are streaming vs non-streaming outputs considerations?– How do I deal with long-running scripts? Handling shutdown/restart of Splunk? Concurrency? State persistence etc.– Why should I not use transactions?– When should I use pivot vs tstats?– Why should I use data models?– When my data source touches on many data models, should I assume complete separation or heavy inheritance?

– How do I extend an existing data model?– What does CIM offer and why should I build CIM-compliant apps?– In the context of CIM, what are the tradeoffs of using my props.conf and transforms.conf and rewriting them on

indexing, completely discarding the vendor supplied field names? How do I reconcile the advantages of a clean interface & normalisation, but at the cost of losing alignment with published vendor documentation, and a learning curve for existing users?

– How do I manage my solution declarative configuration? How do I detect/troubleshoot bad config?– How do I log and analyze data that is not event driven (certain web feeds, html parsing, image meta data)?– Compare and contrast ad-hoc searching vs background searching– How do I handle transient faults?– How do I effectively manage credentials?– What’s the effect of search head location on my app and the overall user experience?– How do I develop an integrated mechanism to let me connect Splunk to my MOM (messaging middleware) and index

my messages?– How do I handle the requirement that app configs must be different across different server types in a distributed

environment (e.g. apps on search heads shouldn't have inputs enabled)?Quality/Compliance– What quality gates should I consider? What kind of para-functional characteristics are important to consider?– What heuristics do I use to bless/block a release?– How do I test a data model?– How do I prepare event generation when building/testing an app?– What kind of perf testing should I do and how?– How do I test UI?– How do I security certify my solution?– How do I design to satisfy my retention and compliance policies?– How do I architect to design my availability requirements?– How do I handle geographic disaster recovery / fault tolerance?– How do I properly instrument my solution so that I know what’s happening?Sustained Engineering– How do I maintain/service/support Splunk apps?– How do my customers handle updating their customized configs once new versions of my app come out? Business – Why should I build on Splunk?– What kind of skill do I need my devs to have to build a Splunk solution?– What is the community building? How are current devs creating unique experiences using Splunk – I typically want to

see some marketplace success– Cost and pricing are very important to me as a entrepreneur developer. If I am coming in to build a tool that will be

commercialized I need to know that the cost structure of Splunk won’t cause my service to be economically unprofitable.

What does a typical Splunk application architecture look like?How should I set up my dev environment to be productive with Splunk?How do I integrate Splunk into existing systems?How do I prepare my event generation when developing &

testing an app?How do I package an app? deal with app versioning and updates?

Page 25: SplunkLive! Seattle - Splunk for Developers

2. Mined business requirements with partner3. Formulated learning objectives4. Reconciled 2 & 3 with our designs

Page 26: SplunkLive! Seattle - Splunk for Developers

Data Search language Aggregating siloed metrics into

meaningful KPIs Data manipulation Data normalization Sub-searches Config-driven Persistence with KV store Macros

Viz: Dynamic scaling Customizing in-the box viz

controls

General search patterns Search optimizations Ux Prototyping Adapting 3rd party viz library Composite charts with interactions Dealing with high-volume data sets Troubleshooting perf issues Post-process or not-post-process –

deployment implications Automated UI testing (w.Selenium)

Setting the stage Overall Splunk app structure UI technology selection:

Simple XML vs SplunkJS Modularity Dev & test env Dev workflow Modularity Data onboarding CIM compliance Tools

Post-processing Integrating with 3rd party

component Unit testing (w.Mocha) Persisting state (per user)

Data modeling Using lookups Building a baseline lookup table Windows of time/Custom time ranges Overlaying time data

Using sub-searches to correlate data Troubleshooting searches

Custom nav Ux activities permeating all dev

Data mining: Exploration Preparation: filtering/deduping/

bucketing Using advanced statistics functions Threshold-based anomaly detection Evaluating goodness /accuracy

Plus non-functional topics:

App versioning Packaging Installation

Security review Deployment Publishing to splunkbase App certification

Page 27: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Copyright © 2015 Splunk Inc.

Demo:Building solutions with Splunk Reference App

Page 28: SplunkLive! Seattle - Splunk for Developers

Copyright © 2015 Splunk Inc.28

Splunk Reference App comes preinstalled in the Cloud Sandbox - www.splunk.com/goto/cloud

Page 29: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform

Copyright © 2015 Splunk Inc.

Resources

Page 30: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform30

Splunk Developer License

30

Page 31: SplunkLive! Seattle - Splunk for Developers

Grigori Melnik, Principal Product Manager – Splunk Developer Platform31

Where to go for more InfoTutorials, Code Samples, Getting Started, Downloads– http://dev.splunk.com

Splunk Developer Guidance– http://dev.splunk.com/goto/devguide

Splunk Base (Apps)– https://splunkbase.splunk.com

GitHub– https://github.com/splunk

Twitter– https://twitter.com/splunkdev

Blogs– http://blogs.splunk.com/dev

31

Page 32: SplunkLive! Seattle - Splunk for Developers

Copyright © 2015 Splunk Inc.32

Takeaways

Application development intelligence

Platform, not just an engine

Open & extensible

On-prem and cloud

Developer Guidance : learn and reuse for the win!

Reach out to my team ([email protected]) and tell us about your experience

@gmelnik / [email protected]

Page 33: SplunkLive! Seattle - Splunk for Developers

33

The 6th Annual Splunk Worldwide Users’ Conference

September 21-24, 2015 The MGM Grand Hotel, Las Vegas4000 IT & Business Professionals2 Keynote Sessions 3 days of technical content– 165+ sessions3 days of Splunk University– Sept 19-21, 2015– Get Splunk Certified for FREE! – Get CPE credits for CISSP, CAP, SSCP, etc.– Save thousands on Splunk education!

80 Customer Speakers

80 Splunk Speakers

35+ Apps in Splunk Apps Showcase

65 Technology Partners

Ask The Experts and Security Experts, Birds of a Feather, Chalk Talks and a new & improved Partner Pavilion!

Register at conf.splunk.com

Page 34: SplunkLive! Seattle - Splunk for Developers

34

We Want to Hear your Feedback!

After the Breakout Sessions conclude

Text Splunk to 878787

And be entered for a chance to win a $100 AMEX gift card!