Top Banner
YOU’VE GOT JUNK IN YOUR SPLUNK Conner Swann NAU Information Technology Services
27

You've Got Junk In Your Splunk

Jan 23, 2017

Download

Software

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • YOUVE GOT JUNK IN YOUR SPLUNK

    Conner SwannNAU Information Technology Services

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE PROBLEM

    WHAT IS THE PROBLEM?

    Most enterprise data is machine-generated

    Machine data is often-times not human readable

    Numerous disparate data sources and formats

    Different implementations and architectures

    Virtualized Applications

    3rd Party Off-Site Solutions (The Cloud)

    On-Site Hardware

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE PROBLEM

    SERIOUSLY? THIS IS A PROBLEM?

    Dan the developer is asked to help figure out why his code is crashing on Sundays at Midnight

    Sally the SysAdmin has no idea why users from one office location cant log in to their computers

    Ivan the InfoSec Analyst has no idea a hacker in Bulgaria is sending spam from his servers

    Billy the Business Analyst needs to figure out what localities are using his companys applications

    Molly the Marketing Executive needs to analyze her affiliate marketing campaigns to see if improvements can be made

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE PROBLEM

    YES, ITS A PROBLEM.

    Machine Data is the most rapidly growing and complex segment of Big Data

    Its generated 24/7/365 by nearly every device in existence and will continue to be generated forever

    Contains categorical record of every activity and behavior

    Value from this data is largely untapped extremely difficult to process and analyze in a timely manner by traditional means

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE DATA

    SOMETHINGS GOT TO GIVE - UNDERSTANDING IMPORTANT DATA

    Business Application Data

    Relational Data, highly structured, inflexible schema

    Financial Records, multidimensional data, computationally intense at times

    Rare reports, never realtime

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE DATA

    SOMETHINGS GOT TO GIVE - UNDERSTANDING IMPORTANT DATA

    Human Generated Data

    Created as a result of Human-Human interaction

    Email, IM, Voice, Text, Video

    Stored in central corporate data centers, on mobile devices and on individual PCs

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE DATA

    SOMETHINGS GOT TO GIVE - UNDERSTANDING IMPORTANT DATA

    Machine Data

    Time series, diverse, unstructured, no predefined all-encompassing schema

    Encapsulates Human Generated Data

    Generated by all IT systems

    Absolutely ridiculous volume of data

  • YOUVE GOT JUNK IN YOUR SPLUNK - MACHINE DATA

    WHAT DOES MACHINE DATA LOOK LIKE?

    2015-10-17 13:08:51-0700 [SSHService ssh-userauth on HoneyPotTransport,2323,93.158.203.167] login attempt [root/12345] succeeded

    64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846

    {"created_at":"Mon Sep 28 19:39:04 +0000 2015,user:yourbuddyconner", "id":648582717068587000,"id_str":"648582717068587009","text":"The amount of local news stations treating the Facebook outage as news is too damn high. #FacebookDown #TwitterIsUp #Facebook,"entities":{"hashtags":[{"text":"FacebookDown","indices":[89,102]},{"text":"TwitterIsUp","indices":[103,115]},{"text":"Facebook","indices":[116,125]}],"symbols":[],"user_mentions":[],"urls":[]}}

    message_id=53088 timestamp="2015-02-03 20:30:06" date_read="2015-02-03 20:29:20" is_from_me=1 is_read=1 handle=+9999999999 service=iMessage message="I mean, I can, those pancakes were so good"

    Honeypot Logs:

    Webserver Logs:

    Tweets:

    Text Messages:

    SERVICE NAME

    USERNAME PASSWORD STATUS MESSAGEIP ADDRESS

    HTTP METHOD

    TIMESTAMP TWITTER HANDLE

    HASHTAGS

    PHONE NUMBER

    MESSAGE

    TIMESTAMP

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE SPLUNK

    ENTER SPLUNK

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE SPLUNK

    WHAT THE HECK IS SPLUNK?

    Splunk consumes text and provides insights about the data contained within

    Splunk stores your historical data and allows you to look at how the baselines have changed over time

    Splunk helps identify anomalies which might affect business decisions

    Splunk allows people who know their data to share it with people who dont

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE SPLUNK

    WHAT THE HECK IS SPLUNK?

    REACTIVE

    PROACTIVE

    SEARCH AND INVESTIGATE

    PROACTIVE MONITORING AND

    ALERTING

    OPERATIONAL VISIBILITY

    REAL-TIME BUSINESS INSIGHTS

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE FUN

    NOW FOR THE FUN PART!

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE FUN

    CASE STUDIES AND EXAMPLES

    7/11 - Uses Splunk to gain a business foothold in Indonesia, predicting shopping trends based on weather, among other things

    Information Security - Northern Arizona University uses splunk to trace intrusion attempts across our network

    Conner Swann (Thats Me) - Used splunk to glean metadata from text messages

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11)

    7/11 - THE CLIMATE

    Expanding to a new market (2009)

    Had to offer an attractive alternative to existing businesses

    Offer local foods, became a place local teens would hang out

    Caused competitors to adapt to new climate, occupying new niches

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11)

    7/11 - THE PROBLEM

    In order to retain their new customers, the company had to offer the best fast food as well as any daily necessities customers might need

    Necessitates a technological solution for providing behavioral insights on consumers

    Original data analytics solution was rigid, involved several rounds of manual analysis

    Analysis took 3-6 business days to complete

    Promotional campaigns took ~3 months to prepare

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11)

    7/11 - THE SOLUTION

    7/11 now uses Splunk for their POS analysis

    Assets are dynamically organized, delivering comprehensive overview of POS data from multiple perspectives

    System also leverages data from external systems (i.e weather, telecom)

    Data is processed in minutes instead of days

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11)

    7/11 - THE RESULT

    Promotion planning time slashed by 80% - 2 weeks

    All people involved have access to the same data and visualizations with little training

    Promotions are evaluates for effectiveness as they occur

    ROI is apparent

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (NAU INFOSEC)

    NAU INFORMATION SECURITY - EXAMPLE USE CASE

    Information Security is best when efforts are proactive

    Identify unwanted activity or actors and see if that data shows up anywhere else

    Honeypots on the network are used to collect data about intruders

    That data can be used to identify other anomalous behavior

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (NAU INFOSEC)

    HOW IT WORKS Northern Arizona University

    Hacker

    IP Address: 68.55.90.112

    Login Attempt From:68.55.90.112

    HoneyPot

    LouieSuccessful Login From:68.55.90.112

    SplunkAnomalous Events Detected:

    68.55.90.112 Sources:

    - Honeypot- Peoplesoft

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (NAU INFOSEC)

    THE IMPACT

    All event detection is done in real-time

    Incident response occurs as the event happen

    Remediation is simpler than in the past

    Easy to share impacts with non-technical people

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)

    TEXT MESSAGES

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)

    TEXT MESSAGES - THE WHY

    Personal analytics is HUGE

    Look for trends in communication

    Shows how much inferential data can be gleaned from behavior

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)

    TEXT MESSAGES - THE HOW

    Extracted messages from iPhone backups SQLite database

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)

    TEXT MESSAGES - THE RESULTS

    Average sentiment of outgoing texts over time

    index=text_messages is_from_me=1 | sentiment twitter message | timechart avg(sentiment) as sentiment span=1mon

    Conclusion: Sentiment fluctuates over time

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)

    TEXT MESSAGES - THE RESULTS

    Average sentiment of outgoing texts with baseline over time

    index=text_messages is_from_me=1 | sentiment twitter message |eval diff=sentiment-0.788400| eval count=count| timechart avg(diff) as sentiment, count span=14d

    Conclusion: Sentiment might correlate with life events and text message frequency

  • YOUVE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)

    TEXT MESSAGES - THE RESULTS

    Comparing incoming sentiment with outgoing sentiment

    index=text_messages is_from_me=0 | sentiment twitter message | eval diff=sentiment-0.788400 | timechart avg(diff) as sentiment_from span=1mon | appendcols [search index=text_messages is_from_me=1 | sentiment twitter message | eval diff2=sentiment-0.788400 | timechart avg(diff2) as sentiment_me span=1mon]

    Conclusion: Outgoing sentiment is at times closely coupled with incoming sentiment

  • YOUVE GOT JUNK IN YOUR SPLUNK - CONCLUSION

    PUT SOME JUNK IN YOUR SPLUNK!

    Splunk is free to play with

    (Developer Licenses are easy to come by)

    http://www.splunk.com/

    Provide value to the shareholders!

    http://www.splunk.com/