Using Splunk to Protect Students, Faculty and the University

Copyright © 2014 Splunk Inc.

Chris KurtzSystem ArchitectArizona State University

Using Splunk to Protect Students, Faculty and the University

2

DisclaimerDuring the course of this presentation, we may make forward looking statements regarding future events or the

expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important

factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other

commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.

Additional Speaker Disclaimer: While I am speaking as an employee of Arizona State University, I do not speak for the University nor dictate policy, procedures, or purchases. Any and all statements made in this presentation are mine

alone, and do not in any way represent an official statement from ASU. The opinions and comments contained herein are entirely my own. ASU does not endorse or represent any product mentioned, up to and including Splunk.

3

Agenda

Introduction to me and Arizona State University– About ASU– About me– Our Environment and our challenges

Use Cases and Examples– Protecting Direct Deposit, two versions– Phishing as a teaching tool– Leveraging your institutional data with lookups and apps

Conclusion: Where we’ve been, where we’re going!

Introduction

4

5

Largest single University in the USMore than 80,000 active students……and another 20,000 accounts (faculty/staff, alumni, affiliates)Located in Tempe, Arizona, suburb of Phoenix, 6th Largest City in USNot located on the surface of the sun…but you can see it from here!

6

Obligatory About Me: ProfessionallyUnix/Linux System Administrator by trade, 23 years experienceSupported NASA/JPL Mars projects at ASU for more than 10 years:

TES & THEMIS Instrument onboard Mars Global Surveyor & Mars OdysseyMTES Instrument on the Mars Exploration Rovers Spirit and Opportunity

ASU’s “Splunk Guy” (System Architect) since early 2013Splunk Video Interview “Value of Higher Education and Splunk”Author of the ISO 3166 Splunk App – more on this later!

7

Obligatory about me: Personally• Self-proclaimed Geek, what’s it to ya?• Steampunk Enthusiast (I made my hat,

goggles, and the gun!)• Beginning Maker (Steampunk and

Arduino/Electronics)• xoff on #splunk on efnet

Little known fact about me:Clyde Tombaugh, the discoverer of Pluto,

was a personal friend growing up

http://about.me/chk

8

First Google Apps for Education customerMultiple campuses with a diverse IT infrastructureMany organic, home-grown, custom, and proprietary systemsLarge number of governing requirements:FERPA, HIPPA, DARPA, DoJ, NASA, JPL, etcClear separation of responsibilities inside the University Technology Office: the Information Security Office (ISO) does not have access to the systems (and more importantly the logs) run by Operations

9

The Power of SplunkSplunk as ASU’s universal aggregator of all machine generated logs

Logs reside in multiple locations, depending on when and where the system was installed: web logs in one location, system logs multiple others (depending on OS); some are on single log concentrator and some in an “old, slow, and unsupported” proprietary search database. ISO requests logs for incident. Ops has to use the proprietary tools (or just as often, just grep through multiple logfiles) based on ISO description and email/share logs. ISO likely has to revise request at least once.

Typical response time to incident: multiple business days

ISO directly accesses logs in Splunk, often using pre-built dashboards, alerts, and saved searches. Ops can concentrate on Operations.

Typical response time to incident: minutes!

WithoutSplunk

WithSplunk

10

Splunk and Arizona State UniversityLicensing

• 750gb/day– Started at 50gb in November 2012– …to 150gb in February 2013– …to 500gb in June 2013– …to 750gb in July of 2014– On track to reach 1TB this FY

Infrastructure• Physical Indexers in Cluster

– ~14TB in Hardware RAID10– NFS for Cold (being phased out)– Architected for 1TB (10 indexers)

• Search Head Pooling– 3 virtual servers (12 CPUs, 32gb)– NFS SSD storage for shared data

• Virtual Support servers:– Deployment Server– License Manager– Cluster Master

The value of Splunk to the Information Security Office has driven the rapid growth…

but other groups are starting to see the value!

11

We didn’t know…

“To ASU, Splunk was like the invention of the microscope: we didn’t know what we couldn’t

see.”Martin Idaszak

Security Architect, Arizona State University

Protecting Direct Deposit

12

13

Use Case: Protecting Direct Deposit• Being able to change your employee

information online is a great convenience, but a target for hackers

• Because of ASU’s international students, faculty, and staff, just blocking other countries isn’t acceptable…

• Splunk is the solution!

Purrfessor Mittens(Advanced String Theory)

14

How we did it…before Splunk.

1. Payroll gets a call that an employee didn’t get their direct deposit.2. …investigates, sees a foreign bank deposit…and contacts the

Information Security Office.3. ISO changes the user’s password.4. ISO requests webserver single sign-on and HR HR system logs from

Operations and our HR Vendor (could take days!)5. Eventually details are discovered (compromised account) and user

is informed. Funds are long gone, and ASU has to re-issue the employee’s check, eating the loss.

15

How we did it with Splunk, Version 1

1. Logs from webserver single sign-on and Peoplesoft now go to Splunk. No more waiting on Operations to retrieve logs! This makes both ISO and Ops very happy!

2. Splunk monitors for Direct Deposit changes via a schedule search, building a transactions to link the change back to the user’s webserver authentication. Ok, now we have an originating IP and a username…so we run geolocation on the originating IP so it’s easier to create reports based on location of the change.

Web auth DB records

IP username

Geo tagcountry

user address

All unusual changes

IPusernamestate/country

16

Version 1 stop here:• ISO creates a scheduled report of unusual originating IPs (Malaysia, etc) and sends it to Payroll

before the close of each payroll run.• Payroll contacts users with unusual changes for verification before payroll is run and if it was a

fraudulent change, the change is reverted, so no funds are lost.• Even at this point, Payroll is ecstatic and saves over 30 hours per payroll run reviewing direct

deposit, and ASU saves tens of thousands of dollars per payroll run!

17

Now…How do we improve this?We asked the question: Where do you change your direct deposit from?

1. Home

So, let’s think about it:If your direct deposit changes from Malaysia, it’s probably fraud…

but what about Ohio, if you live in Arizona?

2. Work

That’s likely fraud, too!

So let’s leverage Splunk’s geolocation features!

18

…Version 2 (now in progress)1. Starting with the originating IP and username from Version 1…we use a

custom lookup tables (more later!) to leverage HR system data, so we can lookup a username’s information: Name, address, etc.

2. Geolocation information about the user’s home zip code (via the zip code) is generated.

3. Using a free Splunk App called haversine, we calculate the distance between the user’s home (technically, the lat/lon of the center of their zipcode) and the lat/lon of the IP the change was made from. We realize both of these are a bit vague, but we’re really only looking for scale.

4. If the distance is unusual (~50 miles) the result will be flagged for Payroll review automatically.

19

Lessons learned…and you can do this too!1. GET YOUR DATA INTO SPLUNK!2. One of the beautiful things about Splunk is that you can modify how the data appears (field

extractions, etc.) once it’s already in Splunk, and that applies to already indexed data. The focus should be getting it into Splunk first, and figure out fields later. Think of it as schema on demand!

3. When you find people who “get it” use them to evangelize Splunk to others in the organization.

4. When you find people who resist, show them how much time and effort they can save, especially interacting with other departments (if appropriate) by using Splunk. We won several people over when they discovered that the number of requests from groups like ISO dropped from 3-5 per week (each taking hours to do) to zero once the data was in Splunk.

5. Don’t get caught up on “use cases”: Once you have the data in Splunk, use cases present themselves again and again! Think of it as use case on demand!

20

Flexibility

“It’s not only it’s schema-on-the-fly,it’s use-case-on-the-fly.”

- Barak ReevesSplunk Sales Engineer, Team TK-421

Phishing as a teaching tool

21

22

Use Case: Phishing as a teaching tool• As a public University, a large amount of our information

is mandated to be publically available, including a directory of email addresses…and we have over 100,000 users, and each can have as many email addresses as they want…

• This means ASU receives a lot of email: In fact, we used Splunk to determine exactly how much. In the last 12 months, ASU received more than ONE BILLION email messages, and more than 750 million of them were spam and phishing!

As usual…Splunk is the solution!

MandatoryPie Chart

Phishing and ASU

InboundPhishing

Email

Mail Filter

EmailStored

User clickson phishing link

Firewall blocks some

Some gets through

Firewall

24

ASU is hard to protectASU, as an entity, is very hard to protect. We have students from all across the world, and by their nature, they are very transient: they move apartments, dorms, travel the US and abroad, and access ASU systems from almost everywhere. Unlike most corporations, we can’t assume that access to ASU from Nigeria, China, or Malaysia are hacking attempts…in fact, it’s probably legitimate!

One of the very first things we saw with Splunk were logins on campus and from India for the same user on the same day. What was this? Hacking? VPN? Multiple people using the same

login? Turns out Indian students often gave their passwords (gasp!) to their parents, who insisted on it, so the parents could regularly check their grades! This let another project to provide limited

access to secondary accounts (just for this purpose) know that their efforts were valid and necessary!

25

Use the data you have…To protect ASU from spam, we use Barracuda Spam & Virus Firewalls, but there is no Splunk app (yet) so we make custom field extractions from the Barracuda logs. …but ASU does not store user emails in Splunk, only the headers of the messages that transit our system.

Do managers ever ask you if a product is worthwhile? We regularly use Splunk to show that other products

are doing their jobs!)

Seems legit?

Phishing and ASUCorrelate Firewall information with our mail logs to get a list of every user who clicked on a phishing link.

Firewall log Email log

IP Bad URL userEmail with link

Table of user clicking bad link

CMDB for contact

27

…and let your data combine!BUT…ASU also uses Palo Alto firewalls to protect our users. These firewalls very often catch phishing URLs that users click on, either via mistake or lack of understanding… and we correlate that Palo Alto information with our mail logs to get a list of every user who clicked on a phishing link.The ISO can then directly contact the users who clicked on a phishing link, explain to them why they need to change their password (and probably run a virus/malware scan), and use the opportunity to explain to the user why what they did was bad. The users are thankful that the University is watching out for them, and some of the potential victims have become our best reporting sources for received phishing and spear phishing emails!

This too is being automated! We plan to use workflows to allow ISO to easily flag a potentially compromised account in Splunk, which (via a REST API call to our authentication system) is automatically disabled and (via another REST API) a ticket is created for the helpdesk, so they can explain the situation to the user when they call in because their password no longer works.

28

…Version 2 (now in progress)1. ISO actively follows phishing links (from a secure and isolated Virtual Machine) and

enters bogus credentials. We are now using Splunk to alert on attempted logins using those honeypot credentials. These active hackers are then blocked on the Palo Alto Firewalls in a quick but manual process…this protects users who might click on the phishing. Eventually, we plan to semi-automate this using Splunk workflows that let ISO directly block several different types of attackers from Splunk, using the Palo Alto’s APIs.

2. ASU is investigating using honeypot full email accounts that will be scraped from the public directory and then sent spam/phishing attempts just like real users. The plan is to use Splunk to index the entire email, so we will have the full body of phishing and spam emails as well as headers. Phishing URLs identified would be blocked using a workflow to the Palo Alto APIs, as above, and the from addresses would be blocked on the Barracudas with their APIs.

29

Lessons learned…and you can do this too!1. LEVERAGE YOUR DATA!2. Combining data from multiple sources is amazing! We use data from the

Barracuda Spam Firewalls as well as the Palo Alto Firewalls to provide multiple points of visibility into phishing.

3. Standardize your data! Follow Splunk’s Common Information Model so that field names are consistent across data types. Once you realize that src_ip, for example, exists in multiple datasets, the possibilities just jump out at you!

4. Fill in the gaps. When you find gaps in your data models, work on how to fill them in. For us, it’s the honeypot registrations and full-email indexing. Once we realized full-email indexing was possible (and easy!) all sorts of new use cases appeared!

30

Value of Splunk

“This is the best tool we’ve seen in 10 years.”- Jay Steed

AVP for UTO Operations, Arizona State University

Leveraging your own custom data

31

32

The Power of Splunk!No schemas! This means if you need to alter your data structure (field extractions, calculated fields, etc.) you can easily do it on the fly, and it’s retroactive! No types! Splunk really doesn’t care if “42” is a string or a number, so you can divide 42 by 7 and get 6, or add a string to make it “42 is the answer” just as easily to modify a field or make a new one on the fly.Eval is your friend!Remember…It doesn’t matter if data is from a logfile, database, textfile, script output, or anything else…combine it in any way you want, on the fly!

Why mention this? Because as a Splunk Admin always remember: the data structure is mutable!If it doesn’t work for your needs, change it on the fly!

33

To correlate data, you have to have data to correlate…

Having data from machine logs such as mailservers and firewalls is great, it’s the first (and easiest) data to get into Splunk.Without a common key, there is no way to know that two pieces of data refer to the same individual.For ASU, the master datasource is the Data Warehouse. These databases contain the records for every student and employee.

Does the email [email protected] belong to John Bunbury?

34

Lookups from Databases• Isolated Splunk server running Database Connect (DBX) runs SQL Queries on

several databases, and writes a series of lookup tables (with the affiliate ID) every 4 hours

• Linux ionotify monitors the lookup tables, and on write-close copies data to production systems (sanity checking applies)

Data Warehouse Isolated Splunkrunning DBX

Production Splunk

100000001, jbunbury7, John Bunbury, [email protected], student100000002, jbunbury, Jane Bunbury, [email protected], employee

35

Splunk (and most other applications) use the ISO3166 standard “alpha-2” country codes (US for United States, for example). This is standard for geolocation services in Splunk.But…our Oracle Databases for Student data get the data from the students, often their passports. And machine-readable passports use the ISO3166 “alpha-3” country codes…and there isn’t a simple conversion!If the Country Code is not in the standard geolocation format, I can’t do any geolocation, which means the data is far less useful.I looked on the Splunk Apps site (http://apps.splunk.com) but didn’t find a solution…

Problem is…

Country alpha-3 alpha-2

United States USA US

China CHN CN

Nigeria NGA NG

36

So, I wrote the app myself!Very simple structure, but so useful!I took the online ISO 3166 country codes (3 kinds: alpha-3, alpha-2, and numeric) and built a lookup table, which I call in the dbquery search before outputting the lookup table

Lookup Sample:alpha-2,alpha-3,numericUS,USA,840CN,CHN,156NG,NGA,566

| dbquery "PS PRD" "SELECT EMPLID,CITY,STATE,POSTAL,COUNTRY_CODE FROM EDS_ADDRESS" | dedup EMPLID CITY STATE POSTAL COUNTRY_CODE | lookup iso3166 iso3166_alpha-3 as COUNTRY_CODE| eval city=upper(substr(CITY,1,1)).lower(substr(CITY,2)) | rename STATE as region_name EMPLID as affiliate_id POSTAL as postal_code iso3166_alpha-2 as country_code | eval postal_code=if(country_code="US",substr(postal_code,1,5),postal_code) | table affiliate_id,city,region_name,postal_code,country_code | outputlookup affiliate_to_address.csv

Why bother publishing as an app? Because it might be useful to someone else, and at least 2 people have now said to me:

“Wow, thanks, that solves my problem!”

37

Building an App is simple!

1. In etc/apps, create a directory for your app, with appropriate subdirs (default is mandatory)

2. All config files go in default – nothing in local!3. Write an appropriate default/app.conf (look at other apps)4. Create a README file and other appropriate documentation.5. Package and test on a generic Splunk install for sanity (hint .spl files are just tgz

files!)6. Upload to apps.splunk.com – if something isn’t right, it’ll let you know.7. Make sure to put the docs online!

http://wiki.splunk.com/Community:Creating_your_first_application

My app took me about a day to do, including an obsessive amount of research on how to do it.

38

#splunk

“It is days like today when I am stuck with a piece of crappy software with horrible documentationand support that I am very thankful that I spend

the rest of my time dealing with Splunk.”- David Shpritz (automine) Splunk IRC channel

Conclusion

39

40

The past and the future

ASU has heavily invested in Splunk because it solves many of our outstanding issues, and a culture of “how can we use Splunk to solve this?” is developing.First round (FY14) of data onboarding concentrated on the needs of the Information Security Office. Second round (FY15) is focusing on Operations needs, with some interesting use cases thrown in as they appear.Splunk is expensive, but the savings in man hours, extreme flexibility, use to validate other systems, and goals to replace antiquated systems has very much paid off.

41

Get some help!Splunk Docs (http://docs.splunk.com) – I use Splunk docs so much I have a Chrome shortcut to just search it. And if you do occasionally find something that is unclear, use the links at the bottom to provide feedback…the team is great at responding!Splunk Answers (http://answers.splunk.com) – I always look (and often post in Answers) here before I contact support. Just looking at what others are posting is often just what you need to rephrase the question to find the answers you need. The users who are on answers are the true heroes of Splunk. In fact there is only one group better…The Splunk Wiki – specifically http://wiki.splunk.com/Things_I_wish_I_knew_then The #splunk IRC channel on efnet (http://wiki.splunk.com/Community:IRC) – Ok, I admit it, I’m a Splunk IRC junkie. This group is just the best…a great mix of Splunkers (aka Splunk employees), customers, and professional services and hysterical to boot. Also check out @splunk, @splunkdev, and @splunkanswers on Twitter

42

I look to the future because that’s where I’m going to spend the rest of my life.

- George Burns

43

Questions and mentioned links

My Splunk App to do ISO 3166 translations: http://apps.splunk.com/app/1775/Free Splunk App to calculate distances on a globe (a “Great Circle” or haversine calculation): http://apps.splunk.com/app/936/ My Splunk Video:http://www.splunk.com/view/SP-CAAAJPW

44

Special Thanks!

IRC Gang: duckfez, DaGryph, cgales, alacer, baconesq, ^Brian^, automine, starcher, Yorokobi, coccyx, madscient, and many many more!Splunkers: Rachel Perkins (piebob), Kelly Feagans, Adam Tice, Mike Clayborn, Jeff Goodman, Dimitri McKay and everyone else!ASU: Martin Idaszak, Ryan Adler (Trex), Jim Salverson, Sean Ganser, Mike Brown, OJ Redhair, Tina Thorstenson, Jay Steed, and Jack Hsu

THANK YOU