Transcript
1
SPLUNK OVERVIEW
ALEXANDER FOK
BIG DATA ARCHITECT
FEBRUARY 2017
2
• What is Splunk?
• Splunk Main Functionality
• Why Splunk?
• Demo
Agenda
3
• Real Time logs collection, indexing and data analytics
• Time Series data management
• Search query language
tail –f ALL_LOGS*| grep “WHATEVER YOU NEED”
tail –f SOME_LOGS| grep “WHATEVER YOU NEED”| count by InterestingField
Commands Pipe
tail –f SOME_LOGS| grep “WHATEVER YOU NEED”| count by InterestingField
3
Splunk Main Functionality
4
• Presentation level – graphs, tables, etc
• Historical analysis
• Automation capabilities
• APIs
REST
Command line
Data Exploration and Visualization Capabilities
5
• Rolls Royce in the field
• Proven field success
• Flexible, User friendly, modern tool
• Enterprise grade – users access management, security, multitenant platform, data retention management policy
•Reach Echo system
•Splunk appstore –splunkbase.splunk.com
5
Why Splunk?
6
•Strong Visualization Capabilities – reports, dashboards
•Infinite scale – up to hundreds of TB logs per day
•Strong Post processing capabilities - Calculated and Extracted Fields
•Various Optimizations
Frequent Reports precalculation
Lookup tables
Field Tags
•Advanced Data Models - CIM
Why Splunk?
7
sourcetype=mysql_config OR sourcetype=remedy_changeticket
| dedup _raw, User
| transaction TicketId, User
| eval hasTicket = if(eventcount > 1, "Yes", "No")
| rename PrevPropValue as "Original_Value", NewPropValue
as "New_Value", hasTicket as "Change_Ticket"
| fields _time, User, Property, "Original_Value",
"New_Value", "Change_Ticket"
8
• Web logs• Log4J, JMS, JMX• .NET events• Code and scripts
• Configurations• syslog• SNMP• netflow
• Configurations• Audit/query logs• Tables• Schemas
• Hypervisor• Guest OS, Apps• Cloud
• Configurations• syslog• File system• ps, iostat, top
• Registry• Event logs• File system• sysinternals
Logfiles Configs Messages Traps Alerts
Metrics Scripts TicketsChanges
Linux/UnixWindows NetworkingDatabasesApplicationsVirtualization
& Cloud
• Click-stream data• Shopping cart data• Online transaction
data
Customer Facing Data
Outside the Datacenter
• Manufacturing, logistics…
• CDRs & IPDRs• Power consumption• RFID data• GPS data
No predefined schema, no custom connectors, no RDBMS, no need to filter/forward.
Splunk – The Big Picture
8
9
Splunk Architecture
10
Splunk’s MapReduce-based Architecture10
Chunk 1
Chunk 2
Chunk 3
Chunk 4
Chunk 1
Chunk 2
Chunk 3
Chunk 4
Chunk 1
Chunk 2
Chunk 3
Chunk 4
Search Head
map
map
map
map
map
map
map
map
map
Answer
reduce
Server 1 Server 2 Server N
time
11
•Events, Indexes, Fields – key value pairs, columns
•Index Time
events are processed, classified, time stamp is extracted indexed
Predefined Fields are extracted
events can be enriched
Events can trigger logic -> alerts, reports, dashboards updates etc
•Search Time
events are searched
fields are extracted or calculated
transactions are closed
Visualizations can be built
Splunk Typical WorkFlow
12
•Show events counts by SFlow
•SFlow|stats count by SFlow
•| transaction SAUPID startswith="Product Start" endswith="Product End"
Demo
13
What is An App?
• Terminology
• Apps – A workspace that solves a specific use case with a navigable view
• Add-on – A reusable Splunk component that does not contain a view
• Example
• Splunk for Cisco Security is an App
• The collection of field extractions/sourcetypes/transforms/eventypes thatmap raw firewall logs is an Add-on
14
•CIM – Common Information Model
•Domain centric data models – OSSEC, networking, ticket management
•Data normalization
•Validation
•Visualization
•Action generation
Splunk as SIEM
15
Marathon Tel Aviv 2017 – See you tomorrow
16
Alexander Fok, Big Data Architect
THANK YOU
top related