© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Design, Methodology and Best Practices Clayton Dukes
Cisco Systems, Advanced Services
BRKNMS-2031
@cdnw
2
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Session Abstract
“This session will help define the design and methodology for implementing a robust syslog solution using open source tools on Linux platforms.
It provides leading practices for deployment of a set of tools and applications to support effective collection, storage, and analysis of syslog messages.
This session provides examples using messages from Cisco IOS Software, but is applicable to all other syslog message types and general event management.”
Tweet about this session: #SYSLOG (but only if you’re saying how awesome it is )
3
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Meet the Engineer—Clayton Dukes
Joined Cisco in 2005 as a Network Consulting Engineer
Background was in routing and switching for 6+ years
Moved to Network Management/OSS automation around 1998
Frequent speaker at Networkers
Author of several Cisco.com whitepapers on network management architectures and large-scale syslog deployment
Self Portrait
4
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Topics
Why Syslog?
Syslog Basics
The Syslog Message
Relevant IOS commands
Syslog vs. SNMP
Management Techniques/Methodologies
Syslog Analysis
Syslog Architectures
Analysis Tools
Implementation Walk-Through (Using Open Source Tools)
5
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Topics
Why Syslog?
Syslog Basics
The Syslog Message
Relevant IOS commands
Syslog vs. SNMP
Management Techniques/Methodologies
Syslog Analysis
Syslog Architectures
Analysis Tools
Implementation Walk-Through (Using Open Source Tools)
6
“Cat6500 IOS 12.2(18)SXF contains about 90 SNMP traps, but has over 6000 syslog event messages.”
Source: Me!
Not this guy
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Why Syslog?
Proactive Syslog management benefits both operations personnel and the company as a whole. Successful event management provides:
Reduced downtime through operational effectiveness
Improved Incident Management through real-time detection and self-remediation
Reduced volume of incidents through proper problem management
Reduced severity of business interruptions
Proactive measures to reduce the need for post-mortem troubleshooting
8
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Why Syslog?
%CDP-4-DUPLEXMISMATCH: ‒ Duplex Full/Half between connections
%ENVMON-3-FAN_FAILED: ‒ Fan failure – may cause overheating
%ENVMON-3-OVER_HEAT: ‒ Device temperature is > 60C (140F)
%PQUICC-5-COLL: ‒ Excessive collisions - broken or unterminated Ethernet cable
%SYS-3-CPUHOG: ‒ The indicated process has run for too long a period of time without relinquishing
the processor
Ignoring Syslog Doesn’t Mean Your Network Isn’t Failing or Degraded
9
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Downtime = Lost Revenue Reducing Downtime Through Proactive Problem Management Reduces Operational Cost
0.09
0.09
0.1
1.2
2.6
4.5
0 1 2 3 4 5
Transportation
Retail
E-Commerce
Media
Banking
Brokerage
Industry Cost of Downtime
Revenue Loss Per Hour (in Millions of Dollars)
10
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Basics
11
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
The Syslog Protocol
Syslog is a client/server protocol ‒ The syslog sender sends a small (less than 1KB) text message to the syslog
receiver. The receiver is commonly called "syslogd", "syslog daemon" or "syslog server". Syslog messages (RFC 3164) can be sent via UDP (514) and/or TCP*. The data is typically sent in clear text.
Originally developed in the 1980s by Eric Allman as part of the Sendmail project, syslog is now standardized within the syslog working group of the IETF Syslog is supported by a wide variety of devices and receivers across
multiple platforms. Because of this, it can be used to integrate log data from disparate systems into a central repository for real-time and historical analysis.
* TCP support is available with some syslog daemons, such as syslog-ng or rsyslog as well as Cisco IOS Software Releases after 12.4(11)T, 12.2(33)SRB, 12.2(33)SB, and Cisco IOS XE Release 2.1 12.2(33)SXI.
12
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
RFC’s
13
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
RFC 3164
http://www.ietf.org/rfc/rfc3164.txt
Written in August 2001 by Chris Lonvick/Cisco Systems
Obsoleted by RFC 5424 (although most vendors still use 3164)
14
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
RFC 5424
http://tools.ietf.org/html/rfc5424
Written in March 2009 by Rainer Gerhards/Adiscon
Provides a method for adding Structured Data to a message which is meant to make it easier to parse and analyze the message once received.
Has many advantages over RFC 3164, but is also a lot more complex.
Meant to replace RFC 3164, but it doesn’t seem to get used much – perhaps because of the complexity?
15
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
The Syslog Message
Every syslog message should contain five distinct fields with the following information:
Facility
Severity
Hostname
Timestamp
Message
16
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Message Facility
Syslog messages are broadly categorized on the basis of the sources that generate them such as OS, process or application and are represented in integers ranging from 0-23, Cisco devices use the local facility ranges 16-23 (local0 – local7)
By default, Cisco IOS devices, CatOS switches, and VPN 3000 Concentrators use facility local7 while Cisco Firewalls use local4
17
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Message—Severity
0 - Emergency: System Is Unusable
1 - Alert: Action Must Be Taken Immediately
2 - Critical: Critical Conditions
3 - Error: Error Conditions
4 - Warning: Warning Conditions
5 - Notice: Normal But Significant Condition
6 - Informational: Informational Messages
7 - Debug: Debug-Level Messages
The log source or facility (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7
Leading Practice
Network Devices Should Log Levels 0-6
Level 7 Should Be Used for Console Troubleshooting
18
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Message—Hostname
The hostname field consists of the host name (as configured on the host itself) or the IP address
Note: Don’t be confused by “host name” and “hostname”. “Hostname” Is typically associated with a DNS lookup. If the syslog message contains a “host name”, it may be (and often is) different than the actual DNS hostname of the device.
19
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Message—Timestamp
The local time, in MMM DD HH:MM:SS format, of the device when the message was generated
The * and . characters preceding a syslog message are indicators of a problem with NTP. ‒ * Means that time is not authoritative: the software clock is not
in sync or has never been set.
‒ . Means that time is authoritative, but NTP is not synchronized: the software clock was in sync, but has since lost contact with all configured NTP servers
*Jun 28 08:50:47.359 EDT: %SYS-5-CONFIG_I: Configured from console by Skeeter McGillicutty (10.10.86.123)
For the Timestamp Information to Be
Accurate, It Is Good Administrative Practice
to Configure All the Devices to Use the
Network Time Protocol (NTP)
Leading Practice
20
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Message—Message Text
This is the text of the syslog message, along with some additional information about the process that generated it
Messages generated by most Cisco devices begin with a percent sign (%) and use the following format:
‒%FACILITY-SEVERITY-MNEMONIC: Message-text
‒ The mnemonic is a device-specific code that uniquely identifies the message such as “up”, “down”, “changed”, “config”, etc.
‒ The term “facility” in Cisco Mnemonics are not the same as the IETF definition of “facility” (such as local7). Cisco facilities are a free-form method of identifying the source message type such as SYS, IP, LDP, L2, MEM, FILESYS, DOT11, LINEPROTO, etc. (the list is very large)
*Sep 16 08:50:47.359 EDT: %SYS-5-CONFIG_I: Configured from console by vty0 (172.18.86.123)
21
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Relevant IOS Commands service timestamps debug datetime localtime show-timezone
service timestamps log datetime localtime show-timezone
clock timezone GMT 0
!
logging source-interface Loopback0
logging buffered 65536
logging host 192.168.100.20
logging host 192.168.100.21
logging host 192.168.100.22
logging trap informational
!
ntp server 143.232.55.5
ntp server 204.34.198.40
ntp peer 192.168.100.2
ntp peer 192.168.100.3
ntp update-calendar
22
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Configuration Command Detail—Time service timestamps debug datetime localtime show-timezone
service timestamps log datetime localtime show-timezone
clock timezone GMT 0
Time stamps can be added to either debugging or logging messages independently
Datetime
Adds time stamps in the format MMM DD HH:MM:SS, indicating the date and time according to the system clock
Uptime
Adds time stamps in the format HHHH:MM:SS, indicating the time since the system was rebooted
23
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Configuration Command Detail—Logging
logging source-interface loopback0
The “logging source-interface” command instructs the system to generate logging to the remote system from this source interface
Ensures that all messages appear to come from the same IP and makes it easier to track in the destination syslog receiver
Allows you to create a DNS entry for that source interface
24
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Logging (Cont.)
logging buffered 65536
Used to reserve a memory buffer for logging to the console of the device
Since today’s devices have plenty of memory, feel free to set this number higher than the old 16k buffer, but be aware that there is a point of diminishing returns
The typical recommendation is to have 256k buffers on core devices and 64k elsewhere
Note: Console refers to the output of the screen when attached to the device either by serial or via telnet/ssh using the “Terminal Monitor” command.
25
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Logging (Cont.)
logging host <ip address 1>
logging host <ip address 2>
logging host <ip address 3>
Sets the remote syslog daemon to send messages to
Use a maximum of four syslog servers
The syslog server can then be configured to forward messages to other Network Management Systems
Devices Should Be Configured with a Maximum of 3-4
Destination Servers
Leading Practice
26
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Logging (Cont.)
logging trap informational
Sets the syslog server logging level (emerg through debug)
The term “Trap” here has nothing to do with SNMP Traps
“Trap” in this context is simply telling the device to log the specified severity levels.
It’s NOT a Trap!
27
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Configuration Command Detail—NTP
ntp server <ip address 1> ntp server <ip address 2> ntp peer <ip address 3> ntp peer <ip address 4> ntp update-calendar
The “ntp update-calendar” command is used to synchronize the time of the internal clock with the clock of the NTP reference server
28
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
NTP Recommendations
Use a minimum of two reference clocks (GPS and Internet derived are popular)
“Peer” time between the reference clocks
29
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Additional/Useful Logging Statements
logging count
Enables local device metrics on syslog messages
http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_logging_count_ps6350_TSD_Products_Configuration_Guide_Chapter.html#wp1025043
30
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog vs. SNMP Notifications
Can’t I just turn on SNMP traps? ‒ The simple answer is: no. In general there are significantly
more syslog messages available within IOS as compared to SNMP Trap messages
‒ Cat6500 IOS 12.2(18)SXF contains about 90 SNMP traps, but has over 6000 syslog event messages
If You Had to Pick SNMP Traps or Syslog, Go with Syslog; However, a Truly Robust and Full-Featured Event Management Solution Would Take Advantage of All Fault Indicators
31
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Management Techniques #SYSLOG
32
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Traditional Syslog Management
File-based storage ‒Traditionally, syslog daemons would store all incoming messages to one or more files for later parsing. This led to a very reactive use of syslog for after-the-fact troubleshooting and could not scale beyond very few devices
grep and tail ‒Great tools in their own right, but hardly useful for scraping through gigabytes of log data…better tools are necessary
33
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
A Better Way
Store all incoming messages in a database
Provides speed and scalability
Capable of storing thousands of messages per second
Allows for trending and metrics
34
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
How?
Syslog-ng ‒“An open source implementation of the syslog protocol for UNIX and UNIX-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport”
syslog-ng is a unix/linux daemon—it listens on a specified port for incoming data and forwards the information to a specified destination
35
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
How?
Rsyslog
An alternative daemon that works on the same principals as syslog-ng such as the ability to multiplex messages, use filters, pipe to programs, etc.
Not quite as “mature” as syslog-ng, but it does have a lot of community support and many of syslog-ng’s “pro” features for free.
36
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Now We’ve Introduced a New Problem!
How do we manage such a large amount of data?
How do we detect errors from a single device?
New processes need to be developed to detect device errors, degradation, change notifications, etc.
37
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Analysis
Database metrics
Collect metrics to show fault indicators and performance degradation
Top hosts/messages/severities, etc.
Messages per second/minute/day, etc.
38
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Analysis (Cont.)
Database metrics
Integrate syslog data with performance managers to trigger baseline thresholds
Example: Collecting the number of average messages per second a single device generates and alerting on variations outside the derived baseline
Integrate with Inventory systems!
If a device is talking to you, there’s a good chance it exists
Example: new devices being added to the network will have to wait until the next polling cycle by discovery systems, but if syslog is turned on in that device, your syslog manager will pick it up almost immediately.
39
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Architectures
40
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Architectures
Syslog-ng
Design Guidelines
Single Server Deployment
Multi-Server Deployment
Server Sizing
41
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog-ng
This design is based on the use of syslog-ng
Syslog-ng allows for collection and “forking” of syslog messages to many hosts
This allows log data to be collected and distributed in a much more robust fashion
42
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Traditional Logging Architecture
Traditional logging collection requires that many logging destinations be stored in each device
43
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog-ng Logging Architecture
Syslog-ng collectors allow for only a few logging hosts to be configured in your devices but then replicates these messages to end hosts
44
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Design Guidelines
Collection stations
Design your syslog architecture in a distributed, hierarchical fashion
Syslog collectors should lie as close to their networks as possible
Some filtering may be done at the collection level to weed out unnecessary log data
These collectors should forward filtered messages to a centralized server/database for further filtering and processing
45
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Design Guidelines (Cont.)
Syslog Event Manager
Deploy a performance management tool such as Cacti to establish a baseline of your logs
Assign people (or groups) to monitor daily Top X events/hosts/messages, etc. and remediate common problems such as fan failures, duplex mismatch, redundant power fails, etc.
Log Rotation and Retention
Establish a log retention and rotation policy
Include logs and log archives in a standard backup process
46
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Single Server Deployment
Can handle ~100 million messages per day
Dependent on server CPU(s), disk(s) and memory
One million logs = ~300-500MB of DB disk space a day, so size accordingly!
47
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Multi-Server Deployment
Assured delivery via TCP
Can handle large amounts of messages
Requires high end (master) server
Distributed collectors can be small servers used to filter and forward
UDP
UDP TCP
TCP
48
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Server Sizing
Please be aware that this is only a rough estimate
Every network is unique, there truly is no way to tell how many events/day you can expect until you start looking.
These are just a few devices that I’ve checked recently that may help you get an idea.
49
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Estimated EPD and Storage Needs
Device Type Device Count
Average Msgs/Week Per day Per Hour Per Min
Per second DB Size
External Indexes
Cisco Call Manager 1 151,200.00 21,600.00 900.00 15.00 0.25 21.09M 22.58M
IP SLA (2000 Probes P2P) 1 302,400.00 43,200.00 1,800.00 30.00 0.50 42.19M 45.15M
VMWare 1 3,780,000.00 540,000.00 22,500.00 375.00 6.25 527.34M 564.42M
Cisco ASA Firewall 1 6,652,800.00 950,400.00 39,600.00 660.00 11.00 928.13M 993.38M
Generic Router/Switch 1 211,680.00 30,240.00 1,260.00 21.00 0.35 29.53M 31.61M
Totals 5 10,886,400 1,555,200 64,800 1,080 18 1.48G 1.59G
50
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
High Level Starter Design
Sample Daily Activities
Identify Hardware related/Restart/Reboot events
Identify configuration changes (and forward to compliance manager)
Identify SNMP Authentication Failures
Identify large numbers of failed logins
Sample Daily Reports
Top 10 Hosts
Top 10 Mnemonics
Top 10 Severities
Top 10 Programs
These are low hanging fruits, take advantage of them!
Syslog Receiver
Syslog Reporter
Syslog Protocol
51
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog Protocol
High Level Advanced Design
Syslog Receiver
Syslog Reporter
Inventory Mgmt
NCCM (Compliance)
Device Synchronization
Filtered Change Notifications
MoM/Fault Mgmt
Event Correlation
Fault Notification
Performance Mgmt
Event Deduplication
DB Poller
Incident/Ticketing
Baseline Threshold
Alerts
52
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Always filter unwanted messages versus wanted
Allows for proper metric trending ‒ You may not care that a VPN session has terminated, but do you care that 1000
of them are terminating every minute?
Saves you the embarrassment of having to explain to upper management why you MISSED the message that caused the outage.
Filtering Events
My Organization Has a LOT of Events.
Can You Please Tell Me What to Look For?
53
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Design an architecture to build filters based on actionable events
Filtering Events
But… My Organization Has a LOT of Events.
54
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Actionable Events
When an Event Is Received, Two Immediate Questions Need to Be Asked:
Have we seen the event before?
Is an action required for the event?
55
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Advanced Design – Actionable Events
Syslog Protocol
Event Initial Analysis Start
Syslog (Event)
Database Query
Response
Yes
Actionable Event?
No
No Action: Leave Event in DB for Later Forensics and Retention
Yes
Immediate Action
Required?
Open Incident
Open Problem
Yes
No
Known Event?
Periodically Move Old Events to Offline (Disk) Storage
No
Mark as Known, Determine Action
56
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Analysis Tools
57
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Analysis Tools
Syslog-ng Store Box (SSB)
LogLogic
Splunk
LogZilla
Cisco Prime (LMS)
58
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog-ng Store Box
Turnkey solution for deploying syslog receivers using the pro version of syslog-ng
High-capacity log server with high-availability support
Able to collect logs from many different platforms
Made by the authors of syslog-ng
http://www.balabit.com/network-security/syslog-ng/log-server-appliance/
59
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
LogLogic http://loglogic.com
Commercial Solution
Capable of handling large amounts of data (70-100k mps)
Highly Scalable
60
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Splunk http://splunk.com
Searches and navigates IT data (not just Cisco devices) from applications, servers and network devices in real-time
Free version available (limited to 500MB of storage)
61
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Highly Scalable Indexing Service Provides Results in Seconds, Not Minutes.
Process Messages for up to 50,000 Hosts
Modular open source code allows for easy customization
VMWare Ready Turnkey Hardware Solutions
Available Optional Cloud-based service
offering allows users to send logs to our datacenter without having to set up their own server
http://www.logzilla.pro
@logzilla
62
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Cisco Prime LMS (RME Component) http://www.cisco.com/go/lms
Supports filtering of unwanted syslog messages
Can trigger user-defined scripts in response to specific syslog messages
Provides reports to quickly view syslog events by severity, device, or message.
Does not scale to large numbers, just handy for alerting if you already own it.
Note: Some message filters are enabled by default, including: Link Up/Down, ASA, DEBUG, and IOS Firewall Audit Trail messages. Large amounts of messages may lock up a server, so plan wisely!
63
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Case Study Cambridge University
64
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Goals
Improve confidence of log integrity in the event of a security compromise. How can we be certain that we’ll receive the event?
Provide mechanism to swiftly analyze logs on all systems. Utilize daily reports to find “top talkers”
Provide mechanism to have instant reports on log-on activity on all systems, and any other ad-hoc reporting required.
Information Provided by Andrew Baughan - Cambridge IT Manager
65
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Goals (Cont.)
Solution must be robust, and not introduce a significant overhead on target systems.
If possible, provide a mechanism to store all system logs for fault analysis and baseline statistical analysis for host data.
Provide a system capable of handling 100 Million events per day
Information Provided by Andrew Baughan - Cambridge IT Manager
66
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Solution Selection
The two technologies chosen for this solution were syslog-ng and LogZilla
Because syslog-ng writes the log data to both the client (localhost) and the LogZilla server simultaneously, everything reaches the LogZilla server and cannot be simply deleted or modified on the local host.
Information Provided by Andrew Baughan - Cambridge IT Manager
67
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Solution Selections (Cont.)
Since all logs are held in a central database, with a feature rich user interface, patterns of events across a large number of systems can be quickly identified and acted upon once the "signature" of a compromise is known.
It is easy to write scheduled jobs on the data stored within the MySQL database used as the back-end to LogZilla.
Open source makes it easy to customize to our needs if we ever need to.
Information Provided by Andrew Baughan - Cambridge IT Manager
68
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Solution Selections (Cont.)
The syslog-ng software does not add significant overhead to the target systems.
Higher priority events can be filtered out from the lower priority background events for improved reliability during high traffic periods on the log server.
An increase to the allocated storage allowed a longer retention period, with all events being filtered to the log host.
Information Provided by Andrew Baughan - Cambridge IT Manager
69
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Server Hardware
1 x Dell PowerEdge R410 ‒ 2 x Intel Xeon E5504 @ 2 GHz
‒ 4 x 4GB DIMMs
‒ 1 x SAS 6/iR, internal raid card for Hotplug drives
‒ 2 x 160GB SATA 7.2k 3.5 inch Hotplug HDD (mirrored system disk)
‒ 1 x PERC H800 RAID Adapter for External JBOD, 512MB, PCIe
‒ 1 x iDRAC6 Enterprise
‒ 1 x 16X DVD+/-RW ROM Drive SATA
‒ 1 x Redundant Power Supply (2 PSU) 500W
‒ 1 x Sliding Ready Rack Rails
Information Provided by Andrew Baughan - Cambridge IT Manager
70
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Storage Hardware
1 x PV MD1000 SAS Chassis ‒ 9 x 500GB NearLine SAS 6Gbps 7.2k 3.5" HD
‒ 1 x Rapid Rack Rails
‒ 1 x 2M External SAS Connector Cable
‒ 1 x PV MD1000 Bezel
Information Provided by Andrew Baughan - Cambridge IT Manager
71
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Hardware Notes
Two internal disks are in hardware mirror (R1), ext3 file-system.
The external disks are in a hardware RAID6 configuration with one hot spare, ext3 file-system.
The memory was a bit low for our needs, so it will most likely get doubled fairly soon, otherwise no substantial bottlenecks that a bit of tweaking cannot fix (I/O caching for example has been tweaked).
Information Provided by Andrew Baughan - Cambridge IT Manager
72
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Comments
Primary concern for large deployments should be DISK I/O and Memory.
When available use SSD disks (at least for write caching) ‒ Google “Cachecade Pro”
You can never have too much memory. Use a min of 32GB, but 96GB or even 128GB would be better. ‒ This determines how long it takes for SQL to do table sorting. The more memory
you have, the less it swaps to disk.
‒ Learn how to tweak MySQL, it can definitely scale, but people tend to not understand all the “knobs” to turn.
73
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Implementation
Syslog Collector (using syslog-ng)
Search and Reporting Tool (using LogZilla)
74
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Implementation
Syslog Collector (using syslog-ng)
Search and Reporting Tool (using LogZilla)
75
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Hardware
My laptop and VMWare
Realistically, start with a dual or quad-core box with 4-8G ram and work up from there unless you expect a large amount of logs (> 10mil/day)
Software installed:
Ubuntu v12.04 Server (64bit is a must!) – Why?
Basic server with a LAMP stack
Updated to latest patches
Syslog-ng
Syslog Collector Server Environment
LAMP = Linux, Apache, PHP, MySQL
76
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
root@log# aptitude install syslog-ng Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libevtlog0 The following packages will be REMOVED: rsyslog ubuntu-minimal The following NEW packages will be installed: libevtlog0 syslog-ng Do you want to continue [Y/n]? Y Setting up syslog-ng (2.0.9-4.1) ... * Starting system logging syslog-ng [ OK ]
Remove ubuntu minimal? Yes – Don’t Worry!
Use sudo if you’re not logged in as root “sudo aptitude…”
Installing Syslog-ng Syslog-ng is available in the apt repositories:
77
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Installing Syslog-ng
That was difficult, wasn’t it?
78
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Configuring Syslog-ng
The syslog-ng configuration file is typically stored in ‒ /etc/syslog-ng/syslog-ng.conf
There are five “steps” to building a syslog-ng configuration
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter and destination
79
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Configuring Syslog-ng
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
80
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Main Configuration Sample
Sets “global” options that apply to everything, for example:
use_dns(yes);
use_fqdn(yes);
keep_hostname(yes);
chain_hostnames(no);
81
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Sample Main Configuration
options {
log_fifo_size(16384);
time_reopen(10);
use_dns(yes);
dns_cache(yes);
use_fqdn(yes);
keep_hostname(yes);
};
82
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Configuring Syslog-ng
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
83
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Defines sources of information to receive messages from
source s_all { internal(); unix-stream("/dev/log"); udp(); };
“s_all” Can Be Named Whatever You Want, Just Be Sure to Use It Consistently
Syslog-ng Sources
Other UDP/TCP options are available, such as:
udp( ip(127.0.0.1) port(514) );
tcp( ip(0.0.0.0) port(5000) );
Only Allow UDP Messages from Localhost
Allow TCP Messages from All Hosts on Port 5000.
84
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Configuring Syslog-ng
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
85
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog-ng Filters
Defines a filter to be applied
Types of filters are: Facility—Match on a facility code (kern, local7, etc.)
Level—Match on a level code (error, notice, emerg, etc.)
Program—Match messages by using a regular expression against the program field
Host—Match messages by using a regular expression against the host field host("^cam(1|2|3|4|5)\.somehost\.tld$"); };
Match—Match a regular expression to the message itself
Filter—Call another filter rule and evaluate its value
Netmask—Determine if the sender’s IP is in the specified IP subnet
“Level” Is Synonymous with “Priority” or “Severity”
86
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Configuring Syslog-ng
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
87
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog-ng Destinations
Destinations Define Where to “Fork” Messages to, Such as: ‒ Files
‒ Programs
‒ Remote Hosts
‒ PIPE/FIFO
‒ Databases
88
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Sample Destination Definitions
File
destination df_syslog { file("/var/log/syslog"); };
destination df_disk { file("/var/log/HOSTS/$YEAR/$MONTH/$DAY/$HOST" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); };
Program
destination d_mydest { program("/var/scripts/db_insert.pl"
template("$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n")
template_escape(yes) );};
Remote Host
destination d_othersyslogbox { udp("170.19.86.100" port (514));};
You can use any name, just be sure to use the same name when applying it later on
89
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog-ng Destinations (Cont.)
PIPE (or FIFO) destination d_mysql {
pipe("/tmp/mysql.pipe" template(
"INSERT INTO logs (host, facility, level, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$LEVEL', '$S_YEAR-$S_MONTH-$S_DAY $S_HOUR:$S_MIN:$S_SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes));
};
Note: for direct DB inserts, it’s better to use the new SQL insert function built into syslog-ng 3.x, but ONLY when you don’t use a pre-processor like PERL (for event correlation, deduplication, etc.)
90
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog-ng Direct Database Inserts
The following example is for MySQL, but other types may be used such as sqllite, pgsql, mssql and oracle.
@version: 3.0 destination d_mysql { sql(type(mysql) host("localhost") username("syslog") password("syslog") database("syslog") table("logs") columns("host", "facility", "level", "datetime", "program", "msg") values("$HOST_FROM", "$FACILITY", "$LEVEL", "$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "$PROGRAM", "$MSG“) indexes("host", "facility", “level", "datetime", "program")); };
@ at the beginning, syslog-ng 3.02 and up complains if this is not present
91
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Using Direct Database Inserts
Plan on implementing this in a large scale environment?
If you have more than 1000 events per second, MySQL will bottleneck causing dropped events.
Instead, use a “program” call to Perl to process the incoming data and use bulk insert methods. This will handle up to ~35k mps (maybe more, but that’s what I’ve tested it to).
destination d_mydest { program("/var/www/myprog/scripts/myscript.pl”
92
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Configuring Syslog-ng
Main configuration options
Source definitions
Filter definitions
Destination definitions
Statement to apply the defined source, filter, and destination
93
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Once you have a defined source, filter (optional) and destination, you must “apply” them in a statement: log {
source(my_source);
filter(my_filter);
destination(my_dest);
};
Apply the Definitions
94
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
source s_all {
internal();
unix-stream("/dev/log");
file("/proc/kmsg" log_prefix("kernel: "));
udp();
tcp(port(2000));};
filter my_filter {
host("^router (1|2|3|4|5)\.cisco\.com$");};
Sources
Filter (Optional)
Final Sample Configuration
95
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
destination my_dest {
file("/var/log/logzilla/syslog.log"
template("$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n"));};
destination my_dest_hosts {
udp("1.1.1.1" port (514));
tcp("2.2.2.2" port (2001));};
log { source(my_source); filter(my_filter);
destination(my_dest);
destination(my_dest_hosts);};
2 Destinations
Apply
Final Sample Configuration
96
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Syslog-ng: Getting Help
Website
http://www.balabit.com/network-security/syslog-ng/
Documentation
http://www.balabit.com/support/documentation/
Mailing list
https://lists.balabit.hu/mailman/listinfo/syslog-ng
97
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Implementation
Syslog Collector (using syslog-ng)
Search and Reporting Tool (using LogZilla)
98
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Installing LogZilla
VMWare images are freely available at http://www.logzilla.pro/downloads
Subversion
Cloud
Hardware
Four Ways to Obtain
99
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Installing LogZilla (Cont.)
Visit http://www.logzilla.pro/downloads to grab the latest VM build
Use VMWare ESXi, Workstation, or VMWare Player and double click on the downloaded .ovf file to install the VM
VMWare Method
Read the console instructions after booting the VM
100
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Installing LogZilla (Cont.)
Make sure your OS has the correct prerequisites! ‒ https://www.assembla.com/spaces/LogZillaWiki/wiki/Prerequisites
Check out the code from the repository root# cd /var/www
root# svn co https://subversion.assembla.com/svn/logzilla.2/trunk/ logzilla
Optional but recommended: create an “answer” file (see next slide)
Install root# cd /var/www/logzilla/scripts && ./install.pl
Subversion Method
101
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Installing LogZilla (Cont.)
You can create an answer file at /var/www/logzilla/scripts/.lzrc so that the install won’t need to prompt you for answers during installation
Please visit https://www.assembla.com/spaces/LogZillaWiki/wiki/RC_File for the contents of this file as it may be updated from time to time.
Answer File
102
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Installing LogZilla
Login to http://<your_url> and check for data
After Install Completes
103
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Troubleshooting
Try these in order. 99% of the time, it will be one of these 3 things
1. Is LogZilla able to receive events?
2. Is syslog-ng sending events to LogZilla?
3. Is syslog-ng getting events from the network?
Three “Parts” of Troubleshooting
104
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Troubleshooting
Bypass syslog-ng by piping a message directly into LogZilla ‒ If this passes, then the problem is most likely not LogZilla
(note: all of this on a single line)
printf "`date \"+%Y-%m-%d %H:%M:%S\"`\ttest\t190\tCRON\tTest\n" | /var/www/logzilla/scripts/db_insert.pl –d5 -v
Is LogZilla Able to Receive Events?
105
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Troubleshooting
Check for errors, for example: ‒ Error 13 from MySQL (unable to write to disk due to permissions)
Starting /var/log/logzilla/db_insert.log for /var/www/logzilla/scripts/db_insert.pl at pid 23614 Using Database: syslog Debug level: 5 Table: logs Adminuser: PW: DB: syslog DB Host: DB Port: Deduplication Feature = 0 Logging results to /var/log/logzilla/db_insert.log Printing results to screen (STDOUT) Snare Enabled …snip… (bunch of debug output – note that it’s not all just errors)
Pipe Output
106
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Troubleshooting
Enable the DEBUG.log in syslog-ng.conf for LogZilla
By default, LogZilla installs the DEBUG.log output in syslog-ng.conf, just enable it and restart syslog-ng
Also useful for “replaying” events into LogZilla by piping them into db_insert.pl
Verify that syslog-ng is Sending Events
107
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Troubleshooting
#/etc/syslog-ng/syslog-ng.conf
log {
source(s_net);
destination(d_logzilla);
# Uncomment below and restart syslog-ng for debugging
destination(df_logzilla);
};
root@myserver# /etc/init.d/syslog-ng restart root@myserver# tail -f /var/log/logzilla/DEBUG.log
(make sure you see events coming into the file)
Verify that syslog-ng is Sending Events
108
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Troubleshooting
Use tcpdump to sniff the udp packets root@myserver# tcpdump udp port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
16:32:58.309204 IP 12.5.92.4.51220 > 12.5.92.2.syslog: SYSLOG daemon.error, length: 113
Is syslog-ng Getting Events from the Network?
109
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
LogZilla - Getting Help
Main Website
http://www.logzilla.pro
Forum
http://forum.logzilla.pro
Support
http://support.logzilla.pro (or email [email protected])
Ideas (submit ideas for future enhancements!)
http://ideas.logzilla.pro
Documentation and Installation Guide
http://docs.logzilla.pro
110
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
General Useful Links
Clayton’s NMS Wiki
http://nms.gdd.net
111
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Personal Goals (For LogZilla)
Track devices’ individual EPS rates using RRD
Implement Known Event/Actionable Event interface with northbound forwarding to incident management systems
Build an AI syslog deterministic tool
What I Want to Do in my Copious Amounts of Spare Time
112
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Personal Goals
Current method (DB entries) do not scale well when the server has 50k devices in it
The answer is to use RRD files to store metrics on individual hosts
The problem with traditional RRD usage is the charts it creates, way too ugly. So I am working on a backend to pull the data into a php array and make much prettier charts using jQuery
Track Devices’ Individual EPS Rates
113
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Personal Goals
Congruent with my whitepaper and the slides in this session.
Plan to add a right-click menu from the LogZilla search results that can mark as known and/or non-actionable.
Ability to trigger a northbound event to a problem or incident manager.
Known Event/Actionable Event Interface
Ability to tag and anonymously share with a community of LogZilla users to help you determine if an event *should* be actionable ‒ e.g.: 3000 other users marked this event as
“requires immediate action”
114
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Personal Goals
Something I’ve been working on for about 8 or 9 years now. The concept is to build an AI that can predetermine certain pattern types and possible
actionable events in order to mitigate network downtime and/or security breaches. ‒ The key is that new messages may never have been seen before. How do I know what I don’t
know? ‒ How can I determine that my network is being attacked AS it is happening versus waiting until
after it has happened? (Anyone want a free credit card?)
Think of it as a spam bot, but with multidimensional probabilities (and quite a bit more complex). ‒ Spam determination is a yes/no (spam/not spam) calculation. ‒ The syslog AI is a determination of a multitude of variables (fcaps, vendor, severity, technology,
etc.)
Prototype is now working and I plan to launch http://sysloggenome.com when time permits. The concept being somewhat of a game where I guess what log you give it and, if I am wrong, you help me “learn” the correct one(s).
Build a Syslog Artificial Intelligence Program
115
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Key Takeaways
People tend to be a bit overwhelmed by the amount of data they have to parse through ‒ Proper implementation of tools, metrics and processes will
solve that problem
Configure ALL devices consistently and properly; Make sure any new device deployment ALSO has the correct configuration
Properly designed, syslog can be one of the BEST sources for proactive network management available
116
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Other “NMS” Sessions If you like this one, you might like some of these:
Source: Placeholder for Notes is 18 points
117
Monday
Session Title
Mo
nd
ay
BRKNMS-2515 A “Difference-Maker” for Operating Cisco Collaboration Services – Proactive Problem Management
BRKNMS-2658 Securely Managing Your Networks with SNMPv3
BRKNMS-3043 Advanced Performance Measurement for Critical IP Traffic with Cisco IOS IP Service Level Agreements
BRKNMS-1037 Cisco Unified Communications Management Suite Methodology and Best Practices
BRKNMS-3021 Advanced Cisco IOS Device Instrumentation
CCSNMS-1903 Customer Presentation: Learn How Human Kinetics Uses a New Cisco Solution to Dramatically Improve Network and Security Operations
PSONMS-1038 Cisco Prime for Enterprise – Innovative Network Management
Tues
day
BRKNMS-1032 Network Management KPI's
BRKNMS-3132 Advanced NetFlow
BRKNMS-2035 Ten Cool LMS Tricks to Better Manage Your Network
BRKNMS-3021 Advanced Cisco IOS Device Instrumentation
BRKNMS-2640 Advanced DHCP and DNS Deployments
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Other “NMS” Sessions If you like this one, you might like some of these:
118
Monday
Session Title W
edn
esd
ay
BRKNMS-2512 Moving Cloud from hype to realty: Delivering Cloud using management architecture
BRKNMS-2031 SYSLOG Design, Methodology and Best Practices
PSONMS-1038 Cisco Prime for Enterprise – Innovative Network Management
BRKNMS-2514 Performance Assurance: An integrated Approach to Managing Service Delivery in Enterprise Networks
BRKNMS-2513 Best practices for cloud automation with Cisco Intelligent Automation software
BRKNMS-2202 Ethernet OAM – Technical Overview and Deployment Scenarios
BRKNMS-2030 Onboard Automation with Cisco IOS Embedded Event Manager
Th
urs
day
BRKNMS-1036 Simplify Network Management and Reduce Operational Costs with Cisco Prime
BRKNMS-2511 Cisco Videoscape Management Solution
BRKNMS-2031 SYSLOG Design, Methodology and Best Practices
BRKNMS-1035 The NOC at CiscoLive
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Complete Your Online Session Evaluation Give us your feedback and you
could win fabulous prizes. Winners announced daily.
Receive 20 Passport points for each session evaluation you complete.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.
119
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Final Thoughts
Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042
Come see demos of many key solutions and products in the main Cisco booth 2924
Visit www.ciscoLive365.com after the event for updated PDFs, on-demand session videos, networking, and more!
Follow Cisco Live! using social media: ‒ Facebook: https://www.facebook.com/ciscoliveus
‒ Twitter: https://twitter.com/#!/CiscoLive
‒ LinkedIn Group: http://linkd.in/CiscoLI
120
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Extra Credit Event Correlation using Simple Event Correlator (SEC)
122
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Installing Simple Event Correlator (SEC)
SEC is available in the apt repositories:
aptitude install sec Reading package lists... Done Building dependency tree Reading state information... Done Reading extended state information Initializing package states... Done Writing extended state information... Done The following NEW packages will be installed: sec
root@log#
Setting up sec (2.4.2-1) ... SEC disabled in /etc/default/sec
Use sudo If You’re Not Logged in as Root (“sudo aptitude…)”
123
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
#Defaults for sec RUN_DAEMON="no" DAEMON_ARGS="-conf=/etc/sec.conf -input=/var/log/syslog -pid=/var/run/sec.pid -detach -syslog=daemon"
Change to “yes”
Installing Simple Event Correlator (SEC)
Edit the SEC config to allow it to start
vi /etc/default/sec root@log#
124
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Installing Simple Event Correlator (SEC)
SEC uses a configuration file and takes input from a file or a named pipe
vi /etc/sec.conf # Example # Recognize a pattern and log it. # type=Single ptype=RegExp pattern=foo\s+(\S+) desc=$0 action=logonly
root@log#
First step is to create a config file:
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 125
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Rules
SEC includes several different types of rules that are useful in event correlation.
This rule is of type Single. # Example # Recognize a pattern and log it. # type=Single ptype=RegExp pattern=foo\s+(\S+) desc=$0 action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 126
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Rules
RegExp is the pattern type, select RegExp for (“Regular Expression”) matching or SubStr, for simpler string matching
# Example # Recognize a pattern and log it. # type=Single ptype=RegExp pattern=foo\s+(\S+) desc=$0 action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 127
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Rules
foo\s+(\S+) is the actual pattern - in this case a perl regular expression pattern
# Example # Recognize a pattern and log it. # type=Single ptype=RegExp pattern=foo\s+(\S+) desc=$0 action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 128
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Rules
desc is a variable definition for the pattern description (captured from the foo pattern using parentheses).
In this case a perl numbered variable, $0, is set to the entire matched pattern
# Example # Recognize a pattern and log it. # type=Single ptype=RegExp pattern=foo\s+(\S+) desc=$0 action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 129
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Rules
The action statement describes the action taken when the pattern is recognized.
In this case, the logonly action simply writes the pattern to the logfile if one is indicated on the command line, or to standard output if not
# Example # Recognize a pattern and log it. # type=Single ptype=RegExp pattern=foo\s+(\S+) desc=$0 action=logonly
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 130
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Rules
Save the file and execute the following command:
sec -conf=/etc/sec.conf -input=- root@log#
SEC (Simple Event Correlator) 2.4.2 Reading configuration from /etc/sec.conf 1 rules loaded from /etc/sec.conf
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 131
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Rules
This example will take input from directly from the terminal. Type the following lines of input:
foo
foo bar foo bar
baz
bar foo baz bar foo
Notice that SEC Responds by Replying Every Time a Pattern Is Matched
bar foo baz
Response
Response
No Response
No Response
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 132
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Actions
SEC has over a dozen different actions it can perform once it matches a pattern in the input stream
Some of the actions depend on their context
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 133
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Actions
write
Writes the specified text to the named filename.
E.g.: action=write - Hello from SEC. Matched text was $0
shellcmd
Causes SEC to execute a shell command.
E.g.: action=shellcmd mycommand.sh
spawn
Identical to the shellcmd action, but output (e.g. from an exit status in that shell script) from the command is fed back into SEC for pattern matching
E.g.: action=shellcmd mycommand.sh
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
134
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Actions
assign and eval
Both assign and eval deal with ``% <letter>'' variables. They are internal SEC variables that can be used in rules
E.g.: action=assign %f Joe bob likes,
action=eval %h ($t = “fishing and nascar")
action=write - %f %h at %t
Note: These are all parts of separate rules, but kept out for brevity
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 135
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC Actions
event
event allows the insertion of input to SEC from inside SEC itself
event is feedback mechanism - one controlled by SEC's own rules.
The time parameter is the number of seconds to wait before inserting the event text into SEC's input stream.
E.g.: action=event 5 baz is now matched. ; write - foo matched at %t. baz event in 5 seconds...
Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html 136
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Sample Rules—Suppress
# System configuration events # suppressed because we don't care about it type=suppress ptype=substr pattern=%SYS-5-CONFIG_I: desc=device configuration
What’s Wrong with This Rule? (Should We Suppress Configuration Changes?)
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec 137
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Sample Rules—Time Based
# Looks for a reload followed by a restart event # type=pairWithWindow ptype=regexp pattern=\d+:\d+:\d+.*?(\S+)\s+\d+:.*?%SYS-5-RELOAD: desc=(CRITICAL) $1 RELOAD_PROBLEM action=pipe '%s' mail -s 'cisco event' [email protected] ptype2=regexp pattern2=($1).*?%SYS-5-RESTART: desc2=(NOTICE) $1 RELOAD_OK action2=pipe '%s' mail -s 'cisco event' [email protected] window=300
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec 138
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Sample Rules—Escalation
# This rule escalates to CRITICAL if there are more than 5 # neighbor changes in 5 seconds # type=SingleWithThreshold ptype=substr pattern=(MINOR) OSPF adjacency change desc=(CRITICAL) More than 5 OSPF neighbor changes in 5 seconds action=pipe '%s' mail -s 'cisco event' [email protected] thresh=5 window=5
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec 139
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Sample Rules—Link Up/Down Pairs
# This rule deals with link down events # type=PairWithWindow ptype=RegExp pattern=\d+:\d+:\d+.*?(\S+)\s+\d+:.*?%LINK-3-UPDOWN: Interface (\S+), changed state to down desc=(MINOR) $1 INTERFACE $2 DOWN and not up in one minute action=pipe '%s' mail -s 'cisco event' [email protected] ptype2=RegExp pattern2=($1)\s+\d+:.*?%LINK-3-UPDOWN: Interface ($2), changed state to up desc2=(WARNING) %1 INTERFACE %2 BOUNCE action2=event %s window=60
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec 140
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Sample Rules—Link Up/Down Pairs
# when the first bounce event is seen, create a reporting trigger # type=Single continue=TakeNext ptype=regexp pattern=(\S+) INTERFACE \S+ BOUNCE context=!INTERFACE_BOUNCE_WAIT_$1 desc=interface bounce summary event for router $1 action=create INTERFACE_BOUNCE_WAIT_$1 10 (report INTERFACE_BOUNCE_$1 mail -s 'cisco events' [email protected]; delete INTERFACE_BOUNCE_$1)
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec 141
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
Sample Rules—Link Up/Down Pairs
# accumulate all interface bounce events into a context # type=Single ptype=regexp pattern=(\S+) INTERFACE (\S+) BOUNCE desc=interface bounce for router $1 interface $2 detected action=add INTERFACE_BOUNCE_$1 %t: %s
Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec 142
© 2012 Cisco and/or its affiliates. All rights reserved. BRKNMS-2031 Cisco Public
SEC—Getting Help
SEC Main Page
http://simple-evcorr.sourceforge.net/
Email list
http://simple-evcorr.sourceforge.net/#mailinglist
Good install and explanation guide
http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
143