Grokking Grok: Monitorama PDX 2015

Grokking GrokA Magic Show of

Regular Expressions

@ferggo (Twitter)GregMefford (GitHub)

Key Take-Away:

Check Out Logstash

Logstash Does Lots of ThingsOutputsInputs Filters

Codecs

Grok

Grok is Magic

Grok is Magic(thanks, @jordansissel )

Grok

https://flic.kr/p/8zAUi6

TransmutationTurning Lead into Gold

TransmutationSpinning Straw into Gold

https://flic.kr/p/j4Jg1u

In TheoryFirewalls are simple

https://xkcd.com/730/http://www.startrek.com/database_article/scott

there’s Variety (T_T)In The Enterprise™

“Syslog”

“Syslog”

“Syslog”

“Syslog”

https://www.etsy.com/listing/154952800/unicorn-poo-adjustable-ring-polymer-clay

SparklyUnicorn


SparklyUnicornPoo


SparklyUnicornPoo

sometimesadjustable?

Cisco ASA

<134>Sep 02 2014 11:50:10: %ASA-6-302013: Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80)

<134>Sep 02 2014 11:50:10: %ASA-6-302014: Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs

Cisco ASA

<134>Sep 02 2014 11:50:10: %ASA-6-302013: Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80)

<134>Sep 02 2014 11:50:10: %ASA-6-302014: Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs

grok { match => ["message", "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:cisco_msg}" ]}

<134>Sep 02 2014 11:50:10: %ASA-6-302013: […]{ "@timestamp" => "2014-09-02T15:50:10.000Z", "cisco_tag" => "ASA-6-302013", "cisco_msg" => "[…]"}

Cisco ASA

cisco_msg:

Built inbound TCP connection 123456789 for inside:10.0.1.1/1234 (10.0.1.1/1234) to outside:10.0.2.2/80 (10.0.2.2/80)

Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs

Cisco ASA

cisco_msg:


Teardown TCP connection 123456789 for inside:10.0.1.1/1234 to outside:10.0.2.2/80 duration 0:00:00 bytes 420 TCP FINs

http://www.cisco.com/c/en/us/td/docs/security/asa/syslog-guide/syslogs.html



730Pages!?

http://geektyrant.com/news/2013/1/10/the-ultimate-movie-training-montage.html

grok {match => [ "cisco_msg", "%{CISCOFW106014}", "cisco_msg", "%{CISCOFW106015}", "cisco_msg", "%{CISCOFW106021}", "cisco_msg", "%{CISCOFW106023}", "cisco_msg", "%{CISCOFW110002}", # ... "cisco_msg", "%{CISCOFW302010}", "cisco_msg", "%{CISCOFW302013_302014_302015_302016}", "cisco_msg", "%{CISCOFW302020_302021}", "cisco_msg", "%{CISCOFW305011}", "cisco_msg", "%{CISCOFW313001_313004_313008}"]}http://www.gregmefford.com/blog/2014/09/24/analyzing-cisco-asa-firewall-logs-with-logstash/

302013: Built {in|out}bound TCP connection <ID> …

302014: Teardown TCP connection <ID> …

302015: Built {in|out}bound UDP connection <ID> …

302016: Teardown UDP connection <ID> …

CISCOFW302013_302014_302015_302016


{ "action" => "Built", "direction" => "inbound", "protocol" => "TCP", "src_interface" => "inside", "src_ip" => "10.0.1.1", "src_port" => "1234", "dst_interface" => "outside", "dst_ip" => "10.0.2.2", "dst_port" => "80"}

Ta-da!

Bonus!

Transfooooorm!(ation)

http://www.deviantart.com/art/Hungry-Luma-210132138

Check Point FW-1

<166>Firewall: 11May2015 14:48:00 drop 1.2.3.4 >bond1.5 rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;

Check Point FW-1

<166>Firewall: 11May2015 14:48:00 drop 1.2.3.4 >bond1.5 rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;

Transfooooorm! (ation)grok {match => ["message", "^<%{POSINT:syslog_pri}>%{WORD}: + (?<cp_time>%{MONTHDAY}[a-zA-Z]{3}%{YEAR} %{TIME}) + %{WORD:action} +%{IP} +%{DATA:interface} + %{GREEDYDATA:cp_msg}" ]}

<166>Firewall: 11May2015 14:48:00 drop 1.2.3.4 >bond1.5 …

{ "@timestamp" => "2015-05-11T18:48:00.000Z", "action" => "drop", "interface" => ">bond1.5" "cp_msg" => "rule: 150; rule_uid: […]"}

Transfooooorm! (ation)rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;

kv { source => "cp_msg" value_split => ":" field_split => ";" trimkey => " " trim => " "}

Transfooooorm! (ation)rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;{ "rule": "150", "rule_uid": "{DEADBEEF-4444-5555-6666-DECAFBAD1234}", "rule_name": "CleanUp", "src": "5.6.7.8", "dst": "9.10.11.12", "proto": "udp", "product": "VPN-1&FireWall-1", "service": "domain-udp" "s_port": "67890", "product_family": "Network",}

Transfooooorm!(ation)

http://www.deviantart.com/art/Hungry-Luma-210132138

Transfooooorm! (ation)mutate { rename => [ "dst", "dst_ip", "src", "src_ip", "s_port", "src_port", "proto", "protocol", "service", "dst_port", "interface", "src_interface" ]}

Transfooooorm! (ation)rule: 150; rule_uid: {DEADBEEF-4444-5555-6666-DECAFBAD1234}; rule_name: Clean Up; src: 5.6.7.8; dst: 9.10.11.12; proto: udp; product: VPN-1 & FireWall-1; service: domain-udp; s_port: 67890; product_family: Network;{ "rule": "150", "rule_uid": "{DEADBEEF-4444-5555-6666-DECAFBAD1234}", "rule_name": "CleanUp", "src_ip": "5.6.7.8", "dst_ip": "9.10.11.12", "protocol": "udp", "product": "VPN-1&FireWall-1", "dst_port": "domain-udp" "src_port": "67890", "product_family": "Network",}

Ta-da!

Thanks!@ferggo (Twitter)GregMefford (GitHub)

{ "hostname": "FileServer.example.com", "EventType": "AUDIT_SUCCESS", "Severity": "INFO", "EventID": 5145, "SourceName": "Microsoft-Windows-Security-Auditing", "Channel": "Security", "Category": "Detailed File Share", "SubjectUserName": "somebody", "SubjectDomainName": "DOMAIN1", "ObjectType": "File", "IpAddress": "67.89.12.34", "ShareName": "\\\\*\\MyFiles", "ShareLocalPath": "\\??\\E:\\MyFiles", "RelativeTargetName": "Documents\\Somebody", "AccessMask": "0x80", # …}

grok { match => [ "ShareName", "\\\\\*\\%{GREEDYDATA:Share}" ]}mutate { add_field => { "ShareFullPath" => "\\%{hostname}\%{Share}\%{RelativeTargetName}" }}

grok { match => [ "ShareLocalPath", "\\\?\?\\%{DATA:LocalPath}(\\)?$" ]}mutate { add_field => { "LocalFullPath" => "%{LocalPath}\%{RelativeTargetName}" }}

{ "hostname": "FileServer.example.com", "ShareName": "\\\\*\\MyFiles", "ShareLocalPath": "\\??\\E:\\MyFiles", "RelativeTargetName": "Documents\\Somebody", "ShareFullPath": "\\\\FileServer.example.com\\MyFiles\\Documents\\Somebody", "LocalFullPath": "E:\\MyFiles\\Documents\\Somebody", "AccessMask": "0x80", # …}

Ta-da!

ruby { code => "mask = event['AccessMask'].to_i(16)field_names = { 0 => 'READ_DATA_LIST_DIRECTORY', 1 => 'WRITE_DATA_ADD_FILE', 2 => 'APPEND_DATA_ADD_SUBDIRECTORY', 3 => 'READ_EA', 4 => 'WRITE_EA', 5 => 'EXECUTE_TRAVERSE', 6 => 'DELETE_CHILD', 7 => 'READ_ATTRIBUTES', 8 => 'WRITE_ATTRIBUTES', 16 => 'DELETE', 17 => 'READ_CONTROL', 18 => 'WRITE_DAC', 19 => 'WRITE_OWNER', 20 => 'SYNCHRONIZE'}event['AccessMaskFields'] = Hash.newfield_names.each do |index, name| event['AccessMaskFields'][name] = mask[index] unless mask[index].nil?end"}

Bonus!

{ "AccessMask": "0x80", "AccessMaskFields": { "READ_DATA_LIST_DIRECTORY": 0, "WRITE_DATA_ADD_FILE": 0, "APPEND_DATA_ADD_SUBDIRECTORY": 0, "READ_EA": 0, "WRITE_EA": 0, "EXECUTE_TRAVERSE": 0, "DELETE_CHILD": 0, "READ_ATTRIBUTES": 1, <==== "WRITE_ATTRIBUTES": 0, "DELETE": 0, "READ_CONTROL": 0, "WRITE_DAC": 0, "WRITE_OWNER": 0, "SYNCHRONIZE": 0 }}

Bonus!

CISCO_TAGGED_SYSLOG:^<%{POSINT:syslog_pri}>%{CISCOTIMESTAMP:timestamp}( %{SYSLOGHOST:sysloghost})?:%%{CISCOTAG:ciscotag}:

<134>Sep 02 2014 11:50:10: %ASA-6-302014: [...]

Behind the Scenes: Grok is Magic

syslog_pri timestamp ciscotag

CISCO_TAGGED_SYSLOG:^<(?<syslog_pri>\b(?:[1-9][0-9]*)\b)>(?<timestamp> %{MONTH} + %{MONTHDAY}(?: %{YEAR})? %{TIME})((?<sysloghost> (?:%{HOSTNAME}|%{IP})))?:%(?<ciscotag> [A-Z0-9]+-%{INT}-(?:[A-Z0-9_]+)):


CISCO_TAGGED_SYSLOG:^<(?<syslog_pri>\b(?:[1-9][0-9]*)\b)>(?<timestamp>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b + (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?: (?>\d\d){1,2})? (?!<[0-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]) (?::(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))(?![0-9]))((?<sysloghost>(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)|(?:%{IPV6}|%{IPV4}))))?:%(?<ciscotag>[A-Z0-9]+-(?:[+-]?(?:[0-9]+))-(?:[A-Z0-9_]+)):


IPV6:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?


CISCOFW302013_302014_302015_302016:

%{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for

%{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port}( $%{IP:src_mapped_ip}/%{INT:src_mapped_port}$)?($%{DATA:src_fwuser}$)? to

%{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}( $%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}$)?($%{DATA:dst_fwuser}$)?

( duration %{TIME:duration} bytes %{INT:bytes})?(?: %{CISCO_REASON:reason})?( $%{DATA:user}$)?


302013: Built {in|out}bound TCP connection <ID> …

302014: Teardown TCP connection <ID> …

302015: Built {in|out}bound UDP connection <ID> …

302016: Teardown UDP connection <ID> …

CISCOFW302013_302014_302015_302016


Built inbound TCP connection 123456789 for

inside:10.0.1.1/1234 (10.0.1.1/1234)

to outside:10.0.2.2/80 (10.0.2.2/80)

(302013)


action direction protocol

src_interface src_ip & src_port src_mapped_ip & _port

dst_interface dst_ip & _port dst_mapped_ip & _port

connection_id

Teardown TCP connection 123456789 for

inside:10.0.1.1/1234 to outside:10.0.2.2/80

duration 0:00:00 bytes 420 TCP FINs

(302014)


action protocol

src_interface dst_interface dst_ip & _port

duration

src_ip & _port

bytes reason

connection_id

Grokking Grok: Monitorama PDX 2015

Technology