LACNOG - Logging in the Post-IPv4 World

Carlos Martinez Cagnazzo

LACNIC

carlos @ lacnic.net

@carlosm3011

Logging for Incident Response in the Post-IPv4 World

Agenda

• The Post-IPv4 Internet– No IPv4, CGNs, some IPv6

• Logging for incident response• Logging and incident response in the post-IPv4 Internet

The Post-IPv4 Internet

• The Internet is at a crossroads. IPv4 exhaustion means that there will not be enough IPv4 addresses for every one, much less for every device

• To an extent, this is already happening, but from now it will the norm

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 20110

20

40

60

80

100

120

The Current, Almost End-to-End, Internet

• Once upon a time there was something called the ‘End to End Principle’– … describing how packets should travel from origin to

destination untouched by the evil middle boxes

• The current Internet is _almost_, but not quite, end-to-end – Proxies, home routers, firewalls, traffic shapers, all of them do

something to packets– But packets travel mostly unharmed

The Current End-to-End Internet

• Well, almost end to end**

• Packets remain (mostly) unchanged along their network path

• A given source IP can be a marker of an individual, a household or an employee of a certain company

D_Addr | O_Addr | Payload

D_Addr | O_Addr | Payload

What happens when there is no IPv4 for every device ?

• The post-IPv4 Internet:

• IPv4 will be provided, in many places, by employing CGNs, or Carrier-Grade NAT boxes

Single public IP address

Web server sees thousands of users coming

from the *same* IP

The CGN-ized Internet

• The CGN Internet hides many users behind a small set of IP addresses

• Our previous assumptions about what a source IP address means are no longer valid– Can represent thousands of users, of different households and

different companies

• Many abuse mitigation measures need to be re-examined– Be careful of blankly filtering out a single /24, that could now

mean 10.000 users

Current practice for Incident Response

• Think for a minute about your usual IR workflow– Phishing, Spam, DDoSing, you name it

• When your incident involves network traffic, you try to find the following information:– Source IP addresses– Destination IP addresses and destination ports– Maybe a packet dump, if available– All of this decorated with nice timing information, preferable

with a common time zone

• You then look the sources in WHOIS or in your friendly CSIRT contact list and send the appropriate notifies

The Post-IPv4 Incident Response Workflow

• Well, source IPv4 address may not be enough of an identifier anymore– The source network will not be able to identify the actual

offender(s) just based on the source IPv4 address

• ISPs will need source port data to actually track any abusers

• Law enforcement also needs to realize what this means– Judges now need to look at an additional number before jailing

a person

Jeez, what do we do now ?

• First of all, accept that now your life as an incident response or site administrator will be harder– Hopefully for a short time, until the world gets its IPv6 act

together

• Additional requirements for post-IPv4 logging– Logging of source ports– Using the highest possible timing resolution– Time sync on distributed logging platforms becomes critical

Example configuration, Source Port Logging in Apache

• [Ref: http://draft.scyphus.co.jp/articles/20110815.html]– Default logging in Apache only provides basic client data– Apache uses a printf()-like format for including additional log

fields in custom log files

## The following directives define some format nicknames for use # with a CustomLog directive (see below).#

LogFormat "[%h]:%{remote}p %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "[%h]:%{remote}p %l %u %t \"%r\" %>s %b" common

Example configuration, Source Port Logging in Apache

• [Ref: http://draft.scyphus.co.jp/articles/20110815.html]– Default logging in Apache only provides basic client data– Apache uses a printf()-like format for including additional log

fields in custom log files<VirtualHost [2001:13c7:7001:4000::10]:80> ServerAdmin [email protected] DocumentRoot /var/www/html/ ServerName w6.labs.lacnic.net

LogFormat "[%h]:%{remote}p %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined CustomLog logs/w6.labs.lacnic.net-access_log "[%h]:%{remote}p %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""

# LogFormat "[%h]:%{remote}p %l %u %t \"%r\" %>s %b" common ErrorLog logs/w6.labs.lacnic.net-error_log

Example configuration, Source Port Logging in Apache

• Must enable “mod_log_config” if not already enabled

[2001:13c7:7003:89:fcda:8bea:3e8a:cedd]:57366 - - [31/Oct/2013:15:01:33 -0200] "GET /site/modules/openid/openid.js?A HTTP/1.1" 304 - "http://w6.labs.lacnic.net/site/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36"[2001:13c7:7003:89:fcda:8bea:3e8a:cedd]:57365 - - [31/Oct/2013:15:01:33 -0200] "GET /site/themes/newlabs/print.css?A HTTP/1.1" 304 - "http://w6.labs.lacnic.net/site/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36”

Example configuration, Exim4 logging

• Sample configuration:– [ http://www.exim.org/exim-html-current/doc/html/spec_html

/ch-log_files.html ]

# uncomment this for debugging# MAIN_LOG_SELECTOR == MAIN_LOG_SELECTOR +all -subject -arguments

.ifdef MAIN_LOG_SELECTORlog_selector = MAIN_LOG_SELECTOR +incoming_port.endif

2013-10-28 17:22:17 1VasOD-0005hG-KT <= [email protected] H=localhost (coco) [127.0.0.1]:47264 P=esmtp S=4742013-10-28 17:22:17 1VasOD-0005hG-KT => marcelo <marcelo@localhost> R=local_user T=maildir_home2013-10-28 17:22:17 1VasOD-0005hG-KT Completed

Distributed logging

• Did I say ‘time sync’ before ? • Use NTP Luke, You Must.

– It was invented for a reason

• Look into fast data stores and mining tools– Splunk– ElasticSearch– NoSQL databases (Redis, MongoDB)

Key Takeaways

• Yes, our sys/netadmin life will be harder, at least until IPv6 is widely deployed– Let’s embrace it with a smile

• Do not assume that a source attack IPv4 address uniquely identifies an attacker anymore– Or a victim, in some cases, like phishing sites

• Start logging source ports now. If you are a CSIRT, do not forget to reach out to your constituency and let them know this

• Send source ports when reporting incidents. Ask for source ports when receiving incident reports

Key Takeaways (ii)

• Log with the highest timing resolution your equipment allows

• And repeat with me…– I will time sync my systems– I will time sync my systems– I will time sync my systems

Thank you very much! Questions?

@carlosm3011