Stupid iptables tricks

Stupid iptables Tricks

Jim MacLeod@shewfig

Bsides Las Vegas 7/28/2010

Who I am

• Network hacker, not OS hacker• Favorite language is bash• If a tool exists to do what I want, why re-

invent the wheel?– Sorry, Frank^2

What I get paid to do

• Retrofit security onto existing products• Try to keep 80% of you from breaking into my

product in less than 1 week• Try to make my job go away by convincing my

team to write secure code

Overview

• Defining the box:– Refresher on what iptables is

• Measuring the box:– How I learned what’s in this presentation

• Outside the box:– Automating & abstracting policy creation– Creating static tables with dynamic elements– Pushing full packets to userspace

Iptables

• Umbrella term– Linux kernel-level stateful packet filter– Part of Netfilter – “iptables” is user-facing app– Integrates with PBR & other fun toys

• Linux characteristics– Rich set of options– Moon-man language– User-friendly, but picky whom it considers a friend

Why we ignore iptables

• “It’s the Linux firewall, right? We already have a border firewall.”

• “It’s old technology” – L4, not L7• [Insert Linux vs. Proprietary argument]• “Linux is a server”

Why we shouldn’t ignore iptables

• Stateful packet filter – most common firewall (!)• Dynamic (non-destructive) policy updates• Rich configuration options• Multiple complex inter-connected policies

– Makes it easy(er) to optimize– Nested policies

• Shorter logic path in common case• Very complex policies for special cases

• Extensible state tables• External hooks

– Pass the entire packet off to a userspace app

Limitations

• Rules exclusive to L2/L3/L4– Iptables / Arptables– Vulnerable to IP spoofing

• “Everything interesting” is at L7– Only L7 capability is pattern match• Anyone remember 1st gen IDS?• Subject to false positives if not pre-qualified

Overcoming limitations

• The “Wrong Way”– TOFU auth (Trust On First Use)• Not “meaty” enough• IP spoofing & session stealing can fry TOFU

– Optional crypto• E.g. SSL cleartext injection / renegotiation

– Do it all in L7• What could possibly go wrong?• “Business Logic” != networking

Overcoming limitations

• The “Cheap Way”– State machine: is the connection in a state which

implies that it has been authenticated?• “Stateful” packet filter – ESTABLISHED state• (TCP session established)

– How is this different than TOFU?• If you don’t have a GOOD answer, it’s not good enough

Overcoming limitations

• The “Right Way”– Strong auth• Non-persistent: require for every new connection

– Mandatory crypto• Can’t hijack / inject on properly implemented crypto• Crypto is an implicit auth of the established session• However: crypto does not imply initial auth –

anonymous crypto is possible

• IPtables can’t do this for you

Relevance?

• What does this have to do with iptables?– Reminder that no Linux app or service exists in

isolation– Useful as 1 of multiple layers

• IPtables can be coupled tightly with additional security tools & methods

Advantages of using IPtables

• Add a bump in the wire before ANY app.– If a problem can be “solved” in iptables, the app

will inherit the solution• IPtables on-box has insight into the app– Process owner, SElinux context, etc.

• App can have insight & control of IPtables

Disadvantages of Solving Problems in IPtables

• There is more than one way to do it• Many ways are “wrong”– Hopefully this talk will show some “right” ways

Origin of this talk: my job

• Clustered, appliance-based web app• Problem: how to add a new node?– Fake new node would be “bad”™– Pwn node before added -> pwn cluster

Problem 1: Stealth new node

• Must allow “real” inbound connections• Should not reveal open ports to just anyone• Not enough time to implement auth

Solution 1: Port knocking

• Create policy to allow only inbound traffic which behaves like a real cluster– Specific order:– ICMP ping– TCP echo– Back-end port– Database port– Profit!

• If not in order, later ports won’t open

Implementation 1 (Step 1)

• iptables -A INPUT -p icmp --icmp-type 8 -m recent --name seenping --set-j ACCEPT

Implemenation 1 (Step 2)

• iptables -N ADD_ECHO• iptables -A INPUT -p tcp --dport echo

-m recent --name seenping --rcheck-j ADD_ECHO

• iptables -A ADD_ECHO-m recent --name seenecho --set-j ACCEPT

• (implicit return to INPUT)– Can also explicitly specify with -j RETURN

Implementation 1 (Step 3)

• iptables -N ADD_BACKEND• iptables -A INPUT -p tcp --dport 8000

-m recent --name seenecho --rcheck--seconds 10-j ADD_BACKEND

• iptables -A ADD_BACKEND-m recent --name seenbackend --set-j ACCEPT

Implementation 1 (Step 4)

• iptables -A INPUT -p tcp --dport 3306-m recent --name seenbackend --rcheck-j ACCEPT

Problem 2: Database sync

• Database sync uses 2 opposite (A->B, B->A) TCP connections

• Connection from old to new: see Problem 1– Empty database, no interesting data

• Connection from new to old: – Family jewels exposed– Must not open port to just anyone• (yes, we’re using auth…)

Solution 2: Quid Pro Quo

• Aka “Reverse Port Knocking”• Old node connects to database outbound• IPtables tracks outbound connection– Explicitly allows inbound connection

Implementation 2

• On existing node:• iptables -A OUTPUT -p tcp --dport 3306

-m recent --name dbout --rdest-j ACCEPT

• iptables -A INPUT -p tcp --dport 3306-m recent --name dbout --rcheck-j ACCEPT

Problem 3: Log Clog

• Firewall exceptions “should” be logged• Some protocols are really chatty– Windows broadcasts– Lots of background noise

• IPtables doesn’t have rich logging solution– Syslog as kernel message– Hard to filter kernel facility– “Previous message repeated 5 million times”

Solution 3: Seen-It Suppression

• Add source address to specific recency table• Log first time, ignore for next ‘n’ minutes• Suppress the log clog– Doesn’t log # of ignored packets– Sorry, Marcus

Implementation 3

• iptables -A INPUT -p udp --dport 138-m recent --name udp-138 --rcheck --seconds 300-j DROP

• iptables -A INPUT -p udp --dport 138-m recent --name udp-138 --set

• iptables -A INPUT -j LOG --log-prefix “INPUT-drop”

• iptables -A INPUT -j DROP

Recap (So far)

• Recency tables for:– Port knocking– Reverse port knocking– Log suppression

• All within IPtables

Version 1.3: Data-driven Firewall

• Plug-In architecture• Each app might open different ports in/out• Endpoints might be dynamic– E.g. defined by user in in GUI

Version 1.3 Solution

• XML firewall policy files– Define inbound/outbound per-plug-in– Definition can also be database entry

• Watchdog to scan for changes– Changes in files– Poll database for changes– Versioning of policies

• Policy updates can be dynamic: insert/remove policies

Version 1.3 Implementation

• [1 KLOC of perl] - watchdog• [1 KLOC of bash] - policy-checker• [1 KLOC of perl] - XML-policy-parser• [1 KLOC of bash] - iptables policy generator

Revisiting Problem 1: Add Auth

• Port knocking is an “open secret”– Watch the wire & learn the combo

• Create auth “shim” layer• Connect to 1 port for auth• Service adds permission for client to connect to

“real” port – use “-m recent” in iptables policy– echo IP > /proc/net/xt_recent/[name]

• Note: must run as root

Persistent Policies

• Confession: my data-driven firewall doesn’t actually intelligently modify the policy

• Flush tables, bulk push• Flushes ESTABLISHED and RELATED tables too• Rules will re-add existing connections– only if packets come from client to server

• What about packets from server to client?– “Destination” and “Source” are a matter of

perspective and initial state

Implementing Persistent Policies

• Policy:– iptables -p tcp --dport 8000 -j ACCEPT

• Persistent reverse policy:– iptables -s [local IP] -p tcp --sport 8000

--tcp-flags SYN NONE -j ACCEPT

• Allow packet in the “wrong” direction if it’s not the start of a TCP session

• Dangerous territory: can reduce stealth– “nmap -sA” will waltz right through if ‘-s’ omitted

Policy Versioning

• Quick hack: use comments• iptables -A PUSHDATE -m comment

--comment `date ‘+%s’`• Embed the date the policy is pushed as a

commentChain PUSHDATE (0 references)num target prot opt source destination 1 all -- 0.0.0.0/0 0.0.0.0/0 /* 1280187751 */

Expanding the box

aka “what I haven’t actually done yet”

• Policy logic• Pattern matching less blindly• Q&D blocking IDS• URL filtering per user

Policy Logic

• Each chain can call other chains– e.g. -j ADD_PING

• Chains can return to their caller– e.g. -j RETURN

• Doesn’t this sound like function calls?– recency tables =~ variables– So is FWMARK

Canonical Firewall Policy

• Firewall access rules– admins can ssh in

• Firewall stealth rule– No one else can access the firewall

• Series of access/stealth rules– Servers, subnets, “3-Tier”, etc.

• Outbound access• Log/Drop all

Optimized Firewall Policy

• Rule of thumb: more policies = more latency• All packets to firewall go to separate chain– -j FIREWALL– Implement access / stealth / log / drop / smite

• Reduce the # of policies for each packet: reduce the overall latency

More “Policy Programming”

• FWMARK– Add “mark” to packet in kernel– local scope: only in the box, not on the wire

• iptables -j MARK --set-mark [unsigned int]• iptables -m mark --mark [unsigned int]• FWMARK is also recognized by:– ip rule / policy-based routing– ipvs (L3 load balancer)

FWMARK example

• Bind multiple protocols / ports together in a load balancer policy

• Typical web site: – front page is http:80– login is https:443– content is http:80 again– spawn whiteboard http:9012

• All connections from same client will go to same back end server

FWMARK vs. recency

• Recency tables store IP & timestamps– useful for applying rules to an address that has

been seen before in a different connection– no port numbers or protocols

• FWMARK is per-packet– no concept of connection

• Potentially interesting in combination– mark all packets from a recent IP– Apply PBR

Pattern Matching

• Arbitrary pattern matches are too coarse• Pattern with offset:– Helps if pattern GUARANTEED to be at that location– TCP options, encapsulation, etc. can move offset

• Use chains to pre-qualify data– Port 80? That’s the HTTP_PATTERNS chain…

• DISCLAIMER:– I have no actual knowledge of pattern matching in

iptables

Q&D blocking IDS

• http://www.snowman.net/projects/ipt_recent/• Example 1: block annoyance for 60 seconds– iptables -A FORWARD -m recent --name noplacelike

--rcheck --seconds 60 -j DROP

– iptables -A FORWARD -i eth0 -d 127.0.0.0/8-m recent --name noplacelike --set -j DROP

• Example 2: rolling block– iptables -A FORWARD -m recent --name noplacelike

--update --seconds 60 -j DROP

– iptables -A FORWARD -i eth0 -d 127.0.0.0/8-m recent --name noplacelike --set -j DROP

URL Filtering

• Transparent Proxy is your friend– http://www.ex-parrot.com/pete/upside-down-ternet.html

• Transparent Proxy uses iptables– iptables -t nat -A PREROUTING

-s 192.168.0.0/255.255.255.0-p tcp --dport 80 -j REDIRECT --to-port 8080

• Use recency tables to redirect some IPs– iptables -t nat -A PREROUTING

-m recent --name kittenwar -p tcp --dport 80 -j REDIRECT --to-port 8081

• Use with user auth, “guest” access, or annoyance triggers

Captive Portal

• “Yes I won’t hack you” page on WiFi• iptables -m recent --name allowed -p tcp --

dport 80 -j ACCEPT• iptables -t nat -A PREROUTING -p tcp --dport

80 -j REDIRECT --to-port 8000• Captive Portal site will add IP to “allowed” list

Stealth Services

• Use iptables to snag traffic before it hits the real service

• iptables -t nat -A PREROUTING -p tcp --dport 80 -m recent --name secretweb -j REDIRECT --to-ports 84

• Simple way to run multiple VirtualHosts with 1 IP and a border firewall allowing 1 inbound port

• Interesting application of pattern match: HTTP Host header

Anti-Spam Blacklisting

• Pre-populate recency table with RBN• iptables -p tcp --dport 25 -m recent --name

spammer --rcheck -j REJECT

Anti-Spam Q&D Graylisting

• Block initially, allow connect after 2 minutes• iptables -p tcp --dport 25 -m recent --name

smtpdelay --rcheck --seconds 120 -j DROP• iptables -p tcp --dport 25 -m recent --name

smtpdelay --rcheck ‘!’ --seconds 120 -j ALLOW• iptables -p tcp --dport 25 -m recent --name

smtpdelay --set -j DROP

Box? What box?

• Expanding iptables with -j QUEUE• Pass entire packet to userspace app• Whole new world to extend iptables• Can only be one app running for this!!!– If multiple “tricks” in place, app must use input

validation to determine what is appropriate for which packet.

QUEUE tutorial

• http://michael.toren.net/slides/ipqueue/slide001.html

• Old (2003) but excellent!

Application: Packet Rewriting(Stolen blatantly from http://michael.toren.net/slides/ipqueue/slide025.html )

my ($dns, $err) = Net::DNS::Packet->new(\$udp->{data}); ... for my $section qw(answer authority additional) { my @rr;

while (my $i = $dns->pop($section)) { if ($i->class eq "IN" && $i->type eq "A" && $i->name eq "slashdot.org") { $i->{address} = "127.0.0.1"; }

push @rr, $i; }

while (my $i = pop @rr) { $dns->push($section, $i); } }

QUEUE example: countertrace

• Add “imaginary” hops• Optionally add latency– Low-priority users get slower net access

• Quick & dirty honeynet / tarpit

Reject with TTL Expired

• The “ultimate” unreachable response• Alternatives:– TCP RST: host exists, port not listening to you– ICMP port / host / net unreachable– Black-hole route– Throw away packet

• TTL expired implies you can’t get there from here– So stop trying

Application Awareness

• Windows Firewall has exceptions per-app– “Outlook” can use internet, “Notepad” can’t

• Iptables is port-focused only– No app awareness

• Create queue handler to look up app for the socket, whitelist apps.

Application Awareness example(Stolen blatantly from http://michael.toren.net/slides/ipqueue/slide017.html ) my %approved = qw( 561c1c9071e8c5723c641273e725c1e3 /usr/bin/telnet 9dc35c04c16d3f2ce2a8537961980913 /usr/bin/nc ); ... my ($user, $pid) = getuserfromtcp $ip, $tcp;

open EXE, "/proc/$pid/exe"; my $md5 = Digest::MD5->new->addfile(*EXE)->hexdigest; close EXE;

if ($pid > 0 && ! $approved{$md5}) { $ipq->set_verdict($msg->packet_id, NF_DROP); syslog "warning", "Blocking outbound connection attempt by unauthorized program"; } else { $ipq->set_verdict($msg->packet_id, NF_ACCEPT); }

Theoretical - 4D Box

Local Darknet

• Multipoint VPV to avoid eavesdropping• Multicast goes everywhere locally, plus

potentially through routers– Detect outbound multicast & split

• Iptables TEE target– Create duplicate of packet & send someplace else– May require patching iptables, maybe kernel

NAT’d host ID

• Fingerprint incoming packets to differentiate a single host behind a shared NAT address

• Base on super-session state: already seen connections?– TOFU again

“Signed” TCP

• TCP includes a checksum – why not a MD?• Add transparent proxy at each end– Use REDIRECT or just QUEUE– Slightly more intrusive with netcat / ssh / etc.

• “Signature” is H(checksum + SEQ + PW)– Easy to generate, easy to verify (re-generate)– Password can be negotiated on first connect

• Hash mismatch -> discard packet– No session hijack possible?

• Easier than IPSec, and “might” survive NAT.

Closing the Box

• iptables is almost a programming language• If iptables isn’t enough of a programming

language, write a program to config it• If it can be done to a packet, it can likely be

done in iptables• If it can’t be done in iptables, it can be done

externally