© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Troubleshooting DNSSEC
Troubleshooting DNSSEC
a few handles to get you started
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
ToolbagToolbag• dig• drill• unbound-host• Packet analysis
• wireshark• tcpdump• dnscap (in combination with the
above)
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Try to understand the wire
Try to understand the wire
• A DNS Packet has a header and 4 sections:• Question• Answer• Authoritative• Additional
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
HEADERHEADERID
QDCOUNT
ANCOUNT
NSCOUNT
ARCOUNT
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
HeaderHeaderAA TC RD RA 0 AD CD
OpcodeOpcode0 Query [RFC1035]1 IQuery (Inverse Query, Obsolete) [RFC3425]2 Status [RFC1035]3 Unassigned4 Notify [RFC1996]5 Update [RFC2136]6-15 Unassigned
Query (0)Query (0)Response Response
(1)(1)
RCODERCODE
Hexadecimal Name Description ----------- ---------- -----------------------------------0 NoError No Error 1 FormErr Format Error 2 ServFail Server Failure 3 NXDomain Non-Existent Domain 4 NotImp Not Implemented 5 Refused Query Refused 6 YXDomain Name Exists when it should not 7 YXRRSet RR Set Exists when it should not 8 NXRRSet RR Set that should exist does not 9 NotAuth Server Not Authoritative for zone 10 NotZone Name not contained in zone 11-15 Unassigned
FLAGSFLAGS
Flag Description ---- ---------------------AA Authoritative Answer TC Truncated Response RD Recursion Desired RA Recursion Allowed ReservedAD Authentic Data CD Checking Disabled
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
; <<>> DiG 9.7.0b2 <<>> @a0.org.afilias-nst.info. org NS +dnssec; (2 servers found);; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49574;; flags: qr aa rd; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 13;; WARNING: recursion requested but not available;; OPT PSEUDOSECTION:; EDNS: version: 0, flags: do; udp: 4096;; QUESTION SECTION:;org. IN NS;; ANSWER SECTION:org. 86400 IN NS b0.org.afilias-nst.org.org.
86400 IN NS d0.org.afilias-nst.org.org. 86400 IN NS a0.org.afilias-nst.info.org. 86400 IN NS a2.org.afilias-nst.info.org. 86400 IN NS b2.org.afilias-nst.org.org. 86400 IN NS c0.org.afilias-nst.info.org. 86400 IN RRSIG NS 7 1 86400 20100415154437 ( 20100401144437 47948 org. BSO2Encp2iwdCtgeKXCyi0PsVZFU8ai1zInCveqPxBuWgGIpy7HRamerEg7fQ+PWvxr3F0k/zTUdFifRi1paOHbG MRfvOG9XHskSxoUqxwi2jRAIXWYmXz3A/NsjgoJVsIEj
3DWGP43cTJMoOsS68qmK7CbbyLrSTRdg6/d/mK4= );; ADDITIONAL SECTION:a0.org.afilias-nst.info. 86400 IN A 199.19.56.1a0.org.afilias-nst.info. 86400 IN AAAA 2001:500:e::1a2.org.afilias-nst.info. 86400 IN A 199.249.112.1a2.org.afilias-nst.info. 86400 IN AAAA 2001:500:40::1b0.org.afilias-nst.org. 86400 IN A 199.19.54.1b0.org.afilias-nst.org. 86400 IN AAAA 2001:500:c::1b2.org.afilias-nst.org. 86400 IN A 199.249.120.1b2.org.afilias-nst.org. 86400 IN AAAA 2001:500:48::1c0.org.afilias-nst.info. 86400 IN A 199.19.53.1c0.org.afilias-nst.info. 86400 IN AAAA 2001:500:b::1d0.org.afilias-nst.org. 86400 IN A 199.19.57.1d0.org.afilias-nst.org. 86400 IN AAAA 2001:500:f::1;; Query time: 409 msec;; SERVER: 2001:500:e::1#53(2001:500:e::1);; WHEN: Thu Apr 8 08:44:33 2010;; MSG SIZE rcvd: 597
Question to Authoritative Question to Authoritative NameserverNameserver
Request DNS informationRequest DNS information
Question SectionQuestion Section
Answer SectionAnswer Section
Authority SectionAuthority Section
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
OPT RR: EDNSOPT RR: EDNS
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
EDNSEDNS
• Communicate ability to deal with 512+ IP packets (fragmentation buffers)
• Communicate willingness to receive DNSSEC resource records• Space for much more resource
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Deeper Understanding?
Deeper Understanding?
• http://www.iana.org/assignments/dns-parameters
• follow the links to the RFCs
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
What Can Go Wrong
What Can Go Wrong
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Possible FailuresPossible Failures
• Local Configuration• Secure Delegation Failure• True validation failure• Transport problems
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Local ConfigurationLocal Configuration
• Time:DNSSEC is critically dependent on time. Check your NTP configuration
• use date -u "+%Y%m%d%H%M%S"• Check signature validity times
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
; <<>> DiG 9.7.0b2 <<>> @a0.org.afilias-nst.info. org NS +dnssec; (2 servers found);; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49574;; flags: qr aa rd; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 13;; WARNING: recursion requested but not available;; OPT PSEUDOSECTION:; EDNS: version: 0, flags: do; udp: 4096;; QUESTION SECTION:;org. IN NS;; ANSWER SECTION:org. 86400 IN NS b0.org.afilias-nst.org.org.
86400 IN NS d0.org.afilias-nst.org.org. 86400 IN NS a0.org.afilias-nst.info.org. 86400 IN NS a2.org.afilias-nst.info.org. 86400 IN NS b2.org.afilias-nst.org.org. 86400 IN NS c0.org.afilias-nst.info.org. 86400 IN RRSIG NS 7 1 86400 20100415154437 ( 20100401144437 47948 org. BSO2Encp2iwdCtgeKXCyi0PsVZFU8ai1zInCveqPxBuWgGIpy7HRamerEg7fQ+PWvxr3F0k/zTUdFifRi1paOHbG MRfvOG9XHskSxoUqxwi2jRAIXWYmXz3A/NsjgoJVsIEj
3DWGP43cTJMoOsS68qmK7CbbyLrSTRdg6/d/mK4= );; ADDITIONAL SECTION:a0.org.afilias-nst.info. 86400 IN A 199.19.56.1a0.org.afilias-nst.info. 86400 IN AAAA 2001:500:e::1a2.org.afilias-nst.info. 86400 IN A 199.249.112.1a2.org.afilias-nst.info. 86400 IN AAAA 2001:500:40::1b0.org.afilias-nst.org. 86400 IN A 199.19.54.1b0.org.afilias-nst.org. 86400 IN AAAA 2001:500:c::1b2.org.afilias-nst.org. 86400 IN A 199.249.120.1b2.org.afilias-nst.org. 86400 IN AAAA 2001:500:48::1c0.org.afilias-nst.info. 86400 IN A 199.19.53.1c0.org.afilias-nst.info. 86400 IN AAAA 2001:500:b::1d0.org.afilias-nst.org. 86400 IN A 199.19.57.1d0.org.afilias-nst.org. 86400 IN AAAA 2001:500:f::1;; Query time: 409 msec;; SERVER: 2001:500:e::1#53(2001:500:e::1);; WHEN: Thu Apr 8 08:44:33 2010;; MSG SIZE rcvd: 597
SIGNATURE validitySIGNATURE validity
$ date -u "+%Y%m%d%H%M%S"20100408095947
against authoritative serveragainst authoritative server
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Secure delegationsSecure delegations
• Secure delegations: Look for the DS• Matching Key IDs?• NSEC proof?• Hard to do manually
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Looking at the chain of trustLooking at the chain of trust
• CLI based• drill -T or drill -S (trace or chace)• dig +sigchace
• Web Based• dnsviz: http://dnsviz.net/• debugger:
http://dnssec-debugger.verisignlabs.com/
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
drill -Sdrill -S
;; Chasing: example.net. SOADNSSEC Trust tree:example.net. (SOA)|---example.net. (DNSKEY keytag: 17000) |---example.net. (DNSKEY keytag: 49656) |---example.net. (DS keytag: 49656) |---net. (DNSKEY keytag: 62972) |---net. (DNSKEY keytag: 13467) |---net. (DS keytag: 13467) |---. (DNSKEY keytag: 63380) |---. (DNSKEY keytag: 63276);; Chase successful
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
drill -Tdrill -T -k < root.ksk > example.net SOA
drill -Tdrill -T -k < root.ksk > example.net SOA
;; Domain: .[T] . 100 IN DNSKEY 256 3 5 ;{id = 63380 (zsk), size = 1024b}. 100 IN DNSKEY 257 3 5 ;{id = 63276 (ksk), size = 1280b}Checking if signing key is trusted:New key: . 100 IN DNSKEY 256 3 5 AQPQyahTOOaR/Pi6p ... Q== ;{id = 63380 (zsk), size = 1024b} Trusted key: . 3600 IN DNSKEY 257 3 5 AQOv6tbkmW+ ... 1iY/ ;{id = 63276 (ksk), size = 1280b} Trusted key: . 100 IN DNSKEY 256 3 5 AQPQyahTOOaR/ ... MiBmsMQ== ;{id = 63380 (zsk), size = 1024b}Key is now trusted! Trusted key: . 100 IN DNSKEY 257 3 5 AQOv6tbkmW+1 ... 1iY/ ;{id = 63276 (ksk), size = 1280b}[T] net. 100 IN DS 13467 5 2 ec9b094786b82c46aa3054c7352b59904b697119d515b4ea536ec3dd9a10ed81 net. 100 IN DS 13467 5 1 de01426e08ddb9186502ccc1081390cd7da0e178 ;; Domain: net.[T] net. 100 IN DNSKEY 256 3 5 ;{id = 62972 (zsk), size = 1024b}net. 100 IN DNSKEY 257 3 5 ;{id = 13467 (ksk), size = 1280b}Checking if signing key is trusted:New key: net. 100 IN DNSKEY 256 3 5 AQPVP6Je ... 8h3J3Gw== ;{id = 62972 (zsk), size = 1024b} Trusted key: . 3600 IN DNSKEY 257 3 5 AQOv6tbkmW+ ... 1iY/ ;{id = 63276 (ksk), size = 1280b} Trusted key: . 100 IN DNSKEY 256 3 5 AQPQyahT ... msMQ== ;{id = 63380 (zsk), size = 1024b} Trusted key: . 100 IN DNSKEY 257 3 5 AQOv6tbkmW ... oewi1iY/ ;{id = 63276 (ksk), size = 1280b} Trusted key: net. 100 IN DNSKEY 256 3 5 AQPVP6 ... 3J3Gw== ;{id = 62972 (zsk), size = 1024b}Key is now trusted! Trusted key: net. 100 IN DNSKEY 257 3 5 AQOsAH77.... QuH ;{id = 13467 (ksk), size = 1280b}[T] example.net. 100 IN DS 49656 5 1 3850efb913aec66275bca53221587d445702397e example.net. 100 IN DS 49656 5 2 9e06b299abe811d699e077fff990ff5a1b496c914deb22697ba22a1da31f0a6e ;; Domain: example.net.[T] example.net. 100 IN DNSKEY 256 3 5 ;{id = 17000 (zsk), size = 1024b}example.net. 100 IN DNSKEY 257 3 5 ;{id = 49656 (ksk), size = 1280b}[T] example.net. 100 IN SOA ns.example.net. olaf.nlnetlabs.nl. 2002050501 100 200 604800 100;;[S] self sig OK; [B] bogus; [T] trusted
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
External viewsExternal views
• DNSVIZ: http://dnsviz.net/
• Verisign’s DNSSEC Debugger:http://dnssec-debugger.verisignlabs.com/
• Secspider:http://secspider.cs.ucla.edu/
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
DNSKEY represented as DLVDNSKEY represented as DLV
No connection betweenNo connection betweensecret-wg.or and orgsecret-wg.or and org
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Connection betweenConnection betweensecret-wg.or and org existsecret-wg.or and org exist
Same zone different, after secure delegation
Same zone different, after secure delegation
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Case StudyCase Study
• NASA DS Rollover failure impacting Comcast:http://www.dnssec.comcast.net/DNSSEC_Validation_Failure_NASAGOV_20120118_FINAL.pdf
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
looking at a validating server
looking at a validating server
• So you got a SERVAIL• dig +cd will disable checking
• you get an answer?• likely validation failure
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Transport problemsTransport problems
• If responses grow beyond 512octets:• UDP may see fragmentation and
dropped fragments (firewalls etc)• fragmentation problems on path?• Fallback to TCP
• Port 53 TCP sometimes blocked
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Does the network support DNSSEC?Does the network support DNSSEC?• One tool you could use for a
quick assesment:• http://netalyzr.icsi.berkeley.e
du/
• you contribute to good science too!
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
On your serverOn your server• val-log-level:
• val-log-level: 0 - prints nothing• val-log-level: 1 - print queries that
fail• val-log-level: 2 - print reason why
it failed
• remember unbound-control set_option?
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Apr 08 12:31:06 unbound[853:0] info: validation failure<dnssec1.gsa.dnsops.gov. A IN>: signature expired from 159.142.174.98for key gsa.dnsops.gov. while building chain of trust
Apr 08 10:28:01 unbound[853:0] info: validation failure<barney.llnl.dnsops.gov. SOA IN>: No DNSKEY record from 128.115.249.61for key barney.llnl.dnsops.gov. while building chain of trust
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
Unbound-hostUnbound-host
• unbound-host tool is useful for taking a first stab• Runs the unbound validator from the command
line• unbound-host -v -f trustanchor example.com• prints the val-log-level 2 error message if it
fails.• with -C it can read unbound.conf for settings.• with -d (or -dddd) you get a high verbosity
trace
© 2006-2012 NLnet Labs, Licensed under a Creative Commons Attribution 3.0 Unported License.
RemediesRemedies• Clean your cache in case of
problems locally• Bind:
• rndc flush
• rndc flushname
• Unbound• unbound-control flush
• unbound-control flush_zone
• unbound-control flush_infra