Chapter 6Configuring, Monitoring &
Troubleshooting IPsec
6.1 Overview of IPsecBenefits of IPsecRecommended Uses of IPsecTools Used to Configure IPsecWhat are Connection Security Rules ?
Benefits of IPsecIPsec – suite of protocols that allows secure,
encrypted communication between 2 computers over an unsecured network
2 goals; to protect IP packet & to defend against network attacks
IPsec secures network traffic by using encryption & data signing
An IPsec policy defines the type of traffic that Ipsec examines, how that traffic is secured & encrypted, and how IPsec peers are authenticated
Recommended Uses of IPsecAuthenticating & encrypting host-to-host
trafficAuthenticating & encrypting traffic to serversLayer 2 Tunneling Protocol (L2TP)/IPsec for
VPN connectionsSite-to-site (gateway-to-gateway) tunnelingEnforcing logical networks (server/domain
isolation)
Tools Used to Configure IPsecWindows Firewall with Advanced Security
MMC (used for Windows Server 2008 & Windows Vista)
IP Security Policy MMC (used for mixed environments & to configure policies that apply to all Windows versions)
Netsh command-line tool
What are Connection Security Rules?Connection security rules involve:- Authenticating 2 computers before they begin
communications- Securing information being sent between 2 computers- Using key exchange, authentication, data integrity &
data encryption (optionally)How firewall rules & connection rules are related:- Firewall rules allow traffic through, but do not secure
that traffic- Connection security rules can secure the traffic, but
creating a connection security rule does not allow traffic through the firewall
6.2 Configuring Connection Security RulesChoosing a Connection Security Rule TypeWhat are Endpoints?Choosing Authentication RequirementsAuthentication MethodsDetermining a Usage Profile
Choosing a Connection Security Rule TypeRule Type Description
Isolation Restricts connections based on authentication criteria that you define
Authentication Exemption
•Exempts specific computers, or a group or range of IP addresses, from being required to authenticate•Grants access to those infrastructure computers with which this computer must communicate before authentication occurs
Server-to-server Authenticates 2 specific computers, 2 groups of computers, 2 subnets, or specific computer & a group of computers or subnet
Tunnel Provides secure communications between 2 peer computers through tunnel endpoints (VPN or L2TP IPsec tunnels)
Custom Enables you to create a rule with special settings
What are Endpoints?Computer endpoints are the computers or the
group of computers that form peers for the connection
IPsec tunnel mode protects an entire IP packet by treating it as an AH or ESP payload
ESP encrypts packets and applies a new unencryptes header to facilitate routing
ESP function in 2 modes:1. Transport mode2. Tunnel mode
IP HDR Data
IP HDR ESP HDR
Encrypted Data
ESP TRLR
ESP Auth
IP HDR Data
ESP HDR
ESP TRLR
ESP Auth
New IP HDR
Encrypted IP Packet
ESP Transport Mode
ESP Tunnel Mode
Choosing Authentication RequirementsOption Description
Request Authentication for inbound and outbound connections
Ask that all inbound/outbound traffic be authenticated, but allow the connection if authentication fails
Require authentication for inbound connections and request authentication for outbound connections
•Require inbound be authenticated or it will be blocked•Outbound can be authentication fails
Require authentication for inbound and outbound connections
Require that all inbound/outbound traffic be authenticated or the traffic will be blocked
Authentication MethodsMethod Key Points
Default Use the authentication method configured on the IPsec Settings tab
Computer & User (Kerberos V5)
You can request or require both the user & computer authenticate before communications can continue; domain membership required
Computer (Kerberos V5) Request or require the computer to authenticate using Kerberos V5
User (Kerberos v5) Request or require the user to authenticate using Kerberos V5; domain membership required
Computer certificate •Request or require a valid computer certificate, requires at least one CA•Only accept health certificates: request or require a valid health certificate to authenticate, requires IPsec NAP
Advanced Configure any available method; you can specify methods for First & Second Authentication
Determining a Usage ProfileSecurity settings can change dynamically with the
network location typeWindows supports 3 network types :- Domain: selected when the computer is a domain
member- Private: networks trusted by the user (home or small
office network)- Public: default for newly detected networks, usually
the most restrictive settings are assigned because of the security risks present on public networks
The network location type is most useful on portable computers which are likely to move from network to network
6.3 Configuring IPsec NAP EnforcementIPsec Enforcement for Logical NetworksIPsec NAP Enforcement ProcessesRequirements to Deploy IPsec NAP
Enforcement
IPsec Enforcement for Logical Networks
SHAs NAP agent NAP ECs
HRAVPN802.1xDHCPNPS proxy
NAP administration serverNetwork policiesNAP health policiesConnection request policiesSHVs
SHAsNAP agentNAP ECs
Certificate servicesEmail serversNAP policy servers
Secure servers
Compliant NAP client
Secure NetworkBoundary Network
Restrictednetwork
NAP enforcement servers
Remediation servers
Non-compliantNAP client
Non-NAPCapable client
NPS servers
IPsec NAP Enforcement Processes
VPN Server Active
Directory
IEEE 802.1x Devices
Health Registration Authority
NAP Health Policy Server
DHCP Server
Remediation Server
NAP Client with limited access
Perimeter Network
Internet
Intranet
Restricted network
IPsec NAP Enforcement includes:
• Policy validation• NAP enforcement• Network restriction• Remediation• Ongoing monitoring of compliance
Requirements to Deploy IPsec NAP EnforcementActive DirectoryActive Directory Certificate ServicesNetwork Policy ServerHealth Registration Authority
6.4 Monitoring IPsec ActivityTools used to Monitor IPsecUsing IP Security Monitor to Monitor IpsecUsing Windows Firewall with Advanced
Security to Monitor IPsec
Tools Used to Monitor IPsecTool Key Points
IP Security Monitor
• Used in Windows XP and higher
• MMC snap-in
• Administrators can monitor local and remote IPsec policy usage
IPsecmon
• Only available in Windows 2000
• Command-line tool
• Reduced level of information available for troubleshooting
Windows Firewall with Advanced Security MMC
New in Windows Vista and Windows Server 2008
Detailed IKE tracing using Netsh
• Trace file found in: systemroot\debug\oakley.log
• Enabled in Windows XP and Windows 2000 through Registry modification
Using IP Security Monitor to Monitor IPsec
Options for using the IP Security Monitor:• Modify IPsec data refresh interval to update information in the
console at a set interval• Allow DNS name resolution for IP addresses to provide additional
information about computers connecting with IPsec• Computers can monitored remotely:
• To enable remote management editing, the HKLM\system\currentcontrolset\services\policyagent keymust have a value of 1
• To Discover the Active security policy on a computer, examine the Active Policy Node in the IP Security Monitoring MMC
• Main Mode Monitoring monitors initial IKE and SA:• Information about the Internet Key Exchange
• Quick Mode Monitoring monitors subsequent key exchanges related to IPsec:
• Information about the IPsec driver
Using Windows Firewall with Advanced Security to Monitor IPsec
• Use the Connection Security Rules and Security Associations nodes to monitor IPsec connections
• The Connection Security Rules and Security Associations nodes will not monitor policies defined in the IP Security Policy snap-in
• Items that can be monitored include: • Security Associations• Main Mode• Quick Mode
The Windows Firewall in Windows Vista and Windows Server 2008 incorporates IPsecThe Windows Firewall in Windows Vista and Windows Server 2008 incorporates IPsec
6.5 Troubleshooting IPsecIPsec Troubleshooting ProcessTroubleshooting Internet Key Exchange
(IKE)Troubleshooting IKE Negotiation Events
IPsec Troubleshooting Process
Stop the IPsec Policy Agent and use the ping command to verify communications
Verify firewall settings
Start the IPsec Policy Agent and use IP Security Monitor to determine if a security association exists
Verify that the policies are assigned
Review the policies and ensure they are compatible
11
22
33
44
55
Use IP Security Monitor to ensure that any changes are applied66
Troubleshooting IKE
Identify connectivity issues related with IPsec and IKEüü
Identify firewall and port issuesüü
View the Oakley.log file for potential issuesüü
Determine Main mode exchange issuesüü
Troubleshooting IKE Negotiation EventsCommon Security Event log codes:
• Success:• 541 - IKE Main Mode or Quick Mode established• 542 - IKE Quick Mode was deleted• 543 - IKE Main Mode was deleted
• Information Log Entries:• Largely pertains to monitoring for denial of service attacks • There might not be any errors but resources will
run low, which affects performance for legitimate clients• Quick Mode audit failures are denoted with 547 error message
End of Chapter 6