Configuring Microsoft Windows IP Security to Operate with HP ...

HP-UX IPSecConfiguring Microsoft Windows IP Securityto Operate with HP-UX IPSec

HP Part Number: J4256-90025Published: June 2007Edition: 1.0

2

Table of Contents

About This Document.........................................................................................................9Typographic Conventions......................................................................................................................9Introduction..........................................................................................................................................11

Testing Environment.......................................................................................................................11Known Problem with Windows 2000 SP1 and SP2...................................................................11

Protocol Implementation Differences..............................................................................................12Windows IP Security Configuration Overview....................................................................................13Configuring a Windows Host-to-Host Policy.......................................................................................14

Step 1: Starting the IP Security Policies Snap-in Configuration Utility...........................................15Step 2: Creating a Policy..................................................................................................................15Step 3: Adding a Rule......................................................................................................................16Step 4: Creating the IP Filter List and Filters for the Rule...............................................................18Step 5: Configuring Filter Actions for the Rule...............................................................................21Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule..................25Step 7: Configuring the Connection Type for the Rule...................................................................26Step 8: Modifying IKE Parameters for the Policy............................................................................26Step 9: Starting the IP Security Service............................................................................................29Step 10: Assigning the IP Security Policy........................................................................................30Step 11: Verifying the Configuration...............................................................................................31Example...........................................................................................................................................31

Windows Configuration.............................................................................................................31HP-UX Configuration................................................................................................................32

Additional Options...............................................................................................................32Configuring a Windows End-to-End Tunnel Policy.............................................................................33

Outbound Tunnel Rule Requirements............................................................................................33Inbound Tunnel Rule Requirements...............................................................................................33Configuring a Tunnel Rule..............................................................................................................33Example...........................................................................................................................................34

Windows Configuration.............................................................................................................34Outbound Rule.....................................................................................................................34Inbound Rule........................................................................................................................35Additional Parameters..........................................................................................................36

HP-UX Configuration................................................................................................................37Troubleshooting Tips............................................................................................................................38

Using IKE Logging on HP-UX Systems..........................................................................................38Using IKE Logging on Windows Systems.......................................................................................38Additional Windows Troubleshooting Tools..................................................................................39

Comparing HP-UX and Windows IPsec Configuration Parameters....................................................40Mirrored Filters...............................................................................................................................41Filter Selection.................................................................................................................................42IKE Parameter Selection..................................................................................................................42IKE SA Key (Master Key) Lifetime Values......................................................................................42

HP-UX IKE SA Lifetime Values.................................................................................................42Windows IKE SA Lifetime Values..............................................................................................43

Maximum Quick Modes..................................................................................................................43Perfect Forward Secrecy (PFS).........................................................................................................43IPsec SA Key (Session Key) Lifetime Values...................................................................................43

HP-UX IPsec SA Lifetime Values...............................................................................................43Windows IPsec SA Lifetime Values...........................................................................................44

Related Publications..............................................................................................................................45

Table of Contents 3

glossary.............................................................................................................................47

4 Table of Contents

List of Figures1 IP Security Policy Wizard..............................................................................................................162 Rules Tab.......................................................................................................................................173 Rule Properties Dialog Box...........................................................................................................174 Creating an IP Filter List...............................................................................................................185 Address Tab for Filter Properties..................................................................................................196 Protocol Tab for Filter Properties..................................................................................................207 Selecting the Filter List for a Rule.................................................................................................218 Security Methods for Filter Action................................................................................................229 Security Method Dialog Box.........................................................................................................2310 Custom Security Methods Settings Dialog Box............................................................................2411 Selecting the Filter Action.............................................................................................................2512 Configuring A Preshared Key.......................................................................................................2613 General Policy Properties Dialog Box ..........................................................................................2714 Key Exchange Settings Dialog Box ...............................................................................................2815 IKE Security Algorithms Dialog Box ............................................................................................2916 IPSEC Services Properties Dialog Box...........................................................................................3017 Assigning the IP Security Policy...................................................................................................3118 Outbound Rule Filter....................................................................................................................3519 Outbound Rule Tunnel Settings....................................................................................................3520 Inbound Rule Filter.......................................................................................................................3621 Inbound Rule Tunnel Settings.......................................................................................................36

5

6

List of Tables1 IPsec Parameters on Windows and HP-UX .................................................................................40

7

8

About This DocumentThis document describes how to configure Microsoft Windows IP Security to operate with theHP-UX IPSec product.

Typographic ConventionsThis document uses the following typographical conventions:%, $, or # A percent sign represents the C shell system prompt. A dollar

sign represents the system prompt for the Bourne, Korn, andPOSIX shells. A number sign represents the superuser prompt.

audit(5) A manpage. The manpage name is audit, and it is located inSection 5.

Command A command name or qualified command phrase.Computer output Text displayed by the computer.Ctrl+x A key sequence. A sequence such as Ctrl+x indicates that you

must hold down the key labeled Ctrl while you press anotherkey or mouse button.

ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH.[ERROR NAME] The name of an error, usually returned in the errno variable.Key The name of a keyboard key. Return and Enter both refer to the

same key.Term The defined use of an important word or phrase.User input Commands and other text that you type.Variable The name of a placeholder in a command, function, or other

syntax display that you replace with an actual value.[] The contents are optional in syntax. If the contents are a list

separated by |, you must choose one of the items.{} The contents are required in syntax. If the contents are a list

separated by |, you must choose one of the items.... The preceding element can be repeated an arbitrary number of

times.Indicates the continuation of a code example.

| Separates items in a list of choices.WARNING A warning calls attention to important information that if not

understood or followed will result in personal injury ornonrecoverable system problems.

CAUTION A caution calls attention to important information that if notunderstood or followed will result in data loss, data corruption,or damage to hardware or software.

IMPORTANT This alert provides essential information to explain a concept orto complete a task

NOTE A note contains additional information to emphasize orsupplement important points of the main text.

Typographic Conventions 9

10

IntroductionThis document contains the following sections:• “Windows IP Security Configuration Overview” (page 13)

This section contains a brief overview of the Windows IPsec configuration parameters andthe terminology used in the Windows IPsec configuration utilities.

• “Configuring a Windows Host-to-Host Policy” (page 14)This section describes how to configure IP Security (IPsec) on a Windows client to secureIP packets sent to and received from an HP-UX system in a host-to-host topology.

• “Configuring a Windows End-to-End Tunnel Policy” (page 33)This section describes how to configure IPsec on aWindows client to secure IP packets sentto and received from an HP-UX system in an end-to-end tunnel topology.

• “Troubleshooting Tips” (page 38)This section contains troubleshooting tips.

• “Comparing HP-UX and Windows IPsec Configuration Parameters” (page 40)This section compares howHP-UXandWindows systems configure anduse IPsec parameters.

• “Related Publications” (page 45)This section contains a list of related HP-UX and Microsoft publications.

The procedures and examples in this document use preshared keys for IKE authentication. Forinformation about using certificates for IKE authentication with Microsoft Windows, see UsingMicrosoft Windows Certificates with HP-UX IPSec, available at http://docs.hp.com.The intended audience for this document is an HP-UX IPSec administrator who is familiar withthe HP-UX IPSec product and with the IP Security protocol suite. If you are not familiar withthe HP-UX IPSec product, see the appropriate version of theHP-UX IPSec Administrator's Guide,available at http://docs.hp.com.

NOTE: The IP Security protocol suite is often referred to as IPsec. The HP-UX product thatimplements the IP Security protocol suite is HP-UX IPSec.

Testing EnvironmentThe procedures in this white paper were tested using the following environment:

DescriptionComponent

Versions A.02.01 and A.02.01.01HP-UX IPSec

Windows XP with Service Pack 2 (SP2)Microsoft Windows Client

Known Problem with Windows 2000 SP1 and SP2For this white paper, HP did not test with Windows 2000 systems. However, there is a knownproblemwithWindows 2000 base systems andWindows 2000 systemswith Service Pack 1 (SP1)or Service Pack 2 (SP2). The IP Securitymodule on these systems does not properly process IPSecESP packets that are fragmented across IP packets and drops these packets. The symptoms varyaccording to how the applications handle the dropped packets.This problem is caused by a defect in theWindows 2000 SP1/ SP2 software and is fixed inWindows2000 Service Pack 3 (SP3).

Introduction 11

The aboveproblem typically occurswith ESP-encryptedUDPor ICMPpackets that are fragmentedby IP. HP-UX 11i systems minimize IP fragmentation of ESP-encrypted TCP packets. You maystill experience problems with ESP-encrypted TCP packets sent from an HP-UX system to aWindows 2000 system if an intermediate IP gateway fragments the ESP packet.

Protocol Implementation DifferencesHP-UX and Microsoft Windows both implement the IP Security protocol suite. However, thereare features in the protocol suite that HP-UX implemented which Microsoft did not implement,and vice-versa.The following features are implemented by HP-UX IPSec version A.02.01 but not by MicrosoftWindows XP:• Advanced Encryption Standard (AES): HP-UX IPSec supports ESP encryption using the

following protocols: AES, Triple Data Encryption Standard (3DES), and Data EncryptionStandard (DES).WindowsXP andWindows 2000 support 3DES andDES, but do not supportAES.

• Aggressive Mode (AM): HP-UX supports AM exchanges to establish IKE SecurityAssociations (SAs). AM is an optional feature and is not supported on Windows.

The following features are implemented by Microsoft Windows XP, but not by HP-UX IPSecversion A.02.01:• Kerberos: Windows supports Internet Key Exchange (IKE) authentication using Kerberos.

RFC 2408 defines an optional Kerberos Token payload, but does not describe how toimplement it. This feature is not supported on HP-UX.

• Perfect Forward Secrecy (PFS) for keys only: HP-UX IPSec supports PFS for keys inconjunction with PFS for all identities, but does not support PFS for keys only. Windowssupports PFS for keys only (“session key PFS”) and PFS for keys in conjuctions with PFS forall identities (“master key PFS”). See “Perfect Forward Secrecy (PFS)” (page 43) for moreinformation.

12

Windows IP Security Configuration OverviewOn Microsoft Windows systems, all IP Security (IPsec) configuration data resides in a single IPSecurity policy. You can create multiple IP Security policies, but only one local policy can beactive on the system. If the system is a member of a Windows Active Directory domain, you canuse an IP Security policy from a Group Policy defined for the domain.A Windows IP Security policy defines the parameters used to negotiate Internet Key ExchangeSecurity Associations (IKE SAs) and IPsec SAs. An IKE SA is a bi-directional, securecommunication channel that two peers establish before negotiating IPSec SAs. One of the primaryactivities during the IKE SA negotiation is the authentication of each peer's identity.After two peers establish an IKE SA, they can negotiate IPsec SAs. Each IPsec SA is auni-directional, secure communication channel. The IPsec SA operating parameters include theIPsec protocol used (Encapsulating Security Payload, ESP, or Authentication Header, AH) andthe cryptographic algorithms. IPsec SAs are negotiated in pairs (one for each direction of traffic).Each Windows IP Security policy contains the following components:• Rules

A policy contains one or more rules. The main purpose of a rule is to assign actions foraddress filters. Each rule contains the following components:— IP Filter List

An IP Filter list contains one or more filters. Each filter contains the followingcomponents:◦ Addressing

The source and destination IP addresses, network masks, and a flag that indicatesif the filter is mirrored (bi-directional).

◦ ProtocolThe upper-layer protocol, and source and destination ports, if applicable.

◦ DescriptionThe filter name and a description.

— Filter ActionThe filter action specifies the action to take for the rule, and can be one of the followingactions:◦ allow: allow the packet to pass◦ block: discard the packet◦ negotiate security: negotiate IPsec Authentication Header (AH) or Encapsulating

Security Payload (ESP) Security Associations (SAs)

— Authentication MethodsThe authentication methods specify the type of Internet Key Exchange (IKE)authentication to use (preshared key or certificates with RSA signatures). If you areusing preshared key authentication, the authentication methods also specify the valueof the preshared key.

Windows IP Security Configuration Overview 13

— Tunnel SettingsThe tunnel settings specify if the rule is a tunnel rule. If it is a tunnel rule, the settingsalso specify the tunnel destination endpoint.

— Connection TypeThe connection type specifies the connection (link) types for the rule, such as LAN.

• GeneralThe general parameters for a policy specify IKE SA parameters, such as the IKE encryptionalgorithm, IKE hash (integrity algorithm), Diffie-Hellman Group, and IKE SA key lifetimes.The parameters correspond to IKE SA proposals. You can configure multiple IKE SAproposals and specify the preference order. The proposals are used for all rules in the policy.

By comparison, aminimalHP-UX IPSec configuration consists of one ormore IPsec host policies,one ormore IKE policies, and one ormore authentication records. The IPsec host policies specifyaddress filters, and you can configure separate IKE policies for each peer. “Comparing HP-UXand Windows IPsec Configuration Parameters” (page 40) lists IPsec configuration parametersand how they are configured in the HP-UX IPSec and the Windows IP Security configurationutilities.

Configuring a Windows Host-to-Host PolicyThis section describes one method for configuring host-to-host policy on a Windows XP clientusing the IP Security Policies snap-in utility. Windows also supports command-line utilities toconfigure IP Security policies: ipseccmd onWindowsXP systems and netsh onWindows 2003systems. For more information about these utilities, see the Windows documentation set.To use this method, complete the following steps:1. Start the IP Security Policies snap-in utility. See “Step 1: Starting the IP Security Policies

Snap-in Configuration Utility” (page 15).2. Create an IP Security policy. See “Step 2: Creating a Policy” (page 15).3. Add a rule to the policy. See “Step 3: Adding a Rule” (page 16).4. Create a Filter List for the rule and configure filters. See “Step 4: Creating the IP Filter List

and Filters for the Rule” (page 18).5. Configure filter actions for the rule. The filter actions contain IPsec transforms or other

actions. See “Step 5: Configuring Filter Actions for the Rule” (page 21).6. Configure the IKE authentication method and preshared key for the rule. See “Step 6:

Configuring the IKE Authentication Method and Preshared Key for the Rule” (page 25).7. Specify the network link (connection) types for the rule. See“Step 7: Configuring the

Connection Type for the Rule” (page 26).8. Modify the IKE SA parameters for the policy. By default, Windows clients will use IKE SA

parameters that are compatiblewith the defaultHP-UX IPSec parameters. If these parametersare acceptable, you can skip this step. See “Step 8: Modifying IKE Parameters for the Policy”(page 26).

9. Start the IP Security service. The IP Security service must be running before you can assignthe new IP Security policy. See “Step 9: Starting the IP Security Service” (page 29).

10. Assign (activate) the new IP Security Policy. See “Step 10: Assigning the IP Security Policy”(page 30).

11. Verify the configuration. See “Step 11: Verifying the Configuration” (page 31).Because this is a host-to-host rule, we will use the default value for the rule tunnel setting (notunnel). For information about configuring a tunnel rule and the tunnel setting, see “Configuringa Windows End-to-End Tunnel Policy” (page 33).

14

Step 1: Starting the IP Security Policies Snap-in Configuration UtilityUse the following procedure to start the IP Security Policies configuration utility. This utility isa snap-in module for the Microsoft Management Console (MMC).1. Start theMicrosoftManagement Console (MMC). From theMicrosoft Start menu, clickRun

and type MMC. Click OK.2. If the IP Security Policies snap-in configuration utility is not loaded, use the following

procedure to add it:a. From the MMC window, click File→Add/Remove Snap-in.b. From the Add/Remove Standalone Snap-in window, click Add.c. From the Add Standalone Snap-in window, scroll down to IP Security Policy

Management and select it. Click Add.d. In the Select Computer or Domain window, select Local computer (in this procedure,

we are configuring IP Security for the local computer). Click Finish.e. Close the Add Standalone Snap-in window by clicking Close.f. Close the Add/Remove Snap-in window by clicking OK.

Step 2: Creating a PolicyUse the following procedure to create an IP Security policy. An IP Security policy is a set of IPsecconfiguration parameters. Only one local IP Security policy can be active (assigned) on a system.1. In the left navigation pane of the IP Security Policy Management snap-in, click IP Security

Policies on Local Computer to display all IP Security Policies. Depending on yourWindowsplatform, there may be IP Security Policies already configured.

2. Right click IP Security Policies on Local Computer and select Create IP Security Policy.3. The Policy Wizard starts and displays a startup message. Click Next.4. The Policy Wizard opens the IP Security Policy Name window. Enter a name in the Name

field. This name is used only for internal identification.Click Next.

5. The PolicyWizard opens theRequests for SecureCommunicationwindow.Clear theActivatethe default response rule check box, as shown in Figure 1. (The default response rule is apre-configured rule that causes the Windows system to dynamically build a filter list basedon the receipt of IKE requests. By default, the Windows system attempts to use IPsec onlyif it receives an IKE request from a remote system.)Click Next.

Configuring a Windows Host-to-Host Policy 15

Figure 1 IP Security Policy Wizard

6. The Policy Wizard opens the Completing the IP Security policy wizard window. Select theEdit properties check box if it is not already selected.Click Finish.The IP Security configuration utility opens the Policy Properties dialog box. The title of thewindow will be name Policy, where name is the policy name.

Step 3: Adding a RuleThe primary purpose of a rule is to assign actions to filters. A rule also specifies IKE authenticationmethods. Use the following procedure to add a rule to the IP Security policy:1. Select the Rules tab in the Policy Properties dialog box.

Clear the Use Add Wizard box check box if it is selected (Figure 2).Click Add.

16

Figure 2 Rules Tab

2. The IP Security configuration utility opens the Rule Properties dialog box, which has a tabfor each category of rule configuration data: IP Filter List, Filter Action, AuthenticationMethods, Tunnel Setting, and Connection Type ( Figure 3).

Figure 3 Rule Properties Dialog Box

Configuring a Windows Host-to-Host Policy 17

TIP: After you have created a rule, you can open the Rules Properties dialog box by rightclicking the rule and selecting Properties.

Step 4: Creating the IP Filter List and Filters for the RuleAn IP filter list can contain one or more filters. IPsec uses the filters to determine which rule toapply to an IP packet. The IP Security configuration utility displays the rules for a policy inreverse alphabetical order based on the name of the IP filter list for the rule.Filter Order There is no method for specifying the search or priority order for the filters in arule or for the order of rules in a policy. The Windows IP Security module automatically createsan internal filter list and orders the filters from most specific to least specific.Use the following procedure to configure the IP filter list and a filter:1. Create an IP filter list.

Select the IP Filter tab from the Rule Properties dialog box.The IP Filter List tab shows a list of filters already defined for IP Security policies. Each rulecan have only one filter list, but the filter list can specify multiple filters. In this example,we will create a new filter list that contains one filter.Click Add at the bottom of the dialog box.The IP Security configuration utility opens the IP Filter List dialog box. In the Name field,enter a name for the filter list. This name is used only for internal identification. Optionally,add a description. In Figure 4, the administrator enters the name foo.Clear the Use Add Wizard box check box if it is selected.Click Add.

Figure 4 Creating an IP Filter List

18

The IP Security configuration utility opens a Filter Properties dialog box.

2. Select the Addressing tab in the Filter Properties dialog box. Use the drop-down menus tospecify the address types for the source and destination addresses. The selections are:• My IP Address1

• Any IP Address• A specific DNS Name• A specific IP Address• A specific IP SubnetEnter the source and destination IP addresses or DNS names for the filter. If you selected Aspecific IP Subnet, enter the subnet mask.

WARNING! Be careful when configuring filters that affect packets required for basicnetwork operation, such as packets exchanged with DNS servers and ICMP packetsexchanged with routers. If you configure a policy that requires IP Security for these packetsand the remote node does not support IP Security, your system can lose network functionality.

Leave the Mirrored check box selected, which creates a bi-directional filter that applies topackets to and from the destination system. See “Mirrored Filters” (page 41) for moreinformation about mirrored filters.In Figure 5, the administrator specifies an address filter with the Windows system address(10.1.1.1) as the source address and the HP-UX system address (10.2.2.2) as the destinationaddress. The Mirrored check box is selected, so the address filter also matches packets fromthe HP-UX system.

Figure 5 Address Tab for Filter Properties

3. Select the Protocol tab in the Filter Properties dialog box. By default, the filter applies to allprotocol types. Select the protocol type (for example, TCP) from the drop-down box. If youselect TCP or UDP, you can also specify the From (source) port and To (destination) port.Click OK to return to the Filter Properties dialog box.

1. HP-UX did not test the My IP Address selection with multihomed Windows systems. However, the Windowsdocumentation states that in a multi-homed system, My IP Address matches every IP address on the system.

Configuring a Windows Host-to-Host Policy 19

In Figure 6, the administrator specifies protocol information for aWindows system that willbe a telnet client. The protocol type is TCP, the source port is a wildcard (any port), and thedestination port is the IANA registered TCP port number for the telnet service, 23.

Figure 6 Protocol Tab for Filter Properties

20

4. From the IP Filter List dialog box, you can add another filter to the filter list by clicking theAdd button.ClickOK in the IP Filter List dialog box to return to the IP Filter List tab in the Rule Propertiesdialog box.

5. Add the filter list to the rule by selecting the option button for the filter list you just created.In Figure 7, the administrator added the filter list foo for the rule.

Figure 7 Selecting the Filter List for a Rule

Step 5: Configuring Filter Actions for the RuleThe filter action specifies the action to take for the rule, such as allow (pass), block (discard), ornegotiate security (negotiate IPsec AH or ESP Security Associations). If you select negotiatesecurity, the filter action also specifies parameters for IPsec Security Association (SA) proposals:ESP or AH transforms and IPSec SA key lifetimes. A rule can have only one filter action, but thefilter action can specify multiple IPsec SA proposals. You can specify the order for the IPsec SAproposals.The filter actions you configure in the Windows IP Security rule must be compatible with thevalue or values specified for the -action argument in the HP-UX ipsec_config add hostor add tunnel command.Use the following procedure to configure filter actions:1. Select the Filter Action tab from the Rule Properties dialog box.

The Filter Action tab shows a list of filter actions already defined for IP Security. In thisprocedure, we will create a new filter action.Clear the Use Add Wizard check box if it is selected and click Add.

2. The IP Security configuration utility opens the Filter Action Properties dialog box with thefollowing tabs:

Configuring a Windows Host-to-Host Policy 21

Security Methods•• GeneralSelect the Security Methods tab, then select Negotiate security. Verify that the followingcheck boxes are not selected:2

• Accept unsecured communication, but always respond using IPSec.• Allow unsecured communication with non-IPSec-aware computer.In addition, verify that the Session key perfect forward secrecy (PFS) check box is not selected.(HP-UX does not support session key PFS, also referred to as PFS for keys only. HP-UXsupports PFS for keys only in conjunction with PFS for identities. See “Perfect ForwardSecrecy (PFS)” (page 43) for more information.)For example:

Figure 8 Security Methods for Filter Action

Click Add.The IP Security configuration utility opens the Security Method dialog box (Figure 9):

2. HP-UX IPSec does not have options that are equivalent to these check boxes. If an HP-UX IPsec policy requires IPsecurity, then HP-UX always requires IP security for packets that match the policy and drops any packets that matchthe policy but are not secured.

22

Figure 9 Security Method Dialog Box

The Encryption and Integrity and Integrity only methods each correspond to a set ofpredefined parameters for an IPsec SA proposal, including an IPsec transform type (suchas ESP). The transforms and additional SA parameters defined for these methods may varyaccording to theWindows release installed.OnWindowsXP systemswith SP2, thesemethodsare defined as follows:• Encryption and Integrity

Authenticated ESP using 3DES encryption and SHA1 authentication (this is equivalentto theHP-UX IPSec transformESP_3DES_HMAC_SHA1).No SA lifetimes are specified,and these settings are compatible with the HP-UX default SA lifetimes (see “IPsec SAKey (Session Key) Lifetime Values” (page 43) for more information).

• Integrity only

Authenticated ESP usingNULL encryption and SHA1 authentication. (this is equivalentto theHP-UX IPSec transformESP_NULL_HMAC_SHA1)No SA lifetimes are specified,and these settings are compatible with the HP-UX default SA lifetimes (see “IPsec SAKey (Session Key) Lifetime Values” (page 43) for more information).

To use a predefined method, select the appropriate method from the list and click OK toreturn to the Security Methods tab in the Filter Actions dialog box.To create a custom method, use the following procedure:a. Select Custom in the Security Method dialog box, then click Settings to open the

Custom Security Method Settings dialog box.b. In the Custom Security Method Settings dialog box (Figure 10), select the appropriate

transform type, algorithms, and session key lifetimes. See “IPsec SA Key (Session Key)Lifetime Values” (page 43) for more information about IPsec session key lifetimes.

Configuring a Windows Host-to-Host Policy 23

Figure 10 Custom Security Methods Settings Dialog Box

c. Click OK to return to the Security Methods tab in the Filter Actions dialog box.If the parameters you configured for a custommethodmatch a predefinedmethod, theconfiguration utility will display an informative message and select the matchingpredefined method.

3. From the SecurityMethods tab, you can addmoremethods (IPsec SA proposals) by clickingthe Add button. If you havemultiple IPsec SA proposals, the configuration utility lists themin the preference order IKE will use when negotiating IPsec SAs. You can change the orderby using theMove up andMove down buttons. You can also use theDelete button to deleteIPsec SAproposals (such as proposals that useDES,which has been cracked—data encryptedusing DES has been decrypted by unauthorized parties) .

4. (Optional) Configure the action name. When you create a new action (transform), theconfiguration utility assigns it the name New Filter Action. To change the name, select theGeneral tab from theAction Properties dialog box. Enter a newname and clickOK to returnto the Filter Action tab in the Rule Properties dialog box.

5. Apply the new action to the rule. In the Filter Action tab, click on the option button for thefilter action you just created to apply the action to the rule you are configuring. For example,in Figure 11, the administrator created the action my_action and selected it for the rule.Click Apply.

24

Figure 11 Selecting the Filter Action

Step 6: Configuring the IKE Authentication Method and Preshared Key for the RuleWhen configuring a rule to be compatiblewithHP-UX IPSec, the authenticationmethod specifiesthe IKE authenticationmethod (preshared key or certificates) for IPsec. The authenticationmethodmust match the value specified for the -authentication argument in the ipsec_configadd ike command.Windows also allows you to configure Kerberos (Active Directory) as an authentication methodfor IKE (this is the default), but HP-UX does not support this authentication method.Use the following procedure to configure the IKE authentication method:1. Select the Authentication Methods tab from the Rule Properties dialog box.2. Click Add to open the Authentication Method dialog box.3. To use IKE authentication with a preshared key, select Use this string. This is equivalent

to specifying -authentication PSK in the ipsec_config add ike command.Enter the preshared key asASCII text. Do not enclose the key in double quotes. The presharedkey must match the preshared key on the HP-UX system, which is configured using the-preshared argument in the ipsec_config add auth command. For example:

Configuring a Windows Host-to-Host Policy 25

Figure 12 Configuring A Preshared Key

To use IKE authentication with certificates, select Use a certificate from this certificationauthority (CA). ClickBrowse. The IP Security configuration utility opens a Select Certificatebox with a list of CA certificates stored on your system. Select the CA for the appropriateCA and click OK. (For additional information about configuring Microsoft Windowscertificates, see Using Microsoft Windows Certificates with HP-UX IPSec, available athttp://docs.hp.com.

4. After you have specified the IKE authentication method, click OK to return to theAuthentication Methods tab in the Rule Properties dialog box.

5. In the Rule Properties dialog box, remove the Kerberos authentication method from theauthentication methods list by highlighting it and clicking Remove.The configuration utility will display a confirmation message (Are you sure?). Click Yes

Step 7: Configuring the Connection Type for the RuleThe connection type specifies the types of network connection to which the rule will apply. Bydefault, the IP Security configuration utility creates rules that apply to all network connectiontypes. To change the connection type, use the following procedure:1. Select the Connection Type tab from the Rule Properties dialog box.2. The IP Security configuration utility opens theConnection Type dialog boxwith the following

selections:• All network connections: the rule applies to all network connections• Local area network (LAN): the rule applies only to LAN connections• Remote access: the rule applies only to VPN and dial-up connectionsSelect the appropriate connection type and click OK. If you have configured all the requiredparameters for a rule, the IP Security configuration utility will return to the Policy Propertiesdialog box.

Step 8: Modifying IKE Parameters for the PolicyBydefault,HP-UX IPSec negotiates IKE SAsusing a single proposalwith the followingparameters:

26

• Encryption algorithm: 3DES• Hash algorithm: MD5• Diffie-Hellman Group: 2• Maximum lifetime: 28,800 seconds (8 hours)• Maximum Quick Modes: 100You can specify alternative values for the above parameters in the ipsec_config add ikecommand.On Windows XP systems with SP2, IP Security policies are pre-configured with four IKE SAproposals. The second IKE proposal matches the default HP-UX IPSec IKE proposal3, and willbe used by the two systems if no changes are made to the default configuration data. If theseIKE parametersmeet your security requirements, you do not need tomodify the IKE parametersand can skip to “Step 10: Assigning the IP Security Policy” (page 30).Use the following procedure to modify the Windows IKE SA parameters:1. From the Policy Properties dialog box, select the General tag. The IP Security configuration

utility opens the General dialog box (Figure 13).Click Advanced4. (Ignore the field labeled Check for policy changes. This field is usedonly when the policy is stored in an Active Directory.)

Figure 13 General Policy Properties Dialog Box

2. The IP Security configuration utility opens the Key Exchange Settings dialog box (Figure 14).

3. By default, the first Windows XP proposal has the following parameters: Encryption - 3DES; Hash - SHA1;Diffie-Hellman Group - 2. The third and fourth Windows proposals are weaker, and use DES encryption andDiffie-Hellman Group 1. Refer to the Windows documentation for more information.

4. On Windows 2003 servers, this button is labeled Settings.

Configuring a Windows Host-to-Host Policy 27

Figure 14 Key Exchange Settings Dialog Box

Configure the fields as follows:• Master key perfect forward secrecy (PFS)

Selecting this check box sets the maximum number of IPsec or Quick Mode (QM)negotiations that IKE can perform using an IKE SA to 1. It is equivalent to specifying-maxqm 1 in the ipsec_config add ike command. PFS is computationallyexpensive and HP recommends that you enable it only in hostile environments. See“Maximum Quick Modes” (page 43) and “Perfect Forward Secrecy (PFS)” (page 43)for more information.

• Authenticate and generate a new key after every: ____ minutes

This field specifies the maximum lifetime for an IKE SA in units of time. It is equivalentto the -life argument of the ipsec_add ike command.

TIP: Note that this value is specified in minutes on Windows systems and in secondson HP-UX systems.

• Authenticate and generate a new key after every: ____ sessions

This field specifies the maximum QM negotiations per IKE SA. It is equivalent to the-maxqm argument of the ipsec_add ike command. See “Maximum Quick Modes”(page 43) and “Perfect Forward Secrecy (PFS)” (page 43) for more information.

Modify the values as appropriate. If you do not want to modify the IKE encryption, hash,or Diffie-Hellman Group parameters, click OK to return to the General dialog box andcontinue to the next step.Tomodify IKE encryption algorithm, hash algorithm, or Diffie-HellmanGroup parameters,click Methods. The IP Security configuration utility opens the Key Exchange SecurityMethods dialog box, which lists IKE algorithms andDiffie-HellmanGroups that correspondto IKE SA proposals in order of preference. You can change the order by using the Moveup and Move down buttons. You can also delete IKE SA proposals (such as proposals thatuse DES, which has been cracked) using the Delete button.To add another IKE SA proposal (method), click Add.The IP Security configuration utility opens the IKE Security algorithms dialog box (Figure 15).

28

Figure 15 IKE Security Algorithms Dialog Box

Use the drop-downmenus to select the appropriate integrity algorithm, encryption algorithm,and Diffie-Hellman Group (these are equivalent to the -hash, -encryption, and -grouparguments of the ipsec_config add ike command).

3. Click OK to return to the Key Exchange Security Methods dialog box.Click OK to return to the Key Exchange Settings dialog box.Click Close to close the Policy Properties dialog box.

Step 9: Starting the IP Security ServiceUse the following procedure to start the IP Security service. The IP Security service must berunning before you can assign a policy.1. From the Microsoft Start menu, select Control Panel→Administrative Tools→Services.2. Scroll down and select IPSEC Services.3. The Service manager opens the IPSEC Services Properties dialog box (Figure 16).

In the Startup type selection menu, select Automatic.If the Service status is not Started, click Start.Click OK to close the IPSEC Services Properties dialog box.Close the Services dialog box.

Configuring a Windows Host-to-Host Policy 29

Figure 16 IPSEC Services Properties Dialog Box

Alternatively, you canmanually start the IP Security service by entering the followingWindowscommand:net start policyagent

You can also use the following sequence of commands tomanually stop and restart the IP Securityservice. This also clears any existing IPsec SAs,:net stop policyagent

net start policyagent

Step 10: Assigning the IP Security PolicyThe IP Security subsystem will not use the new policy until you assign (activate) it. Only one IPSecurity policy can be assigned or active for the system. To assign the new IP Security policy,return to the MMC window. Right click the policy in the MMC window and select Assign, asshown in Figure 17.

30

Figure 17 Assigning the IP Security Policy

Step 11: Verifying the ConfigurationTo verify your configuration, generate traffic that matches the address filter.On the HP-UX system, enter the following command to verify that the IKE SA and IPsec SAsare established:ipsec_report -sa

ExampleIn this example, IPsec secures telnet connections from theWindows system to theHP-UX system,using authenticated ESP.The Windows system's address is 10.1.1.1The HP-UX system's address is 10.2.2.2.

Windows ConfigurationThe Windows administrator configures and assigns an IP Security policy with the followingparameters:• One rule, with the following parameters:

— Filter List: One filter, with the following parameters:◦ Addressing:

– Source address: the Windows system's address.– Destination address: the HP-UX system's address.– Mirrored: yes (the Mirrored box is selected).These parameters are shown in Figure 5 (page 19).

◦ Protocol: TCP; source port any, destination port 23 (telnet).– Protocol: TCP– From port: any– To port: 23 (telnet server)These parameters are shown in Figure 6 (page 20).

— Filter Action: Negotiate security, using the default settings for Encryption and Integrity(authenticated ESP using 3DES and SHA1).

— Authentication Method: IKE using the preshared key my_preshared_key, as shownin Figure 12 (page 26).

— Tunnel Settings: No tunnel (this is the default).— Connection Type: All network connections (this is the default).

• General parameters: The general parameters for the policy are set to the default values (fourIKE SA proposals, including 3DES encryption, SHA1 integrity and Diffie-Hellman Group2).

Configuring a Windows Host-to-Host Policy 31

HP-UX ConfigurationOn the HP-UX system, the administrator configures the following policies and records:ipsec_config add host telnet_from_foo1 \-source 10.2.2.2/32/TELNET -destination 10.1.1.1 \-action ESP_3DES_HMAC_SHA1

ipsec_config add ike foo1 -remote 10.1.1.1 -auth PSK

ipsec_config add auth foo1 -remote 10.1.1.1 \-psk my_preshared_key

If the HP-UX IPSec subsystem is not already started, the administrator starts it using theipsec_admin -start command.

Additional Options

For information on additional options and commands, see theHP-UX IPSec Administrator's Guideand the following manpages:ipsec_admin(1M)ipsec_config(1M)ipsec_policy(1M)ipsec_report(1M).

32

Configuring a Windows End-to-End Tunnel PolicyThe only IPsec tunnel topology supported between an HP-UX system and aWindows system isan end-to-end tunnel.5The procedure for configuring an end-to-end tunnel policy on Windowssystem is the same as procedure for configuring a host policy, except that you must configuretwo, non-mirrored rules: one rule for outbound packets and one rule for inbound packets, asdescribed in the sections that follow.

NOTE: Do not configure any other rules in the policy with the HP-UX system address as thedestination address. This prevents theMicrosoft system from applying the tunnel transform overa host-to-host (transport) transform. In end-to-end tunnel topologies, HP-UX IPSec does notsupport transport transforms over a tunnel transform.

Outbound Tunnel Rule RequirementsThe outbound tunnel rule must have the following parameters:• Filter List: One filter, with the following parameters:

— Address:◦ Source address: the HP-UX system's address.◦ Destination address: this must be a specific IP address and must be the Windows

system's address.◦ Mirrored: no (the Mirrored box is cleared).

— Protocol Type: none (wildcard). The Windows documentation states that the filters intunnel rules must not specify protocols or ports to ensure that IP Security can correctlyprocess IP fragments.

• Tunnel Setting— Tunnel endpoint: theHP-UX system's address. This is the address of the tunnel endpoint

closest to the destination. Since this is an end-to-end tunnel, it is the same as thedestination address in the address filter.

Inbound Tunnel Rule RequirementsThe inbound tunnel rule must have the following parameters:• Filter List: One filter, with the following parameters:

— Address:◦ Source address: the Windows system's address.◦ Destination address: this must be a specific IP address and must be the HP-UX

system's address.◦ Mirrored: no (the Mirrored box is cleared).

— Protocol Type: none (wildcard).• Tunnel Setting

— Tunnel endpoint: the Windows system's address. This is the address of the tunnelendpoint closest to the destination. Since this is an end-to-end tunnel, it is the same asthe destination address in the address filter

Configuring a Tunnel RuleUse the following procedure to configure an outbound or inbound tunnel rule.

5. You can also configure an IPsec topology where packets exchanged between an HP-UX system and a Windowssystem are tunneled through an IPsec gateway device, but neither HP-UX nor Windows systems can be configuredas IPsec gateways. The only topology in which an HP-UX system can act as an IPsec gateway is when the HP-UXsystem is a Home Agent for Mobile IPv6 clients. The HP-UX IPSec Administrator's Guide describes how to configurea host-to-gateway IPsec topology using HP-UX and a Cisco router.

Configuring a Windows End-to-End Tunnel Policy 33

TIP: The tunnel setting is used by all packets selected using the address filters for the rule. Donot include any filters for host-to-host (non-tunneled) packets in the filter list for a rule with atunnel.

1. Start the IP Security Policies snap-in if necessary.2. Create an IP Security policy ormodify an existing policy. Tomodify an existing policy, select

the policy in the right navigation pane and right click the policy. Select Properties.3. The IP Security configuration utility opens the Policy Properties dialog box. Select theRules

tab. Click Add to create a new rule or select a rule you want to modify and click Edit.4. Configure a new rule or modify an existing rule with the appropriate address filter for the

outbound tunnel rule or inbound tunnel rule, as described in “Outbound Rule” (page 34)or “Inbound Rule” (page 35). See “Step 4: Creating the IP Filter List and Filters for the Rule”(page 18) if you need additional information about configuring address filters.Record the destination address; you will need it to configure the tunnel endpoint.

5. Return to the Rule Properties dialog box. Select the Tunnel Setting tab.6. The IP Security configuration utility opens the Tunnel Setting dialog box.

Select The tunnel endpoint is specified by this IP address.Enter the IP address of the tunnel endpoint closest to the destination. Since this is anend-to-end tunnel, it is the same as the destination address in the address filter.

7. Click Close to close the Tunnel Setting dialog box.8. If this is a new rule, complete the configuration by configuring the appropriate filter action,

authentication methods, and connection type.Click Close to close the Rule Properties dialog box.

ExampleIn this example, IPsec secures all packets between the Windows system and the HP-UX systemusing authenticated ESP.The Windows system's address is 10.1.1.1The HP-UX system's address is 10.2.2.2.

Windows ConfigurationOn the Windows system, you configure one rule for outbound packets and one for inboundpackets.

Outbound Rule

The outbound rule is for packets from the Windows system (source address 10.1.1.1) to theHP-UX system (destination address 10.2.2.2). Figure 18 shows the address filter for this rule, andFigure 19 shows the corresponding tunnel settings:

34

Figure 18 Outbound Rule Filter

Figure 19 Outbound Rule Tunnel Settings

Inbound Rule

The inbound rule is for packets to the Windows system (destination address 10.1.1.1) from theHP-UX system (source address 10.2.2.2). Figure 18 shows the address filter for this rule, andFigure 21 shows the corresponding tunnel settings:

Configuring a Windows End-to-End Tunnel Policy 35

Figure 20 Inbound Rule Filter

Figure 21 Inbound Rule Tunnel Settings

Additional Parameters

You must configure the remaining rule parameters (filter action, authentication methods, andconnection type) to be compatible with the HP-UX configuration. In addition, the generalparameters for the rule (the IKESAparameters)must be compatiblewith theHP-UX configuration.

36

HP-UX ConfigurationOn theHP-UX system, the host and tunnel policies are bi-directional (mirrored), so you configureonly one host policy and only one tunnel policy. Since this is an end-to-end tunnel, the tunnelpolicy does not have to specify the tunnel endpoints. HP-UX IPSec will use the end source andend destination addresses as the tunnel addresses (the tsource and tdestination valuesdefault to the source and destination values).ipsec_config add host foo1 -source 10.2.2.2 \-destination 10.1.1.1 -action PASS -tunnel foo1_tunnel

ipsec_config add tunnel foo1_tunnel -source 10.2.2.2 \-destination 10.1.1.1 -action ESP_3DES_HMAC_SHA1

Youmust also configure an IKEpolicy and an authentication record to complete the configuration:ipsec_config add ike foo1 -remote 10.1.1.1 -auth PSK

ipsec_config add auth foo1 -remote 10.1.1.1 \-psk my_preshared_key

Configuring a Windows End-to-End Tunnel Policy 37

Troubleshooting TipsMost interoperability problems occur during IKE negotiations, so examining IKE log events isuseful. You can use the following procedures to enable and view IKE log events:

Using IKE Logging on HP-UX SystemsUse the following procedure to view detailed IKE log events on HP-UX systems:1. Enter the following command to set the HP-UX IPSec log level to debug and increase the

maximum log file size:ipsec_admin -al debug -maxsize 99999

IPSec creates the log files in the /var/adm/ipsec directory. The log file names areauditdateinfo.log

2. Reproduce the problem.3. Enter the following command to format the audit file:

ipsec_report -audit /var/adm/ipsec/auditdateinfo.log

4. Use the following command to set the HP-UX IPSec log level back to warning (the defaultlog level):ipsec_admin -al warning

Using IKE Logging on Windows SystemsUse the following procedure to view IKE log events on Windows systems:1. Enable IKE logging. OnWindowsXP systems, use the regedit utility to enable IKE logging

in the system registry. On Windows systems, IKE logging is configured using the Oakley6key.

CAUTION: Incorrectly editing the registrymay severely damage the system. Beforemakingchanges to the registry, HP recommends that you back up the registry and any valued dataon the computer. Refer to the article How to back up, edit, and restore the registry in WindowsXP and Windows Server 2003 in the Windows Knowledge Base for more information. TheWindows Knowledge Base is available at http://support.microsoft.com

Set theHKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Oakley\EnableLoggingREG_DWORD value to 1. On some Windows versions, you may need to create the Oakleykey.On Windows 2003 systems, enter the following command to enable IKE logging:netsh ipsec dynamic set config ikelogging 1

2. Stop and restart the IP Security service. You can use the following commands at theWindowscommand prompt:net stop policyagent

net start policyagent

Refer to “Step 9: Starting the IP Security Service” (page 29) for more information.

3. Reproduce the problem.4. View the IKE log file. Windows creates the log file in the directory systemroot\Debug (by

default, this is the WINDOWS\Debug directory). The file name is Oakley.log.

6. The Oakley protocol is a key-agreement protocol that is incorporated in the IKE protocol.

38

5. Disable IKE logging. On Windows XP systems, set theHKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Oakley\EnableLoggingREG_DWORD value to 0.On Windows 2003 systems, enter the following command:netsh ipsec dynamic set config ikelogging 0

6. Stop and restart the IP Security service.

Additional Windows Troubleshooting ToolsWindows supports an IP Secu;rityMonitor snap-in utility for theMicrosoftManagement Console(MMC) that provides IPsec statistics and information about IKE and IPsec Security Associations.Refer to the Windows documentation set for more information.

Troubleshooting Tips 39

Comparing HP-UX and Windows IPsec Configuration ParametersThis section contains Table 1, which compares howHP-UX andWindows systems configure andstore IPsec parameters. It also contains the following subsections, which provide additionalcomparative information:• “Mirrored Filters” (page 41)• “Filter Selection” (page 42)• “IKE Parameter Selection” (page 42)• “IKE SA Key (Master Key) Lifetime Values” (page 42)• “Maximum Quick Modes” (page 43)• “Perfect Forward Secrecy (PFS)” (page 43)• “IPsec SA Key (Session Key) Lifetime Values” (page 43)

Table 1 IPsec Parameters on Windows and HP-UX

NotesHP-UX ConfigurationWindows ConfigurationParameter

Windows and HP-UXsupport subnetmasks for IPaddresses andwildcards forIP addresses, protocols, andport numbers.See “Mirrored Filters”(page 41) for additionalinformation.

Specify one filter per host,tunnel, or gateway policy.Use the -source and-destination argumentsin theipsec_config addhost , tunnel, orgateway command.

Specify them in the FilterList for a rule. The Filter Listcan contain multipleaddress filters.

Address Filters

HP-UX IPSec supports ESPencryption using thefollowing protocols:Advanced EncryptionStandard (AES), Triple DataEncryption Standard(3DES), and DataEncryption Standard (DES).WindowsXP andWindows2000 support 3DES andDES, but do not supportAES.

Specify them using the-action argument in theipsec_config addgateway,host, ortunnelcommand.

Specify them in the FilterAction for a rule.

IPsec SA Proposals

See “Filter Selection”(page 42) for additionalinformation.

Specify it using the-priority argument inthe ipsec_config addgateway or hostcommand.

Not applicable.Filter Priority

See “IPsec SA Key (SessionKey) Lifetime Values”(page 43) for additionalinformation.

Specify it in the transformspecification for the-action argument in theipsec_config add hostor tunnel command.

Specify it in the CustomSecurity Methods dialogbox under the Filter Actionfor a rule.

Maximum IPsec SALifetime, measured by timeor by data

See “Mirrored Filters”(page 41) for additionalinformation.

Specify the endpoints usingthe -tsource and-tdestinationarguments oftheipsec_config addtunnel command.

Specify the destinationtunnel endpoint (theendpoint for thedestination) in the TunnelSettings for a rule. Youmustconfigure twouni-directional(non-mirrored) rules.

Tunnel endpoint address

Specify it using the -authargument of theipsec_config add ikecommand.

Specify it in theAuthenticationMethods fora rule.

IKEAuthenticationMethod

40

Table 1 IPsec Parameters on Windows and HP-UX (continued)

NotesHP-UX ConfigurationWindows ConfigurationParameter

Specify it using the-preshared argument ofthe ipsec_config addauth command.

Specify it in theAuthenticationMethods fora rule.

IKE Preshared Key

Specify it using the-exchange argument ofthe ipsec_config addauth command. Thedefault value is MM (MainMode).

Windows supports onlyMain Mode exchanges.

IKE Exchange Type

The Windows IP SecurityPolicy snap-in utility usesminutes as the time unit.TheHP-UXipsec_configcommand uses seconds asthe time unit. See “IKE SAKey (Master Key) LifetimeValues” (page 42) foradditional information.

Specify it using the-lifeargument in theipsec_config add ikecommand.

Specify it in the KeyExchange Settings dialogbox. (To navigate to the KeyExchange Setting dialogbox, select the General tabin the Policy Propertiesdialog box, then selectAdvanced settings.)

Maximum IKE SALifetime,measured by time

See “Maximum QuickModes” (page 43) foradditional information.

Specify it using the-maxqmargument in theipsec_config add ikecommand.

Specify it in the KeyExchange Settings dialogbox. (To navigate to the KeyExchange Setting dialogbox, select the General tabin the Policy Propertiesdialog box, then selectAdvanced settings.)

Maximum Quick Mode(QM) negotiations per IKESA

See “Perfect ForwardSecrecy (PFS)” (page 43) formore information.

HP-UX does not supportPFS for session keys.HP-UXsupports only PFS formaster keys.Specify PFS for master keysusing the-maxqm 1argument in theipsec_config add ikecommand.

Windows supports PFS forkeys only (PFS for sessionkeys) and supports PFS forkeys in conjunction withPFS for all identities (PFSfor master keys).Specify PFS for master keysin the Key ExchangeSettings dialog box. (Tonavigate to the KeyExchange Setting dialogbox, select the General tabin the Policy Propertiesdialog box, then selectAdvanced settings.)

Perfect Forward Secrecy(PFS)

See “IKE ParameterSelection” (page 42) foradditional information.

You can specify theparameters for one IKE SAproposal in an IKE policy,using the -encryption,-hash, and -grouparguments in anipsec_config add ikecommand.

Specify it in the Generalparameters for a policy. Youcan configure multiple IKESA proposals and theirpreference order.

IKE SA Proposals

Mirrored FiltersMicrosoft filters can be mirrored (bi-directional) or not mirrored (uni-directional). If the filter ismirrored, the filter will match IP packets with the source and destination addresses and portsreversed. For example, a filter has the following specifications:Source address: 10.1.1.1Destination address: 10.2.2.2

Comparing HP-UX and Windows IPsec Configuration Parameters 41

The filter matches packets with the following addresses:Source address: 10.1.1.1Destination address: 10.2.2.2If the filter is mirrored, it also matches packets with the following addresses:Source address: 10.2.2.2Destination address: 10.1.1.1The mirror setting only affects Windows IP Security behavior before IPsec SAs are established.If the Windows IP Security module receives a packet via an existing SA, it does not verify thatthe packet address fields match the address filter used when the SA was established.By comparison, HP-UX IPSec host and tunnel policies are always mirrored. (Gateway policiesare the only HP-UX IPSec policies that are not mirrored.)

Filter SelectionWindows does not allow you to specify the search or priority order for the filters in a rule or forthe order of rules in a policy. TheWindows IP Security module automatically creates an internalfilter list and orders the filters from most specific to least specific.HP-UX IPSec allows you to specify a priority value for IPsec and IKE policies. HP-UX IPSecsearches the policies in priority order within each type of policy. Lower priority values havehigher priority (priority value 1 is the highest priority).If you do not specify a priority value when creating a policy on HP-UX, ipsec_configautomatically assigns a priority value so that the new policy is the last policy searched beforethe default policy within its policy type. The output of the ipsec_config show commandincludes the priority values for configured policies.

IKE Parameter SelectionOnHP-UX systems, only one IKE SA proposal is used for each peer. You can configure multipleIKE policies, but only one IKE policy is selected per peer, and each IKE policy specifies only oneIKE SA. During IKE negotiations, IKE searches policies in priority order and selects the firstpolicy with a matching remote address. IKE then uses the IKE SA parameters to send an IKE SAproposal, or to evaluate the IKE SA proposal(s) it receives.On Windows systems, you can configure a set of multiple IKE SA proposals, but only one setper IP Security policy, and only one IP Security policy can be in use (assigned) on the system.

IKE SA Key (Master Key) Lifetime ValuesIKE SA key lifetimes (referred to as Master key lifetimes on Windows systems) specify themaximum lifetimes for IKE SA keys and are specified by units of time (seconds). In addition,users can specify the maximum number of IPsec SA negotiations that can be completed per IKESA (“Maximum Quick Modes” (page 43)).

HP-UX IKE SA Lifetime ValuesThe HP-UX IPSec default preferred lifetime value for IKE SAs is 28,800 seconds (eight hours).If theHP-UX system initiates IKE SAnegotiations, theHP-UX IKEdaemonproposes the preferredlifetime value to the remote system. The remote system may process this value in any manneraccording to the IPsec protocol suite.If the remote system initiates IKE SA negotiations and sends a proposed value that is longer than(less secure than) the HP-UX preferred value, HP-UX sends an IKE NOTIFY message with itspreferred value, and this value is used for the SA.If the remote system initiates IKE SA negotiations and sends a proposed lifetime that is the sameor more secure (shorter than) the HP-UX preferred value, the HP-UX IKE daemon accepts the

42

proposed value sent by the remote system if it is within the range specified by the IPsec protocolsuite.

Windows IKE SA Lifetime ValuesBy default, Windows XP systems use the following values for preferred IKE key lifetime values:480 minutes (eight hours)0 (infinite) IPsec SA negotiations (sessions)In testing with HP-UX IPSec, HP configured a shorter IKE SA lifetime value on the Windowssystem. When the Windows system was the initiator, it sent the configured lifetime value to theremote system. When the Windows system was the responder, it accepted the value sent by theHP-UX system but did not send a notification message.

Maximum Quick ModesHP-UX andWindows enable you to specify themaximumnumber of IPsec or QuickMode (QM)negotiations that IKE can complete per IKE SA. Each IPsec SA negotiation establishes two IPsecSAs (one in each direction).The default maximum QM values are as follows:HP-UX: 100Windows: 0 (infinite)If the value for maximum QM is 1, Perfect Forward Secrecy (PFS) for both keys and identities isimplemented. See “Perfect Forward Secrecy (PFS)” (page 43) for more information.

Perfect Forward Secrecy (PFS)With Perfect Forward Secrecy, the exposure of one key permits access only to data protected bythat key. RFC 2409, The Internet Key Exchange (IKE), defines two forms of PFS:• PFS for both the keys and the IKE identities. PFS is provided for keys in conjuctionwith PFS

for identities. IKE deletes the IKE SA after the IPsec negotiation completes. Each IKE SA isused for only one IPsec negotiation.The Windows interface refers to this type of PFS as master key PFS.

• PFS for IPsec keys only. The IKE peers perform a key exchange (Diffie-Hellman exchange)to create new keying material for each IPsec negotiation. The IKE SA is re-used until theIKE SA lifetime expires.The Windows interface refers to this type of PFS as session key PFS.

HP-UX IPSec supports PFS for both the keys and the IKE identities but does not support PFS forIPsec keys only. To be compatible with HP-UX IPSec, do not configure session key PFS onWindows systems.Configuring PFS is computationally expensive. In most topologies, the strength of thecryptographic algorithms is sufficient protection. HP recommends that you enable PFS only inhostile environments.

IPsec SA Key (Session Key) Lifetime ValuesIPsec SA key lifetimes (referred to as session key lifetimes on Windows systems) specify themaximum lifetimes for IPsec SA keys and are specified by units of time (seconds) and by dataunits transferred (kbytes).

HP-UX IPsec SA Lifetime ValuesBy default, HP-UX uses the following values for preferred lifetime values:28,800 seconds (eight hours)0 (infinite) data units

Comparing HP-UX and Windows IPsec Configuration Parameters 43

If the HP-UX system initiates IPsec SA negotiations, the HP-UX IKE daemon proposes thepreferred lifetime values to the remote system. The remote system may process these values inany manner according to the IPsec protocol suite.If the remote system initiates IPsec SA negotiations and sends proposed lifetime value that is assecure or more secure than the HP-UX preferred value (it is shorter than or equal to the HP-UXpreferred value), the HP-UX IKE daemon accepts the lifetime value proposed by the remotesystem if it is within the ranges specified by the IPsec protocol suite.If the remote system initiates IPsec SA negotiations and a proposed lifetime value is less secure(shorter than) the HP-UX preferred value, HP-UX sends an IKE NOTIFY message with itspreferred value. If this value is acceptable to the remote system, the SA negotiation succeeds andthe value sent in the NOTIFY message is used.

Windows IPsec SA Lifetime ValuesBy default, the Windows configuration does not specify any IPsec SA lifetime values and doesnot propose any during IPsec SA negotiations. This is equivalent to proposing the lifetime values28,800 seconds (eight hours) and 0 (infinite) data units.In testing with HP-UX, HP also configured specific IPsec SA lifetime values on the Windowssystem and observed behavior equivalent to HP-UX behavior. When the Windows systeminitiated the IPsec SA negotiation, it sent the configured lifetime values in the proposal. Whenthe remote system initiated the IPsec SA negotiation, theWindows system accepted the proposedlifetime value if it was more secure than its configured value, and sent a notification messagewhen its configured lifetime value was more secure than the value proposed by the remotesystem.

44

Related PublicationsThe following documents are available at http://docs.hp.com:• HP-UX IPSec Administrator's Guide• Using Microsoft Windows Certificates with HP-UX IPSec• HP-UX IPSec manpagesThe following documents are available at http://microsoft.com:• Step-by-Step Guide to Internet Protocol Security (IPSec)• IPSec troubleshooting tools

Related Publications 45

46

glossary3DES Triple Data Encryption Standard. A symmetric key block encryption algorithm that encrypts

data three times, using a different 56-bit key each time (168 bits are used for keys). 3DES issuitable for bulk data encryption.

AES Advanced Encryption Standard. Uses a symmetric key block encryption.HP-UX IPSec supportsAES with a 128-bit key. AES is suitable for encrypting large amounts of data.

AH The AH (Authentication Header) protocol provides data integrity, system-level authenticationfor IP packets. It can also provide anti-replay protection. The AH protocol is part of the IPsecprotocol suite.

authentication The process of verifying a user's identity or integrity of data, or the identity of the party thatsent data.

DES Data Encryption Standard. Uses a 56-bit key for symmetric key block encryption. It is suitablefor encrypting large amounts of data.DES has been cracked (data encoded using DES has been decoded by a third party).

Diffie-Hellman Method to generate a symmetric key where two parties can publicly exchange values andgenerate the same shared key. Start with prime p and generator g, which may be publiclyknown (typically these numbers are from a well-known Diffie-Hellman Group). Each partyselects a private value (a and b) and generates a public value (g**a mod p) and (g**b mod p).They exchange the public values. Each party then uses its private value and the other party'spublic value to generate the same shared key, (g**a)**b mod p and (g**b)**a mod p, which bothevaluate to g**(a*b) mod p for future communication.TheDiffie-Hellmanmethodmust be combinedwith authentication to preventman-in-the-middleor third party attacks (spoofing) attacks. For example, Diffie-Hellman can be usedwith certificateor preshared key authentication.

ESP The ESP (Encapsulating Security Payload) protocol provides confidentiality (encryption), dataauthentication, and an anti-replay service for IP packets. When used in tunnel mode, ESP alsoprovides limited traffic flow confidentiality. The ESP protocol is part of the IPsec protocol suite.

IKE The Internet Key Exchange (IKE) protocol is used before the ESP or AH protocol exchanges todetermine which encryption and/or authentication services will be used. IKE also manages thedistribution and update of the symmetric (shared) encryption keys used by ESP and AH.

IKEauthentication

Themethod used by IKE peers to authenticate each party's identity. HP-UX IPSec supports twoIKE authentication methods: preshared keys and RSA signatures using certificates.

IKE SA IKE Security Association. An IKE SA is a bi-directional, secure communication channel thatIKEuses to negotiate IPsec SAs. IKE can establish IKE SAs using eitherMainMode orAggressiveMode negotiations. Also referred to as IKE Phase One SA, ISAKMP SA, ISAKMP/MM SA,Aggressive Mode SA, Main Mode SA.

IPsec SA IPsec Security Association. An IPsec SA is a uni-directional, secure communication channel.The IPsec SA operating parameters include the IPsec protocol used (ESP or AH), the mode(transport or tunnel), the cryptographic algorithms (such asAES and SHA-1), the cryptographickeys, the SA lifetime, and the endpoints (IP addresses, protocol and port numbers). IKEestablishes IPsec SAs using Quick Mode negotiations. Also referred to as IKE Phase Two SA,IPsec SA, Quick Mode SA.

Perfect ForwardSecrecy (PFS)

With Perfect Forward Secrecy the exposure of one key permits access only to data protectedby that key. HP-UX IPSec supports PFS for keys and all identities (the IKE daemon can beconfigured to create a new IKE SA for each IPsec negotiation). HP-UX IPSec does not supportPFS for keys only (the IKE SA is re-used for multiple IPsec negotiations, with a newDiffie-Hellman key exchange for each IPsec negotiation).

SA See SecurityAssociation.A secure communication channel and its parameters, such as encryptionand authentication method, keys and lifetime..

SHA1 (Secure Hash Algorithm-1). Authentication algorithm that generates a 160-bit message digestusing a 160-bit key.

47

transform A transform defines the IPsec action(s) to be taken on the IP data, such as passing the data inclear text, discarding the data, authenticating and encrypting the data using ESP, orauthenticating the data using AH.

48 glossary