Implementation of the IPSec Protocol in Microsoft Windows 2003/XP Environment

1

Implementation of the IPSec Protocol in Microsoft Windows 2003/XP Environment

Paul Szymanski

MCSE

2

Acknowledgments

I would like to thank the following people for taking time to review this document:

Monika Szymanski Boris Taratine Ph.D. Yang Yong Jean Paul Bourget Bénoni MARTIN Guglielmo Alfieri Sean Lewis I am not a graphic artist so I had to use the graphics from the various sources. They are:

Deploying IPSec http://technet.microsoft.com/en-us/library/cc737024.aspx

TCP/IP Guide Charles M. Kozierok http://www.tcpipguide.com

An Illustrated Guide to IPsec by Steve Friedl http://www.unixwiz.net/techtips/iguide-ipsec.html

3

Table of Contents

INTRODUCTION TO ENCRYPTION .................................................................... 5

Preparing confidential information .............................................................................................. 6

Encrypting information ................................................................................................................. 6

Establishing secure link ................................................................................................................ 6

Mutual Authentication ................................................................................................................... 7

Exchange of Keys .......................................................................................................................... 7

Error Free Transmission ............................................................................................................... 7

Decryption ...................................................................................................................................... 8

Accessing decrypted information ................................................................................................ 9

Processing decrypted information .............................................................................................. 9

Replying .......................................................................................................................................... 9

WHAT IS THE IPSEC? ......................................................................................... 9

How does IPSec work? ................................................................................................................ 11

Tunnel Mode vs. Transport Mode .............................................................................................. 12

Core Protocols: Authentication Header (AH) and Encapsulating Security Payload (ESP) .. 17

Authentication Header (AH) ........................................................................................................ 18

Encapsulating Security Payload (ESP) ...................................................................................... 19

Internet Key Exchange ................................................................................................................ 21

Data Encryption Algorithms and Hash Function Algorithms .................................................. 23

Hash Function Algorithms .......................................................................................................... 23

Encryption Algorithms ................................................................................................................ 25

Microsoft Windows XP Pro\2000\2003 IPSec implementation ................................................ 27

Windows IPSec in transport mode ............................................................................................. 29

Tunnel Mode ................................................................................................................................. 31

Default IPSec Policies in Windows 2000/2003/XP Pro ............................................................. 33

4

IPSEC FOR SMTP .............................................................................................. 57

The overview of the IP Security Monitor Console .................................................................... 69

The Main Mode ............................................................................................................................. 72

The Quick Mode ........................................................................................................................... 76

IPSec Statistics ............................................................................................................................ 79

NETWORK MONITOR AND IPSEC .................................................................... 81

APPENDIX .......................................................................................................... 90

5

Introduction to encryption

In order to successfully troubleshoot encryption, one should understand its purpose and the general principles of how it works. Believe it or not, the principles of encryption have not changed since its invention thousands of years ago. Encryption is as old as human civilization and it is a tool that allows scrambling the information, so that only the authorized recipients can access it after applying specific predefined key. The ancient Egyptians used non-standard hieroglyphs to encrypt secret texts.

The ancient Greeks used a “scytale”, which is a wooden baton of a specific size and shape onto which they rolled a leather strap. The sender wrote a message onto the strap and filled the rest of the strap with random letters. The recipient would use the baton of the same size as a key to decrypt the message that was written

by the sender. Obviously, the technology evolved over the thousands of years. The “scytale” was replaced by the Enigma machine and later, by very powerful algorithms that run on computers that encrypt data and generate the security keys. However, the principle of encryption remained the same. First, you must write the message or generate data. Then you encrypt the message or your data. Finally, you provide a key to a recipient and yourself with a set of instructions on how to apply the key so that he/she and you can read what you wrote. You also have to remember that not only the recipient must be able to read the massage and access the data, but that the sender must be able to access his or her own encrypted creation. Human memory fades with time. People often confuse coding with encryption. To code is to replace one word or information with another word or a set of characters or even pictures. For example, a sender could replace a word “cat” with a word “dog”. The sender could also code a word “dog” in a set of the Morse’s characters and represent it

as: -.. --- --. . The same could be done with numbers. For example, we could represent the number “15” in as binary digits 00001111, but even though the information is represented in a different format, it is not encrypted. Encryption is the process of obscuring information to make it unreadable without some special knowledge. It is sometimes referred to as scrambling. Coding is often used along with encryption to ensure the security of the confidential communication. All secure communications, regardless of their nature and the means of communication, follow the same basic steps:

• Preparing confidential information

6

• Encrypting information • Establishing secure link • Secure Authentication • Exchange of security keys • Secure Transmission • Decryption • Accessing decrypted information • Processing decrypted information • Replying

Preparing confidential information First, you must decide what information you would consider as confidential. This step is the most critical and also the most difficult. Not all communications must be encrypted, but at the same time, defining which information must be encrypted could present a lot of challenges. The encryption algorithm can be broken through logical analysis of the unencrypted information. You do not want to encrypt every single message either because you could put a lot of strain on your communication systems and eventually overload them. Encrypting information You have to decide how and what technology you are going to use to encrypt the message. There are many ways to do it and some of them are better than others. The type of encryption depends on the sensitivity of the information. For example, the government may use the most complicated ciphers to encrypt the launching sequence of the nuclear missiles, but you, on the other hand, may use a simple encryption as a toy when sending a romantic message to your loved one or simply place a letter into a secure envelope. You must not forget that you are trying to communicate with a recipient of the message; therefore, the recipient has to be able to decipher the message in a relatively quick and easy fashion. Once you have chosen the appropriate cipher, then you encrypt the information. Establishing secure link Once the information is encrypted, it needs to be delivered to the recipient. Encryption can be compared to secure envelop or a special safe that can be opened only with a key. That key is kept secret, but the secure envelop is exposed to the public. However, if the key is compromised and intercepted, the message could be opened by unauthorized recipient. Special consideration has to be given to the fact that the encrypted message and its security could be also compromised at the point of decryption through eeavesdropping or mishandling of the secure message. For example, you established a secure and encrypted

7

tunnel between two remote networks to send sensitive e-mails. Your tunnel goes from secure gateway A to secure gateway B and the encryption and decryption of the messages is handled by the gateways. However, once the message is decrypted, it is being sent to its final destination in clear text. On route, it could have been picked up by an unauthorized person with relative ease. Your architecture has a serious hole between the gateway that encrypts and decrypts the message and the final destination. Therefore, defining how to design your secure link is very critical to the entire secure communication process. You should define the path the secure message will take to reach the recipients involved in this communication, then you should decide at what point the messaged should be encrypted and decrypted. Business and technical requirements will dictate the approach to be taken. Mutual Authentication Before the secure link is established parties have to be sure they talk to those who they claim they are. The authentication itself also has to be done over a secure channel. Exchange of Keys The keys allow the parties to decrypt encrypted information. Without the key, the information cannot be accessed. The sender and the recipient must validate the authenticity of the key to make sure the messages are not forged. The security keys provide the only protection from unauthorized access to the encrypted message, given a strong cipher is chosen. They must be secured and protected. They are the keys to your safe. In the spy movies, the agents not only exchange passwords (secure authentication), but they also exchange prearranged tokens, such us ripped out book pages, money or playing cards, before they continue exchanging the information. The reason why they do it is very simple. The passwords can be guessed or compromised, but it is a lot more difficult to fake prearranged tokens. The same principle is used in secure communication between computers and computer networks. But in the case of IPSec, the symmetrical keys are being exchanged via secure channel (after Diffie-Hellman public key exchange) hence they cannot be compromised, which makes IPSec a very secure protocol. Error Free Transmission Any encrypted communication requires error free transmission. Distortion and any noise during the transmission could have negative impact not only on encrypted messaged itself, but also on authentication negotiations required by IPSec. For example, you can securely transmit information over the radio waves. that the waves are broadcasted by antennas the open air is the medium through

8

which they travel. The link between point A and point B has inherent security problems because the airspace is a public domain. The radio waves can be picked up by anyone. The radio waves are susceptible to all kinds of atmospheric disturbances that are beyond your control. The transmission of the data could be secured and encrypted, but because of the bad weather it may not reach its destination. The owners of the Satellite TV dish are very familiar with the service outages during the weather storms. The TV signal from the satellite is encrypted, it travels through the public airspace but the storms can block it all together. Another example are the cell phones. You may not get appropriate coverage and you will not get an encrypted signal. In both cases, you the disturbances during the transmission affect the quality of the link between the two points. Data can and is transmitted securely over public broadcast systems. Public broadcast systems can be secured. The encrypted data must be delivered without errors. Errors caused by inter Decryption Decryption is the process of getting clear text, data or other information from the cipher text If key is secure and algorithm is strong, it is assumed that the only way for the unauthorized person to gain access to the encrypted message is through a brute force. This is tedious and time consuming process which is compounded by key lengths. Decryption of the data is as important and serious as the encryption process. Sloppy decryption procedures can cause errors or jeopardize the entire secure communication system. During the Polish-Soviet War of 1920, Polish cryptologists were able to obtain some of the keys to the Soviet ciphers through eavesdropping of their radio traffic. In some cases, Polish cryptologists pretended to be Soviet radio operators and asked them to repeat the key portions of the encrypted information in clear text. The obtained information allowed Poles to decrypt those messages, and with the combination of crypto analysis, they were able to break all Soviet ciphers. The lack of training and sloppiness of the Soviet soldiers responsible for encrypting and decrypting the messages cost the Red Army the war. The Soviets had not learned about this breech until the end of the World War II. Let me stress this point again, the encrypted message can be accessed by anyone once the keys that are used to decrypt it are compromised. They can be compromised during the faulty decryption process. Sloppy encryption practices by the Germans during WW II helped Allies to decrypt messages encrypted by Enigma machines. The operators of the Enigma machines quite often forgot to reset the rotors or they did not follow the established reset policies. With this in mind, you should carefully define where the encryption and decryption takes place and how it affects the encrypted message.

9

Accessing decrypted information Once the data is decrypted, it is ready to be accessed. But since you bothered to encrypt this data in the first place, it must contain sensitive information. Therefore, it must be securely accessed after decryption; otherwise the whole process would make no sense. You should be aware how, where, and by what means you will access decrypted information. You should ask yourself if the environment in which you access it, will not undermine the entire process you have just gone through. For example, you get a very sensitive database file. You decrypt it and decide to access it by mounting the database on server that would allow unauthorized personal to see the entire content of the sensitive database. Consequently, your elaborate security process has just become a mirage. Processing decrypted information You should also put a lot of thought into actions that you will use to process decrypted information. These procedures should be written down and analyzed before you implement them. You should ask yourself who should process the data, how this data is going to be processed which servers you will use to process it and who is going to ensure that the processing of sensitive information is secure. These questions sound easy, but once you start answering them, you will find out that the answers could be very tricky. Replying Since the communication is a process of exchanging information, you will most likely reply to the encrypted information. There is no point in encrypting anything, if you send a reply in clear text. You must always keep in mind that the reason why you encrypt the data is to protect its content from unauthorized access. Yes, it is an obvious statement, but a lot of people go through the trouble of encrypting the data and securing the links, but at the end, they dump sensitive information on the server so that everyone can access it.

What is the IPSec? The IPSec is a suite of protocols for securing one or more Internet Protocol (IP) paths between a pair of hosts or security gateways through authenticating and or encrypting each IP packet in a data stream. It is designed to provide interoperable, cryptographically-based security for IPv4 and IPv6. Therefore, IPSec includes protocols for cryptographic key establishment.1 1 Request for Comments: 2401 S. Kent & R. Atkinson, the Internet Society (1998).

10

The IPSec protocol operates on the level 3 of the OSI model and on the layer 2 of the TCP/IP stack, which is very important to understand when it comes to implementing and troubleshooting IPSec.

The Internet Protocol was not designed to provide any type of security. When the IP was developed, only chosen few had access to what we call now the Internet. Now, the Internet is an integral part of our lives. Most of us must stay connected all the time and we get very cranky when we can’t access our e-mails for more than an hour. We also transmit a lot of sensitive information, which we do not want to share with the rest of the world. For example, we may not want to share our banking information with fellow human beings on the other side of the globe. He or she or they may not take us to cleaners when they access our bank accounts, but they may have a good laugh when they see a negative balance on our checking account. Therefore, the scientists developed various security standards such us TLS, SSL and IPSec to protect the data traffic from prying eyes. The IPSec stands for Internet Protocol Security. As I have mentioned earlier, it operates on the Network layer of the OSI model and because of this, it can provide security for protocols that operate on higher layers, such us TCP, UDP, BGP, FTP, Telnet or SMTP.

11

(Please note that “non secure” IP also operates on the layer 3 of the OSI model.) Among other things, you can configure your Microsoft Exchange 2003 server or later to use IPSec encryption to send and receive e-mails. Microsoft Windows Server 2000, 2003 and 2008 support IPSec. IPSec is also supported by Windows XP and Vista. It is also supported by Unix operating system and MAC OS as well. In other words, the IPSec protocol is used by many vendors on many platforms. The IPSec provides:

• Access control • Connectionless integrity • Data origin authentication • Rejection of replayed packets (a form of partial sequence integrity a.k.a

Replay Attack) • Confidentiality through encryption • Limited traffic flow confidentiality • Negotiations of the IP compression

How does IPSec work? First of all, IPSec is a complex suite of protocols and is not easy to implement because the human factor plays an important in its configuration. It is a very powerful and quite flexible protocol stack. The implementation requires a lot of planning and testing. Administrators must put a lot of thought and energy in order to configure it properly. Because of its complexity and flexibility, IPSec is not a user friendly protocol. The IPSec protocol stack is comprised of the two core IPSec protocols that you must be very familiar with:

• Authentication Header (AH) • Encapsulating Security Payload (ESP)

There are also three support components:

• Internet Key Exchange (IKE) • Security Policies/Security Associations • Encryption/Hashing algorithms

12

You should be familiar with the two modes in which IPSec can operate:

• Tunnel mode • Transport mode

And you should understand three most common implementation methods of IPSec:

• Integrated architecture • Bump-in-the stack • Bump-in-the wire

How you decide to implement the IPSec implementation will depend on what you want to protect and how you want to do it. It is also important to read and understand the RFC 2401 document because it specifies the base architecture for IPSec compliant systems. Since I want to present and explain how IPSec functions within Windows 2000/2003 environment, I will not get into details of each of the key concepts listed above. However, I will try to briefly explain them and reference corresponding RFC articles for more detailed information. Tunnel Mode vs. Transport Mode The tunnel mode is a common choice for VPN implementations and it is used to provide secure connection between two networks. For example, you may want to establish IPSec connectivity with a remote office over public links. In this case, you most likely have chosen the tunnel mode, which encapsulates a complete IP datagram, thus forming a virtual tunnel between IPSec-capable devices, such us firewalls or other security gateways.

13

The transport mode, on the other hand, may be implemented within a LAN between two servers or workstations, where VPN tunneling is not necessary since the LAN resides behind the firewall and is not exposed to the public. It is also called the end-to-end communication. The implementation of IPSec in the transport mode protects data from hacking attacks, which can occur within LANs. You should keep in mind that most hacking attacks are conducted by employees. Therefore, you may want to encrypt sensitive data such us payroll from crafty personnel, who want to test the latest version of free sniffing software that was downloaded from the Internet.

In the transport mode, the IPSec Authentication Header protects both the payload and the IP header fields and inserts a new header between the original IP header and the payload. The transport mode requires IPSec to be integrated into IP, because the AH/ESP must be applied during original packaging of the datagram, which occurs on the transport layer of the OSI model. This type of datagram packaging requires the end-to-end security with hosts that run IPSec directly. In other words, an IPSec connection will never happen, if you decide to establish connection between two networks, over public lines, separated by firewalls that run NAT.

In the tunnel mode, the whole IP datagram is encapsulated within the Authentication Header and new IP header. You basically put the IP into the IPSec envelope. This encapsulation forms a virtual tunnel between IPSec capable devices such as firewalls. The tunnel mode is a common choice for VPN implementations, which create a virtual tunnel through public lines by encapsulating the IP datagram.

14

15

There are two methods used to achieve the IPSec encapsulation in a tunnel mode: bump-in-the-stack and bump-in-the-wire. The bump-in-the-stack refers to the TCP/IP model. As we all know, the TCP/IP model has five layers, unlike the OSI model which has seven. The IP protocol resides on the layer 2, the Internet Layer, in the TCP/IP stack, whereas in the OSI model the IP resides on the layer 3, the Network Layer. In the bump-in-the-stack encapsulation, the IP is intercepted by the IPSec as it passes down from the upper layers. The IPSec encapsulates the datagram before it is being sent on its way through the Network Interface Layer of the TCP/IP stack, hence creating a “bump” in the stack. The bump-in-the-stack method of encapsulation is usually used with the hosts that use IPv4.

16

The bump-in-the-wire method depends on two hardware devices that provide IPSec encapsulation. Usually the encapsulation of the IP packet is done by the firewall. The IPSec device intercepts the IP packet and encapsulates it as the IPSec packet. It then sends it over public lines to another IPSec device, which strips the packet from the IPSec envelope and routes it to its destination.

The differences between these two modes have significant impact on your entire network architecture and the implementation of the IPSec itself. You must not only decide which mode of the IPSec to use, but also which version of the IP protocol you are going to deal with. The version of the IP protocol is very important. Not all networks run IPv6 and there are significant differences between different versions of the IP protocol. For example, The IPv4 does not have any built in security features. It was not designed to be a secured protocol. However the IPv6 from its conception was designed with the IP security in mind. Therefore, the exact way the headers are

17

arranged in the IPSec datagram in both transport and tunnel modes depends on which version of the IP you use. In addition, the placement of the header will depend on the version of the IPSec protocol and it will vary from AH to ESP. You could apply AH and ESP to the same datagram and in this case, the AH header will always appear before ESP header.

The understanding of the location of the header plays an important role when you try to troubleshoot problems with the IPSec. Latter, I will illustrate the authentication process of IPSec on Windows 2003. Core Protocols: Authentication Header (AH) and Encapsulating Security Payload (ESP) The Authentication Header (AH) and Encapsulation Security Payload (ESP) are core protocols of the IPSec. They are responsible for encoding the payload, enforcing security, and negotiating the encryption algorithm that is used to encrypt and decrypt the information. These two protocols do all the work in the IPSec stack. However, the IPSec allows you to choose which one of these

18

protocols to use. They can be applied alone, in combination with each other, or in a nested fashion in order to achieve the desired level of security for the IPv4 and IPv6. AH and ESP protocols support both modes of operation: the transport mode and tunnel mode. In the transport mode, the protocols provide protection primarily for upper layer protocols; in the tunnel mode, the protocols are applied to tunneled IP packets.2 Your IPSec implementation will depend on the way you choose to implement these two protocols. They will play a key role in establishing secure communication with a remote network. You must keep in mind that the remote site’s IPSec configuration must match your site’s IPSec configuration; otherwise you will spend a lot of time to troubleshoot IPSec connectivity issues. You should also be aware that AH and ESP do not specify the encryption mechanism and because of this flexibility, they can work with a variety of cryptographic and authentication algorithms. This is both a blessing and a curse. On one hand, it makes the IPSec very secure and hard to pick, but on the other hand, you may have a configuration and support nightmare, especially when the IPSec configurations and algorithms do not match. For Example, Microsoft XP Professional and Windows 2000 and 2003 use Kerberos authentication for the IPSec as a default setting. These operating systems accept unsecured traffic by default and respond using IPSec. To make things even more complicated, by default, these operating systems do not support Authentication Headers in conjunction with the Encapsulating Security Payload to encrypt the data payload. They do use ESP by default when replaying and encrypt the payload using 3DES or DES encryption algorithms. The integrity of ESP is provided by hash algorithms SHA-1 and MD5. The Authentication Header protocol is an optional feature that must be turned on manually in Windows environment. Windows 2000 requires the Service Pack 2 for AH to be enabled. Windows XP home edition does not offer the AH protocol at all. In other words, Windows operating systems require administrators to customize settings of the IPSec in order to achieve desired levels of data security. Later in the article, I will explain the IPSec options available in Windows XP, 2000, and 2003 operating systems. Authentication Header (AH) The Authentication Header protocol provides authentication services for the IPSec. No kidding, someone may say. But putting the sarcasm aside, AH plays an important role during the authentication negotiations. It allows the recipient of the data to verify the authenticity of the data source. The recipient must make sure that it establishes a session with an authorized and valid source before it 2 Request for Comments: 2401 S. Kent & R. Atkinson, the Internet Society (1998).

19

negotiates encryption algorithms. The AH also allows the recipient to verify that the devices used in establishing a secure link have not changed any of the data in the datagram. In other words, it protects against the recipient and sender from replay attacks. Let me remind you again that in the transport mode, the IPSec Authentication Header protects both the payload and the IP header fields and inserts a new header between the original IP header and the payload. In the tunnel mode, the entire IP packet is encapsulated within the Authentication Header and new IP header. However, AH implemented alone does not encrypt the data payload. AH implemented alone does not support any encryption algorithms. If you want to encrypt the data payload, you have to use a second core IPSec protocol called Encapsulation Security Payload (ESP). A replay attack is a form of network attack, in which a valid data transmission is maliciously or fraudulently repeated or delayed in order to intercept and decrypt it. Suppose you are an intelligence agent and you want to verify the identity of the currier that brought you an important message. You ask the currier to provide a password. Meanwhile, a counterintelligence officer is eavesdropping on your conversation and intercepts the identification password. After you finish your communication with the currier, the counter intelligence officer contacts you and poses as a valid currier. When asked to prove his/her identity, he/she provides the intercepted password, which you must accept since you have no other ways to verify the currier’s true identity. The protection against replay attacks prevents IPSec from establishing connections with the firewalls that have the Network Address Translations (NAT) enabled. The AH cannot verify the identity (IP address) of the recipient or the sender, therefore, it will not establish the IPSec connection. Let’s assume for a moment that the IP address from which the data has originated is 192.168.32.33 and it has been translated by NAT to 172.31.16.33. During the NAT translation from one IP Address to another, the header of the packet is changed by the NAT. The AH mechanism will detect the change of the header and cause the packet to be discarded, as the calculation of the data integrity will indicate that the packet has been altered. When you plan an implementation of the IPSec, you must identify any devices that perform NAT translations on your network and decide how you want the IPSec to operate. Otherwise, you will spend countless hours troubleshooting IPSec authentication issues. Lastly, you should also keep in mind that the AH in combination with the ESP requires higher end processors, and the processes associated with the AH and ESP are memory intensive. Encapsulating Security Payload (ESP)

20

Authentication Header protocol verifies the authenticity of the source, but it does not encrypt the data payload. The AH has no encryption capabilities. Therefore, you need another protocol that will allow you to choose the desired encryption and encrypt your data. The ESP protocol provides the functions of data confidentiality through:

• encryption • data origin authentication • connectionless integrity • an anti-replay service • limited traffic flow confidentiality

The ESP protocol is designed to provide a mix of security services in the IPv4 and IPv6. Since the ESP provides data origin authentication and encryption, it can be applied alone. But you may decide to use the AH in addition to the ESP authentication. Because the ESP has its own authentication mechanism, the Authentication Header is an optional protocol in the IPSec implementation. If you want to use the AH in the MS Windows environment, you have to configure it manually. In the ESP the data origin authentication and connectionless integrity are joint services; therefore, they are referred as “authentication”. The authentication is an optional service in the ESP, which can be configured to work in conjunction with encryption that provides confidentiality services for data payload. It is up to you how you want configure the ESP. You should remember though that the anti-replay service in the ESP requires data origin authentication. Furthermore, the traffic flow confidentiality requires a selection of the tunnel mode. It is most effective, when implemented at a security gateway such as a firewall, where traffic aggregation may be able to mask the true source-destination patterns. Although both confidentiality and authentication are optional, at least one of them must be selected for the ESP to work properly.3 As I have mentioned before, the ESP provides confidentiality by encrypting the IP data. The encryption algorithm encrypts data and transmits it to the destination, where it is decrypted with the same algorithm. Therefore, the configuration of the source and the destination must match. The detailed and well explained information about the ESP encapsulation and encryption can be found in the “The TCP/IP Guide: A Comprehensive, Illustrated Internet Protocols Reference” by Charles Kozierok. It is a must-read for anyone who considers the IPSec implementation. 3 RFC 2406 - IP Encapsulating Security Payload (ESP) S. Kent & R. Atkinson the Internet Society; November 1998

21

Internet Key Exchange This is all nice, but the AH and ESP must somehow negotiate the secure connection. They also have to agree on the type of encryption they will use to encrypt and decrypt the data. All this has to be done in such a way that the keys required to decrypt the payload will not be compromised. The handshake between the two devices must be secured and “secret”. You must remember that every secured communication can be eavesdropped. Every encryption can be compromised or broken. Your goal is to make it very difficult and time consuming for the intruder to hack your defenses. The primary protocol that is used by the IPSec to negotiate the exchange of the “secret” authentication procedures is called the Internet Key Exchange (IKE). The Internet Key Exchange protocol has been defined in the RFC 2409 document, which explains the existence of encryption algorithms DES, 3DES, and hash algorithms SHA1, and MD5 in the MS Windows IPSec implementation. Think of the Internet Key Exchange protocol as a manager, who first of all coordinates Key Exchange for further use in secure channel to exchange main symmetric cipher key. Further, IKE automatically negotiates the IPSec Security Associations (SA) and enables secure communications without the time consuming and cumbersome manual pre-configuration. The IKE provides the following services to the IPSec:

• Eliminates the need to manually specify all the IPSec security parameters in the crypto maps at both peers.

• Allows you to specify a specific timeframe for the IPSec Security Association (SA).

• Allows encryption keys to change during IPSec sessions. • Allows IPSec to provide anti-replay services. • Permits Certification Authority (CA) support for a manageable, scalable

IPSec implementation. • Allows dynamic authentication of peers.

The IKE is a hybrid protocol. It basically means that the IKE uses features of three protocols to manage the IPSec authentication and the exchange of cryptographic keys between the peers involved in the IPSec communication. The protocols used by IKE are:

• ISAKMP • OAKLEY • SKEME

22

ISAKMP (Internet Security and Association and Key Management Protocol) is a protocol responsible for establishing the Security Associations (SA) and cryptographic keys in an Internet environment. It is described in detail in the RFC 2408. The ISAKMP defines procedures and packet formats necessary to establish, negotiate, modify, and delete Security Associations.

Security Associations contain all the information required for execution of various network security services, such as header authentication and payload encapsulation, transport or application layer services, or self-protection of negotiation traffic. ISAKMP defines payloads for exchanging key generation and authentication data. These formats provide a consistent framework for transferring key and authentication data, which is independent of the key generation technique, encryption algorithm and authentication mechanism.

ISAKMP protects form:

• Denial of Service attacks • Connection Hijacking • Man-in-the-Middle Attacks

You should keep in mind that there may be many different key exchange protocols, each with different security properties. However, a common framework is required for agreeing to the format of SA attributes, negotiating, modifying, and deleting SAs. ISAKMP provides such a framework.

The type of encryption used during the IPSec session is determined by the IPSec Security Association, or SA. The ISAKMP protocol establishes SA and negotiates cryptographic keys. A Security Association (SA) is a combination of a destination address, a security protocol, and a unique identification value, called a Security Parameters Index (SPI). The MS Windows 2000/2003 and XP Pro operating systems offer the following encryption algorithms:

• Data Encryption Standard (DES) with a single 56-bit key, which is designed for international use and adheres to he United States export encryption laws.

• Triple DES (3DES), which uses three 56-bit keys and is designed for high-security environments in North America.

ISAKMP can be implemented over any transport protocol or over IP itself. Implementations MUST include send and receive capability for the ISAKMP using the User Datagram Protocol (UDP) on the port 500. The UDP Port 500 has been assigned to ISAKMP by the Internet Assigned Numbers Authority (IANA).

23

Implementations may additionally support ISAKMP over other transport protocols or over IP itself.4 OAKLEY: is a protocol which provides mechanism to exchange cryptographic keys between authenticated parties across insecure links. It uses Diffie-Hellman key exchange algorithm and is described by the RFC 2412. OAKLY allows two systems to agree on a shared value without requiring encryption. Because OAKLEY is a generic key exchange protocol, and because the keys it generates might be used for encrypting data with a long privacy lifetime, 20 years or more, it is important that the algorithms underlying the protocol be able to ensure the security of the keys for that period of time, based on the best prediction capabilities available for seeing into the mathematical future.5

SKEME: (Secure Key Exchange Mechanism) is a compact protocol that supports a variety of realistic scenarios and security models over the Internet. It provides tradeoffs between security and performance as required by the different scenarios without incurring in unnecessary system complexity. The protocol supports key exchange based on the public key, key distribution centers, or manual installation, and provides for fast and secure key refreshment. In addition, SKEME selectively provides perfect forward secrecy, allows for replaceability and negotiation of the underlying cryptographic primitives, and addresses privacy issues as anonymity and repudiatability.

Data Encryption Algorithms and Hash Function Algorithms Microsoft Windows 2003, XP Pro and Windows 2000 allow encryption of the payload using 3DES or DES algorithms by default. The integrity of the ESP is provided by SHA1 and MD5 hash function algorithms. In other words, the data is encrypted by DES or 3DES and then, before the already encrypted information is send over to another IPSec device, the data is chopped to pieces and coded (hashed) by SHA1 or MD5 hash algorithms. Hash Function Algorithms Hashing is a function that takes one value, lets say X, and transforms it into a fixed-size string. The end result is called the hash value. For example, after applying a hash function, a word “computer monitor is broken” could change its form to a hash value “DHG07X”. In most cases, the hash value is shorter than the value X, which in this example, is represented in a phrase “computer monitor is broken”. 4 Request for Comments: 2408 D. Maughan, National Security Agency; M. Schertler; M. Schneider, National Security Agency; J. Turner, RABA Technologies, Inc. November 1998 5 Request for Comments: 2412 H. Orman, Department of Computer Science, University of Arizona, November 1998

24

The hash value uniquely represents the longer message, document or other forms of data. This shorter version of the original input is called the message digest or digital fingerprint. Because the hash values are shorter than the original value X, they are considered to be faster than encryption algorithms. Consequently, the hashed functions are used to compute the digital signatures or to perform integrity checks on some documents. They are also used to hash passwords on computers, firewalls, and routers. A message digest can be transmitted over public links without revealing the content of the hashed data but susceptible for brute force if the hashed is a short message; therefore, it is not wise to send hashed passwords over the Internet. This is important in digital timestamping where, using hash functions, one can get a document timestamped without revealing its contents to the timestamping service. This function is used by SSL and TLS cryptographic protocols. The basic requirements for a cryptographic hash function are as follows:

• The input can be of any length. • The output has a fixed length. • H(x) is relatively easy to compute for any given x. • H(x) is one-way. • H(x) is collision-free.6

In various standards and applications, the two most-commonly used hash functions are MD5 and SHA-1. However a cryptographic hash function is considered insecure if either of the following is computationally feasible:

• Finding a (previously unseen) message that matches a given digest • Finding "collisions", wherein two different messages have the same

message digest.

An attacker who can do either of these things might, for example, use them to substitute an unauthorized message for an authorized one and because of these reasons, hashed functions are used in combination with encryption algorithms such us DES and 3DES. SHA-1 SHA-1 stands for the Secure Hash Algorithm 1. There was an SHA-0 algorithm, but the research showed that it was relatively easy to break. The SHA-1 combines five cryptographic hash functions designed by the national Security Agency, which were published by the National Institute of Standards and Technology (NIST) as the U.S. Federal Information Processing Standard. The 6 Crypto FAQ: Chapter 2 Cryptography: 2.1 Cryptographic Tools, RSA Laboratories' Frequently Asked Questions About Today's Cryptography, Version 4.1, http://www.rsa.com/rsalabs/node.asp?id=2152

25

research showed that there is a way to compromise the SHA-1 algorithm and the NIST has published four additional hash functions in the SHA family, each with longer digests. They are collectively known as SHA-2. MD5 MD5 stands for the Message-Digest algorithm 5. It was developed by Professor Ronald Rivest from MIT. He developed this algorithm to replace an earlier version of message-digest called MD4. The MD5 algorithm is describe in detail in the RFC 1321. The MD5 algorithm is intended for digital signature applications, where a large file must be "compressed" in a secure manner before being signed with a private (secret) key under a public-key cryptosystem such as RSA algorithm. MD5 algorithm is usually used to verify that the file that you sent or downloaded from a web site has not been damaged or altered during electronic transfers, and also to verify that the file you are getting has not been tampered with. If you have ever downloaded Red Hat Linux software, you probably run into such digest. You most likely compared the digest of the download with the one listed by the Red Hat to make sure the compressed files did not get corrupted during the download. The MD5 Message Digest Algorithm takes an input (in this case, a file) and produces a 128-bit (or 16 byte) output, which can be represented as a string of 32 hexadecimal values. This output is a 'fingerprint' or 'message digest' for the file. It is assumed that it is computationally infeasible to produce two messages having the same message digest, or to produce any message having a given pre-specified target message digest. However, MD5 has a known collision weakness. In 2004 Chinese researchers Xiaoyun Wang, Dengguo Feng, Xuejia Lai, and Hongbo Yu were able to crack MD5 digest through collisions. Currently the MD5 "rainbow tables" are easily accessible online, and are used to reverse many MD5 hashes into strings that collide with the original input. Rainbow Tables help in recovery of lost passwords. MD5 should not be considered a safe algorithm. It should be used along with other encryption algorithms to provide protection against dictionary attacks. Encryption Algorithms I will provide just a brief description of two encryption algorithms used by the Windows operating system. I will not get into the details of DES and 3DES algorithms, because the information about them is widely available on the Internet and in the libraries.

26

DES stands for the Digital Encryption Standard. The algorithm is described in the Federal Information Processing Standard (FIPS) document FIPS 46-37. DES was selected as an official Federal Information Processing standard for the US in 1976 to protect non-classified information. There was a big controversy about the alleged tempering and modifications of this algorithm by the National Security Agency. The introduction of DES is considered to have been a catalyst for the academic study of cryptography, particularly of methods to crack block ciphers. DES belongs to, what scientists define. as block cipher algorithms. It takes a fixed-length string of plaintext bits and transforms it through a series of operations into another ciphertext bitstring of the same length. The block size of the DES algorithm is 64 bits. DES uses the secure key to customize the transformation, so that the decryption of the data can only be performed only by those who possess the secure key that was used to encrypt the data. This secure key consists of 64 bits, but only 56 bits are used by the algorithm. The remaining 8 bits are used only for checking the parity, and are discarded at the end. Therefore, the effective key length is 56 bits. DES is considered as an insecure encryption algorithm for many applications, because of its 56-bit key size, which is considered too small to protect the data. DES keys have been broken in less than 24 hours. There are devices called DES Crackers that are dedicated to break DES keys. You should be aware of the DES weakness during the planning and implementation process of the IPSec in order to avoid nasty surprises. The DES algorithm is still widely used by many applications. 3DES, or if you like Triple DES, is another block cipher, which was derived from the DES cipher by using it three times; hence its name. The first encryption is encrypted with the second key, and the resulting cipher text is again encrypted with the third key. The 3DES algorithm was introduced in 1978, when it was discovered that the DES 56-bit key did not protect from the brute force attacks. The 3DES encryption is a simple way to increase the size of the key without switching to a new algorithm. The use of three steps is essential in prevention of meet-in-the-middle attacks that are effective against the double DES encryption. The 3DES is a combination of three different DES keys. It has a length of 168 bits (3 times 56 = 168). With parity, the 3DES key can be as long as 192 bits. Due to the meet-in-the-middle attack, the effective security that the 3DES provides is only 112 bits (2 times 56 = 112). You can use the 3DES in a variety of applications, including:

• Electronic financial transactions • Secure communications

7 FIPS 46-3, http://csrc.nist.gov/publications/fips/fips46-3/fips46-3.pdf

27

• Secure video surveillance systems • Encrypted data storage

The brute force attacks could crack the 3DES encryption, but they require a know-how, money and time. You should also keep in mind that the attack that could jeopardize the 3DES encryption may come from within the organization. The 3DES requires a lot more processing power than the DES algorithm. It could significantly slow down the computing power of the server. It would also put a lot of processing stress on security gateways, such as firewalls. You should make sure that the equipment can handle processing required by the 3DES algorithm. Currently, the 3DES is being replaced by the Advanced Encryption Standard (AES) algorithm, which is also a block cipher, but is a lot faster and more secure than the 3DES. There are four modes of DES that are applicable to any block cipher of any block size. Those standard modes are the following: Electronic Code Book (ECB), Cipher Block Chaining (CBC), Cipher Feedback (CF), and Output Feedback (OF). Microsoft Windows XP Pro\2000\2003 IPSec implementation The Microsoft Windows implementation of the IPSec provides security encapsulation of the IP traffic using IPSec both in a transport mode and in a tunnel mode.

The Microsoft Windows IPSec is also capable of securing remote access from a client-to-gateway over the Internet using the Layer Two Tunneling Protocol (L2TP).

In a pure IPSec tunnel mode or L2TP tunnel mode secured by the IPSec, the Windows implementation of the IPSec can provide a secure link between two security gateways across outsourced private wide area networks. However, you may run into difficulties, if you try to connect two remote networks over public links using the IPSec tunnel mode. You may have to relay on your firewall to provide the IPSec tunnel mode connection between these two networks over public links. You should also keep in mind that the Microsoft IPSec implementation in a tunnel mode is not designed to be used for a virtual private network (VPN) remote access.

In a transport mode, the IPSec provides security for IP connections in the following end-to-end security scenarios:

• client-to-server • server-to-server

28

• client-to-client.

Microsoft integrated the IPSec protocol with Windows 2000\2003 domains and the Active Directory services. The Active Directory delivers policy-based, directory-enabled networking using the Group Policy to provide IPSec policy assignment and distribution to Windows 2000\2003\XP Pro computers that are members of an Active Directory domain. Group Policies allow the network administrators to enable the IPSec for specific workstations, servers, and entire Organization Units.

The ability to assign IPSec policies through the Group Policies has a significant impact on grouping computers in the Active Directory’s organizational units (OU’s), and on how and where you secure computers on your network. For example, if the IPSec is required to encrypt all traffic only to a trusted domain member computers, but there are weak controls on how computers can join the domain, the IPSec-protected traffic might be exposed to a trusted man-in-the-middle attack. Similarly, if attackers can join their computers to the domain, they might obtain trusted, encrypted access to IPSec-protected servers. If computers cannot be properly authenticated, the IPSec does not provide any security.

The IPSec in Windows environment does not require Active Directory services. You can deploy it on Windows 2000\2003 servers and computers that run the Windows XP Professional, which do not belong to any domain and are just stand alone workstations or servers. The IPSec features offered by different versions of the Windows operating system vary significantly. Some features offered by the Windows Server 2003 may not be available in Windows XP Pro or Windows 2000. The Microsoft Windows Server 2003 enhanced the IPSec implementation by offering users additional IPSec services:

• Filters update IP configuration of IPSec partners • NAT traversal support • Netsh command-line support for IPSec • Resultant Set of Policy (RSoP) support, which helps to analyze IPSec

policy assignments • The IPSec Monitor snap-in • Stateful filtering of network traffic • Persistent IPSec policy • 2048-bit Diffie-Hellman keys, which provide a very strong cryptographic

key generation to maximize the 3DES encryption strength8

8 Windows 2003 Server IPSec Concepts http://technet2.microsoft.com/windowsserver/en/library/6ea5e4b9-13a3-48bc-bb2e-0f801501c5f51033.mspx?mfr=true

29

I would like to stress that using the Authentication Header (AH) protocol across network address translators is not supported by the IPSec NAT traversal (NAT-T). Some applications might not work when their traffic is first protected with the IPSec ESP and then passed through a network address translator. The Internet Key Exchange (IKE) protocol automatically detects network address translators and uses the UDP ESP encapsulation to send all user data by using a UDP port 4500. You should also remember that remote management and monitoring of the IPSec is supported only between computers running the same version of the Windows operating system. Microsoft recommends that the administrators should use the Remote Desktop Connection for remote management and monitoring of IPSec. Windows IPSec in transport mode In a transport mode, the IPSec can be used to filter specific traffic based on a source and destination address combinations and specific protocols and specific ports. This is called the IPSec Packet Filtering.

In this scenario, an IPSec filter acts as a firewall. The IPSec Packet Filtering can be used to quarantine servers that have become infected with the virus by blocking specific well-known ports. The infected servers are still part of the network, but they will no longer pose a threat to other servers and workstations. This is especially useful when your network spreads across several remote sites and one of the sites or specific remote server must be walled off to contain the virus outbreak. The infected server or site is contained through IPSec filters. The troubled ports are blocked, but the administrator still has the ability to access the site and its servers remotely to troubleshoot and fix the problems caused by the virus. The IPSec Packet Filtering can be also used to block an unauthorized access to specific ports on the servers, thus making it more difficult for the intruder to attack them. The IPSec Packet Filtering allows the administrator to control exactly the type of communication that is allowed between systems, thus preventing an unauthorized access. You should keep in mind that most hacking attacks are conducted by creative internal users.

30

In the transport mode, the IPSec can be configured to provide an end-to-end security between servers or workstations. In this case, the IPSec establishes a secure link from a unicast source IP address to a unicast destination IP address.

In this scenario, only trusted and authorized computers have access to a secured server. The administrator can use the IPSec to grant access to some or all applications and services running on the server. The administrator can either encrypt all the traffic between computers or establish security simply based on the authentication of the application traffic.

You can use Kerberos V5 or a preshared authentication key to authenticate computers with each other. However, a preshared key authentication in the Windows IPSec implementation could be compromised, and it is not a good idea. The preshared authentication key is like a password. It must be the same on both ends and is case sensitive. However, it can be guessed if it is too simple or could be retrieved from the unsecured system. In both cases, the preshared authentication key makes you system susceptible to the man-in-the-middle

31

attack. Preshared keys are stored in plaintext in the registry and Active Directory, They are stored in a readable hexadecimal format. In the end-to-end configuration of IPSec, the sending and receiving computers must be configured and running the IPSec, because the end-to-end configuration is essentially a secure virtual tunnel between two nodes. But you should not confuse this configuration with IPSec in the tunnel mode. Please note that if there is a firewall that separates two IPSec peers, it must be configured to forward the IPSec traffic on the UDP source and destination port 500, IP protocol 50 (ESP), or IP protocol 51 (AH). If the computers use the network address translation, they must both support the IETF IPSec NAT-T specification draft version 2. A firewall must also be configured to forward the traffic on the UDP source and destination port 4500. Tunnel Mode As I have mentioned earlier, the IPSec in the tunnel mode allows connecting two networks with each other. We can distinguish three types of configuration of IPSec in a tunnel mode:

• Gateway-to-gateway

• End-to-end, where one of the endpoints does not support the IPSec

32

• End-to-end between two computers with a third party firewall or network

intrusion detection system, which decrypts the traffic for inspection and then re-encrypts it again and forwards it to the destination.

33

Default IPSec Policies in Windows 2000/2003/XP Pro Now, we are ready to examine the configuration options offered by the Microsoft implementation of IPSec. But first you have to access the IPSec policies. The IPSec policies are accessed through the Microsoft Management Console, or MMC for short. You basically type mmc at the Run command to launch it. You will get the following screen:

Then click File and select Add/Remove Snap-in

Click the Add button to select two stand alone snap-ins: IP Security Monitor and IP Security Policy Management.

34

The IP Security Monitor is a useful tool to monitor the IPSec authentication and to troubleshoot problems with the IPSec. I will talk about it later, but for now lets go to the IP Security Policy Management console.

As you can see, there are three default IPSec policies:

• Client (Response Only) • Secure Server (Require Security) • Server (request Security)

In addition, you can create your custom policies as well. You just need to highlight the IP Security Policies on Local Computer and select the Create IP

35

Security Policy from the drop down menu. The configuration wizard will guide you through the set up process. Client (Response Only) is a default policy and contains only one rule called <Dynamic> by default. It is using Kerberos encryption for authentication and by default does not support tunneling mode

The <Dynamic> filter in the Client mode secures communication only upon request by another computer hence the Response Only comment in the Filter Action. If there is no request for IPSec, then the traffic will not be encrypted. You should also keep in mind that this policy does not attempt to negotiate security for any other traffic. It is just a client mode. You can add a new custom rule or modify the existing one, Once again, unlike the server mode a client mode will only respond to requests for IPSec communication. The existing rule supports DES and 3DES algorithms for data encryption, and SHA1 and MD5 algorithms for hashing authentication headers. You can also add, remove or modify these settings.

36

You can also arrange the order, in which the algorithms are negotiated as well as the security methods for each of them. Once you highlight the algorithm and click the Edit button, you will get the menu with three choices:

• Encryption and Integrity • Integrity only • Custom

This is the place where you modify the headers of the IPSec datagram. As I have mentioned earlier, the IPSec is a protocol stack and the core IPSec protocols include the Authentication Header (AH) and Encapsulating Security Payload (ESP). They can be used either together or separately.

37

If you select an option Encryption and Integrity, the Authentication Header protocol (AH) will provide the authentication services and Encapsulation Security Payload protocol (ESP) will provide encryption encapsulation services. If you choose the Integrity only, the ESP protocol will encapsulate data payload and AH will not be added to the IPSec header. If you choose the Custom option, you will get additional choices:

You will be able to choose whether or not to use the AH, session key settings and integrity and encryption algorithms. The default authentication method for the client mode is Kerberos V5, but you can add other methods as well, such as a Pre-shared Key or Certificate of Authority (CA).

It is important to remember that Kerberos V5 is the primary security protocol for authentication within the Windows domain. When you click on the Edit button to

38

modify Kerberos authentication, you will see that Kerberos V5 requires the Active Directory services.

An important service within Kerberos V5 is the Key Distribution Center (KDC). The KDC runs on each domain controller as part of the Active Directory directory service, which stores all client passwords and other account information. Therefore, you cannot use Kerberos V5, if you do not run the Active Directory services. In addition, Kerberos V5 authentication is not supported on computers running Windows XP Home Edition and computers running any other version of Windows 2000, Windows XP, or the Windows Server 2003 operating system must be members of the same Active Directory domain. You can also use the Certificate of Authority (CA) to authenticate IPSec sessions, but implementation of CA is not an easy task and requires a lot of planning and testing. The preshared key is the last choice and should not be used in production environment under any circumstances. Preshared keys are stored in plaintext in the registry of the computer. In the Active Directory, preshared keys are stored in a readable hexadecimal format. Preshared keys provide no security. You may as well not bother with the IPSec implementation, if you decide to use preshared keys as an authentication method in the production environment. I had a case where network administrators used a word “microsoft” as a preshared key! Secure Server (Require Security) policy has three rules:

• All IP Traffic (the default response rule) • All ICMP Traffic • <Dynamic> - the rule that allows the initial communication to be unsecured

The third rule called <Dynamic> allows the initial inbound communication request to be unsecured, but requires that all outbound communication be secured. The

39

filter action for this rule does not allow the IKE to fall back to the unsecured communication. If the IKE security negotiation fails, the outbound traffic is discarded and the communication is blocked. The <Dynamic> filter requires that all connections be secured with the IPSec. Any clients that are not IPSec-enabled, cannot establish connections. This IPSec policy is used, when you must provide the highest level of communication security between two systems or two networks. For example, you may want to use it, when two database servers must replicate payroll information between each other and you want to protect the data payload from eavesdropping by unauthorized personnel. This is especially important when you have multiple remote offices throughout the country or overseas and they must replicate the sensitive data with the home office. In this case, you should assume that the data at some point will pass through public lines and there is no guarantee that someone will not intercept it. Companies do engage in industrial espionage and at some point, someone may decide to spy on you, especially if you are competing for a lucrative business contract.

The All ICMP Traffic filter applies to any Internet Control Message Protocol (ICMP) traffic that is sent and received using a unicast IP address of network interface on a computer and all other computers. In the Windows Server 2003, this filter matches outbound IP broadcast and multicast traffic, when the traffic also uses the ICMP protocol. You should keep in mind that the ICMP protocol resides on level 3 of the OSI model and the utilities like PING and Trace Route are based on ICMP messages.

The All IP Traffic filter applies to all IP traffic that is sent and received using the unicast IP address of any network interface to any destination IP address.

40

Because inbound broadcast and multicast traffic use a multicast or broadcast type for the destination address, the inbound traffic is exempt. All ISAKMP traffic sent over UDP port 500, which is required for establishing IPSec-secured communication, is also exempted from IPSec filtering.

<Dynamic> the filter allows initial communication to be unsecured, but the data payload will be encrypted either with the 3DES or DES algorithm, depending on result of the IPSec negotiations. The authentication headers will be encrypted ether with SHA1 or MD5 hash algorithms.

You can modify the settings of these filters. Let’s examine the standard settings of each of them.

When you try to edit the All IP Traffic and All ICMP Traffic rules, you will get a window with five tabs:

• IP Filter List • Filter Action • Authentication Method • Tunnel Settings • Connection Type

The IP Filter List tab has two default IP filters: All ICMP Traffic and All IP Traffic. In this tab, you can create custom filters that fit your needs or just modify the existing ones.

41

If you decide to add a new filter, the wizard will guide you through the set up by asking you specific questions about the new filter. You can also modify the filter by clicking the Edit button. You will see the following window:

In order to make changes to this filter, you must click the Edit button again and you will get a window with three tabs:

• Addressing • Protocol • Description

On the Addressing tab, you can set up the Source Address and the Destination Address of the computers or the networks, between which you want to implement the IPSec encapsulation.

In the Source Address menu, you have four choices:

42

• My IP Address - the IP address of the workstation or a server, on which you currently work

• Any IP Address - indicates that any computer could be a source of the IPSec request

• A specific IP Address - you can specify the exact IP address of the IPSec source computer

• A specific IP Subnet - allows you do specify an entire subnet as a source of the IPSec traffic.

43

The Destination Address menu offers exactly the same configuration options. Please note that the default setting for the Destination Address is Any IP Address, whereas for the Source Address, the default setting is always My IP Address.

It is important to define which computer or a subnet will be the source of IPSec traffic and which will be the destination, because the correct IPSec filtering depends on it.

The Protocol tab allows you to choose the protocol you want to filter with the IPSec. Since we are investigating the ICMP filter, the default protocol in this case is ICMP.

If you look closely, you will notice a box under the Select a protocol type box. This box has no description. I guess Microsoft thought it is not important to

44

describe this field or they assumed that the user should know what it is. But for those who do not know what this number represents it is called “Protocol” field in IPv4 and in the Internet Protocol version 6 (IPv6) [RFC1883], this field is called the "Next Header" field.9

For example if this is ICMP protocol the value will be 1. In case of TCP the value will be 6 and if you decide to use UDP the value will be 17. You can read more about it at http://www.iana.org/assignments/protocol-numbers .

In the Internet Protocol version 6 (IPv6) [RFC1883], this field is called the "Next Header" field.10

Windows allows you to filter other protocols as well.

9 http://www.iana.org/assignments/protocol-numbers 10 http://www.iana.org/assignments/protocol-numbers

45

You can select from the list of predefined protocols:

• RAW - Any IP protocol other than TCP pr UDP is known as a RAW protocol

• RDP - Reliable Data Protocol • RVD - MIT Remote Virtual Disk Protocol • TCP - Transmission Control Protocol • UDP - User Datagram Protocol • XNS-IDP - Xerox Network Services - Internet Datagram Protocol

You can also select Any or Other as an option for the protocol.

When you select protocols TCP or UDP, you will be allowed to set a filter for the specific port or a whole range of ports.

The Description tab allows you to enter the description of the filter, which is very useful when you have multiple IPSec filters.

46

The Filter Action tab specifies whether this rule negotiates the secure traffic and how it will secure it. You have three options:

• Permit • Request Security • Require Security

The Permit filter policy will allow unsecured packets to pass through the IPSec filter. Each of these three predefined filters has three predefined security options. Let’s take a closer look at the properties of the Permit filter.

47

As you can see, there are three options at the Security Methods tab:

• Permit • Block • Negotiate security

As I have mentioned, the Permit option allows all unsecured packets to go through. The Block option blocks all unsecured packets. But when you select Negotiate security option, you are allowed to customize your filter to fit your needs. You can choose how the packet should be encapsulated. You can use the AH and ESP together to encapsulate the data payload. In this scenario, both Authentication Header and ESP will authenticate the source of the payload. But you can also select either ESP or AH to provide only authentication services. You cold also choose the ESP to encrypt the payload with the DES or 3DES encryption algorithm and provide authentication/integrity services using the MD5 or SHA1 hash algorithms.

The Security methods can be set to follow creation order. In the example below, the first method will be the Encryption and Integrity. Using this method, the ESP provides both encryption of the data payload and authentication\integrity services. The data is encrypted by the 3DES algorithm and the data integrity and the source authentication are provided by the SHA1 hash algorithm.

The second option in my example provides only the authentication services of the ESP protocol. The data itself is not encrypted by any algorithm. In this case, ESP provides only authentication and payload integrity services and the source integrity is secured by the SHA1 hash algorithm.

48

The third option uses the AH only and the authentication security is provided by MD5 hash algorithm. The data payload is not encrypted. Only the source of the data is checked by the IPSec device on the other end of the cable.

These three Security methods can be moved up or down, depending on your needs.

It is very important to understand the difference between encryption services and data integrity services. You should also know the roles, strength and weakness of each of the algorithms used in IPSec filters.

There are also three additional choices that control how the filter responds when it receives and forwards IPSec packets.

The first option is: Accept unsecured communication but always respond using IPSec. This option allows unsecured requests to come through, but the response will always be encrypted and the computer that makes a request must be IPSec ready in order to decrypt the data that it requested.

However, when you chose the Allow unsecured communications with non-IPSec aware computers option, you will receive the following warning:

49

Essentially, it says that your IPSec traffic may or may not be encrypted, when it reaches the destination. The data payload could be stripped off of all security encapsulation that is provided by the IPSec, because the destination may or may not be support the IPSec. Obviously, you will not use this option, when you must provide a secure and encrypted link between computers. But sometimes, you may have legacy systems that are not capable of processing the IPSec or you may have an unknown setup on the other end of the wire. In these cases, this option is a viable solution.

The Session key perfect forward secrecy (PFS) refers to the Internet Key Exchange (IKE) protocol. This protocol is a hybrid protocol and one of its components is the OAKLEY Key Determination Protocol (RFC 2412). As I have mentioned earlier, the IKE protocol provides mechanism to exchange cryptographic keys between authenticated parties across insecure links. One of the protocols it is using is OAKLEY protocol. It protects the session in case a session key or a private key are compromised by eavesdropping. Let say, the session keys are swapped periodically between computers, but the new session key is not generated too frequently and the attacker has enough time to record all the communications between the machines. The attacker can have enough time to break one session key and consequently unwind all the rest of the session keys that have been exchanged. The Session key perfect forward secrecy (PFS) is using the Diffie-Hellman key exchange algorithm to protect the session key exchanges. The eavesdropper would have to break each session key individually to gain access to the session and this would take a lot of time. Here is a small confusion. I have mentioned earlier that the frequency in which the new session key is generated is set in the Custom Security Methods Settings window. The new key can be generated after a fixed number of kilobytes are swapped or specific amount of time elapsed or both. But this setting should NOT be confused with the Session key perfect forward secrecy (PFS). This setting is not a part of the IKE protocol, because it is not using the Diffie-Hellman key exchange algorithm to protect the session key exchanges.

50

As you have probably noticed, the default settings for a session key generation are 100000 kilobytes and 3600 seconds. Since not too many administrators change this setting and it is not protected by the Diffie-Hellman key exchange algorithm, the potential hacker can set his eavesdropping tools to these values and get interesting results.

However, if you decide to use the Session key perfect forward secrecy (PFS) and modify its settings, you should click on the General tab, and then the Advanced button.

In the Key Exchange Settings, you can modify and enable PFS settings. To enable PFS, you must check Master key perfect forward secrecy (PFS).

51

By default, the new PFS keys are generated every 480 minutes with every new session. In other words, they are generated every eight hours, with the assumption that a typical working day lasts eight hours. You should always put some thought into keeping default security settings. They may or may not be sufficient.

The identities of cryptographic keys are protected by 3DES, DES, SHA1 and MD5 algorithms.

These IKE settings could be modified to fit your needs. In addition to setting integrity and confidentiality algorithms, you can specify which Diffie-Hellman groups to use.

The available IKE settings available to users depend on the version of the operating systems. For example, the Windows XP Professional offers two Diffie-Hellman groups: Low (1) and Medium (2), whereas Windows 2003 offers three: Low (1), Medium (2) and High (2048) as seen in the graphics below.

52

Diffie-Hellman groups are used to determine the length of the base prime numbers used during the key exchange process. The cryptographic strength of any key depends, in part, on the strength of the Diffie-Hellman group upon which the prime numbers are based.

The group 2048 (high) is stronger (more secure) than the Group 2 (medium), which is stronger than the Group 1 (low). The Group 1 provides 768 bits of keying strength, The Group 2 provides 1024 bits, and the Group 2048 provides 2048 bits. If mismatched groups are specified on each peer, negotiation fails. The group cannot be switched during the negotiation.

The Windows 2000 or Windows XP do not support the Diffie-Hellman group 2048. You should use the Diffie-Hellman group 2, if you are concerned with interoperability with these operating systems. You should not use the Diffie-Hellman group 1, if you want to provide enhanced IKE security. You should try to use the Diffie-Hellman group 2048 whenever possible.

The Windows 2000 does not provide the 3DES encryption algorithm unless you install the Service Pack 2. If a computer running Windows 2000 receives a 3DES setting, but does not have the High Encryption Pack or Service Pack 2 (or later) installed, the 3DES setting in the security method is set to the weaker DES in order to provide some level of confidentiality for communication, rather than block all communications. However, you should only use the DES as a fallback option if there are some computers in your environment that do not support the use of the 3DES. Computers running the Windows XP or Windows Server 2003

53

operating system support the 3DES and do not require installation of the High Encryption Pack.11

Once you configure these options, you should remember to check the Session key perfect forward secrecy (PFS) box to enable this function.

If you decide to enable the Session key perfect forward secrecy (PFS) for default IPSec policies, you should enter the <Dynamic> filter for each of them and check the Session key perfect forward secrecy (PFS) box.

11 http://technet2.microsoft.com/windowsserver/en/library/98b63c06-f05c-4597-955a-eb27303a0ac51033.mspx?mfr=true

54

The Authentication Method tab specifies how the trust is established between IPSec computers. These methods are used during IPSec negotiations. You are allowed to have multiple authentication settings. They can be moved up or down depending on which of them you want to use first. However, the default setting is Kerberos v5, which requires the Active Directory.

The Tunnel Setting tab allows you to set up the IPSec in the tunnel mode.

The default setting is the This rule does not specify an IPSec tunnel. If you wish to set an IPSec tunnel, you must enter the IP address of the IPSec device that is on the other end of the tunnel. Usually, it is an IP address of the IPSec enabled firewall.

55

The Connection Type tab allows you to specify a type of the connection you want to establish.

You have three choices:

• All network connections • Local area network (LAN) • Remote Access

They are self-explanatory, but you could be confused between the All network connection and Remote access. The All network connections include LAN and Remote access, but the Remote access does not include LAN connections. This is important distinction, if you deal with VPNs. The VPN connections are treated as LAN connections, even though you deal with computers that access your network remotely.

The last default IPSec policy is the Server (Request Security). This policy also contains three predefined rules:

• All IP Traffic - the default response rule. • All ICMP Traffic • <Dynamic> - the rule that allows the initial communication to be

unsecured.

The All IP Traffic default response filter and <Dynamic> filter allow initial incoming communication to be unsecured, hence the word Optional just after the Request Security.

56

The <Dynamic>, filter negotiates security for all outbound unicast IP traffic (security is not negotiated for multicast or broadcast traffic). The filter action for the <Dynamic> rule allows the IKE to fall back to the unsecured communication, when required. This is the main difference between the Server (Request Security) policy and Server (Required Security) policy.

This policy can be combined with the Client (Respond Only) policy, when you want traffic secured by the IPSec whenever possible, yet allow the unsecured communication with computers that are not IPSec-enabled. If the IKE receives a response from an IPSec-enabled client, but the IKE security negotiation fails, the communication is blocked. In this case, the IKE cannot fall back to the unsecured communication.

This set up is used in the end-to-end IPSec in the transport mode which allows servers to send some unencrypted traffic to other servers. For example, the HR application server provides employees with the access to their payroll information and other employee related data, which is located in an internal corporate network. The data is very confidential and must be encrypted. The employees must be able to securely communicate with the HR application server from their workstations that run the Windows 2000 or Windows XP Professional operating systems. However, the HR application server must also communicate with the WINS server, DNS server, DHCP server, Active Directory domain controllers, and a third-party data backup server. In this scenario, it is not necessary to secure the specific traffic between the HR application server and WINS server, DNS server, DHCP server, Active Directory domain controllers, and a third-party data backup server, because these servers may not support the IPSec. It is also not recommended to use the IPSec to secure communications between domain members and their domain controllers, because of the complexity of the IPSec policy configuration and management required in the Active Directory. The traffic

57

between the third-party backup server and the HR application server should also be permitted, because the backup server most likely does not support the IPSec.

IPSec for SMTP

This is all nice, but how do I configure the IPSec to encrypt let’s say the SMTP traffic? You simply create a new custom IPSec policy on your local computer and the matching IPSec policy on the e-mail server. But first, you have to access the IP Security Policies on Local Computer module through the Microsoft Management Console snap-in (mmc.exe). Since you are already at the MMC console, you may as well add the IP Security Monitor. In the next step, you will have to specify which computer or a domain this snap-in will manage.

In this example, I will show you how to establish the IPSec tunnel between the Exchange 2003 mail server and the workstation on a Local Area Network, where there is no firewall between them. Therefore, I selected the Local Computer for this snap-in to manage and clicked finish.

Before we go any further, I would like to mention that both the workstation and the e-mail server must have matching IPSec policies. The setup process is the same for both of them. In an essence, you are creating a virtual tunnel between them. However, you must remember that you are digging a virtual tunnel from both ends: from the e-mail server and from the workstation. It is like tossing ropes to tie two ships. First you toss one rope from a ship A to ship B and then another rope from the Ship B to ship A.

The presence of the firewall dramatically changes the behavior of the IPSec protocol. The entire setup could vary because you may encounter NAT

58

translations on the firewalls and, as I have explained before, the IPSec does not support the NAT. Secondly, you may decide to implement an IPSec tunnel between two firewalls. In this case, you may decide not to implement the IPSec protocol between your server/workstation and the IPSec gateway, such as the firewall to reduce complexity of the setup and the support overhead.

If you decide to implement the IPSec tunnel between the firewalls only, you may have to reference the user manual of your firewalls for a proper IPSec configuration, especially when you have to configure firewalls from different vendors. Don’t hesitate to contact the vendor of assistance. You don’t have to know everything and time is money.

Once you are in the IPSec management console, you will have three default IPSec policies:

• Client • Secure Server • Server

I have already explained each of these modes earlier.

At this point, you have two choices. You could modify one of them or create a new policy. In my opinion, it is better to create a new policy. As you will notice later, with a new policy and a good description, you will avoid confusion about which policy needs to be applied for the IPSec to work.

To create a new policy, you simply right click on the IP Security Policies on Local Computer and select the Create IP Security Policy from the drop down menu.

59

Click the Next button on a welcome screen of the configuration wizard.

The next screen will ask you to name the policy. This is an important and helpful feature. You may have several different IPSec policies configured for your convenience. It is a good habit to describe them in detail as it can save you a lot of time and frustration later.

60

The next screen will ask you, if you want to activate the default response rule. In other words, your computer will respond with this IPSec policy when a remote computer will request for the IPSec connection. Therefore, you should leave it checked.

At the next screen, you are allowed to select three types of the initial authentication. As I have mentioned earlier, the default authentication method for the IPSec in Windows 2003/XP is the Active Directory Kerberos V5 protocol. The most secure method is the CA certificate. The least secure method, which should be avoided, is a Preshared Key. For this example, I will use the third option, but please keep in mind that the Preshared Key is very easy to be compromised, because it is stored in clear text in the computer registry. Secondly, some people will use simple words like “microsoft” or “password” as a key. The Preshared Key works like a password. It is case sensitive and must exactly match on both ends of the tunnel.

61

The last screen of the wizard will inform you that the policy has been established, but you are not done with the configuration yet. You must edit its properties and select appropriate ports.

Once you click the Finish button and, if the Edit properties option is still checked, you will be taken to a window, where you set the rules for this filter. This is where you configure the IPSec tunnel.

62

To configure the IPSec tunnel, you click on the Add button and launch the IP Security rules wizard. You will get the following welcome screen.

You click Next and you will see the following two options:

• This rule does not specify a tunnel • The Tunnel endpoint is specified by this IP address

Basically, at this point, you can specify if you want your IPSec protocol to work in the transport mode or tunnel mode.

For this example, we will select the tunnel mode with the IP address of the Exchange server 192.168.16.101.

63

Once you click Next, the wizard will ask you to specify which network type this rule will apply to. You have three choices:

• All network connections • Local area network • Remote access

I have already explained them earlier. For this example, we will select the LAN as a network type.

After you click the Next button, you will be asked to select the type of the authentication method. For the sake of this example, I will use the preshared key method with the password microsoft.

64

Now, you will be asked to set up the IP filter. Please note that the wizard sets up two default filters. In theory, you could modify them for the SMTP traffic, but it will be very confusing. It is a lot better to create a brand new filter, dedicated to the SMTP traffic. To do so, you just need to click the Add button and the next popup window will guide you through the process.

Once you click the Add button, you will be asked to provide the description of this filter. You should describe how it works and between what computers or networks it provides the IPSec connectivity. You could also reference the network documentation manual, where the administrator could find more detailed information with the network diagram of the IPSec set up.

65

The next screen of the set up wizard will ask you for the source of the IP traffic. You have several self-explanatory choices:

• My IP Address • Any IP Address • A specific DNS Name • A specific IP Address • A specific IP Subnet

I have chosen My IP Address, which is the IP address assigned to the computer, which I am setting up the IPSec filter on.

On the next screen, you need to provide the endpoint of the tunnel. In this case, it is the IP address of the test Exchange 2003 server, 192.168.16.101. But you also have other choices available as well such us Specific Subnet. In this case the destination of the encrypted traffic will be an entire subnet.

66

After specifying the IP address of the test Exchange 2003 server, the wizard asks to assign the appropriate protocol and communication port. The SMTP resides on the Application layer of the OSI model and requires the TCP connection-oriented protocol that resides on the Transport layer. Therefore, we will select the TCP from the drop down menu.

Please note that you can also select the following protocols as well:

• EGP • HMP • ICMP • OTHER (it is an option to configure protocols that are not listed) • RAW • RDP • RVD • TCP • UDP • XNS-IDP

67

Since the SMTP traffic requires port 25, you should specify that the incoming and outgoing traffic will use the port 25.

The last screen will inform you that your custom filter has been configured successfully.

68

Once you finish the configuration of the IPSec filter, you must assign it, otherwise it will not work. In a tunnel environment, it must be assigned to both ends of the tunnel. In this example, it must be implemented both on the server and the workstation.

To assign the IPSec policy, you right click on it and select Assign from the drop down menu.

69

A small green dot will appear on the assigned IPSec policy to indicate its status.

Yes, there are a lot of windows to go through and it could be a very confusing process. But this is not the end. Just because you have configured and assigned the IPSec filter, it does not mean that you have established the encrypted tunnel between two computers. You have to frequently monitor this connection to make sure it is working correctly. Unlike the TLS protocol, the IPSec will not leave a trace in the e-mail header.

The IP Security Monitor console allows you to monitor the IPSec behavior. However, there are differences between the consoles on the Windows XP Professional and Windows 2003 Server.

The overview of the IP Security Monitor Console

The IP Security Monitor is a snap-in, which displays the details about the IPSec communication. The server version of the IPSec Monitor differs a little bit from the Windows XP Pro version by offering two additional features: Active Policies and Statistics.

As you have noticed, the IP Security Monitor is divided into two modes of operation:

• Main Mode • Quick Mode

The Main Mode’s primary function is to display the IKE Policies and statistics. Please notice that the Quick Mode does not have a container called the IKE Policies. The Main Mode container is divided into four separate sub containers:

70

• Generic Filters • Specific Filters • IKE Policies • Security Associations

On the server version of IP security Monitor, you also get the fifth container called Statistics.

The Quick Mode provides the information about the behavior of the IPSec protocol. The Quick Mode does not have the IKE Policies container, but instead, it has the Negotiation Policies container. In the server version, it also lists the Statistics container.

The only real difference between the Main Mode and Quick Mode is the statistics that are displayed.

Since the Windows XP Pro can be networked as a workgroup without the Active Directory and the Domain Controller, there is no need for the Active Policy container in the IP Security Monitor console. The Windows XP Pro allows you to implement the IPSec protocol in a workgroup environment. However, when you implement the IPSec in the Active Directory environment, you may run into all kinds of policy conflicts. Therefore, you should know what active policy is currently implemented.

71

The Active Policy container allows you to view individual IPSec policies. The IPSec policies could be applied in a form of a group policy in the Active Directory. The group policies have hierarchical structure and can be applied at the local computer level, the site level, the domain level, and the Organizational Unit Level. The IPSec policies are based on filters and rules and usually work very well. But when they are implemented in the Active Directory, conflicts between different policies are unavoidable. When conflicts between filters do occur, one filter takes the priority over the other filter and the end result may create undesirable result. The IPSec configuration could be compromised or could affect other traffic on the network and block users from accessing the network resources. The Active Policy container allows you to view the properties of the active IPSec policy that is running on the particular server.

In addition, the server IP Security Monitor gives you the ability to view individual IPSec policies. You can use the console to view things such as:

• Policy Name • Policy Description • Policy Last Modified (modification date) • Policy Store • Policy Path • Group Policy Object Name (name of the group policy that it is being called

from)

72

You can also use the Network Connectivity Tester, a command-line diagnostic tool called netdiag, to debug the IPSec protocol by typing the following command: netdiag /test:ipsec /debug.

The Main Mode

Let’s examine in detail each container in the IP Security Monitor. The first container in the main mode is called the Generic Filters, which is a generic representation of the current IKE policy. As I have mentioned before, the Internet Key Exchange protocol is a manager, which coordinates authentication and encryption of IP packets. The IKE automatically negotiates IPSec Security Associations (SA) and enables IPSec secure communications without time consuming and cumbersome manual pre-configuration.

The Generic Filter will display the following IKE information:

• The name of the IKE filter • The source and destination of IPSec tunnel • The IKE policy type • The authentication method • The connection type

Please note the name of the filter. It has been generated during the setup of this filter. A good description of the filter is crucial, especially when you have to manage several IPSec filters and troubleshoot connectivity issues. The Generic Filter will also indicate the type of the authentication method. In this example, I am using the preshared key: microsoft, which is essentially a password.

The second container is called Specific Filters. This is an expansion of the information listed in the Generic Filters container.

73

As you can see, the Specific Filters lists two IPSec filters, one is an outbound filter and the other is an inbound filter. Unlike the Generic Filter, where the source computer is listed as Me, the Specific Filters lists the IP addresses of the source and destination computers. It also lists the authentication method.

There is also a column labeled Weights, which contains a long number. It is a quite important value, which determines the order of the IPSec filters. The filters of an IPSec policy have no predefined order. However, the computer must be able to determine the order of these filters, so that they won’t block each other and cause mayhem. The IPSec Policy Agent component is responsible for deriving an IPSec filter list from the rules of the assigned IPSec policy to determine the best rule, which should be applied to incoming and outgoing traffic, and thereby determine the appropriate action.

The IPSec filter list order is based on a weight value calculated by the IPSec Policy Agent component. The highest weight value is applied first.

When processing an incoming or outgoing packet, the IPSec Driver component first searches the IPSec filter list to find a filter that matches the following values in the packet field:

• the addresses

• ports

• IP Protocol

In the next step, the first IPSec filter that matched the packet applies the appropriate action (permit, block, secure).

The main parameters for determining the weight of an IPSec filter are the following:

• The source IP address • The subnet mask of the source IP address • The destination IP address • The subnet mask of the destination IP address • The IP Protocol field value • For UDP and TCP traffic, the Source Port number • For UDP and TCP traffic, the Destination Port number

The weight value is based on how specific the originally defined IPSec filter is. The more specific IPSec filters will produce filters with a higher weight value. The highest weight IPSec filter has all seven IP traffic parameters. The lowest weight IPSec filter has none of the seven IP traffic parameters. All of these parameters can be specified when creating an IPSec filter.

74

The IKE Policy container shows the IKE policy number and the four security methods associated with it. If you right-click on the policy and select the Properties from the drop down menu, you will see exactly what the four methods are. Most likely, they are the encryption algorithms, DES, 3DES, MD5, and SHA1. The custom algorithms also will be listed there.

The Statistics container provides IKE statistics. The Main Mode deals with the IKE and the Quick Mode deals with the IPSec protocol. Therefore, the Quick Mode also has the Statistics container and yes, it is confusing. It is not you!

Let’s take a closer look at the information provided in this container.

75

The following table describes the items in the list of the IKE statistics in the IP

Security Monitor snap-in.12

IKE Statistic Description

Active Acquire An acquire is a request by the IPSec driver to have IKE perform a task. The active acquire

statistic includes the outstanding request and the number of any queued requests. Typically, the

number of active acquires is 1. Under a heavy load, the number of active acquires is 1 and the

number of requests that are queued by IKE for processing.

Active Receive The number of IKE messages received that are queued for processing.

Acquire Failures The number of times that an acquire has failed.

Receive

Failures

The number of times that the Windows Sockets WSARecvFrom() function has failed when

receiving IKE messages.

Send Failures The number of times that the Windows Sockets WSASendTo() function has failed when

sending IKE messages.

Acquire Heap

Size

The number of entries in the acquire heap, which stores active acquires. This number increases

under a heavy load and then gradually decreases over time, as the acquire heap is cleared.

Receive Heap

Size

The number of entries in the IKE receive buffers for incoming IKE messages.

Negotiation

Failures

The total number of negotiation failures that occurred during main mode (also known as Phase

I) or quick mode (also known as Phase II) negotiation. If you are having difficulty communicating

securely, attempt the communication and determine whether the number of Negotiation Failures

increases. If it does, check your authentication and security method settings for an unmatched

authentication method, an incorrect authentication method configuration (for example, the use of

preshared keys that do not match), or unmatched security methods or settings.

Authentication

Failures

The total number of identity authentication failures (Kerberos, certificate, and preshared key)

that occurred during main mode negotiation. If you are having difficulty communicating securely,

attempt the communication and determine whether the number of Authentication Failures

increases. If it does, check your authentication settings for either an unmatched authentication

method or an incorrect authentication method configuration (for example, the use of preshared

keys that do not match).

Invalid Cookies

Received

A cookie is a value contained in a received IKE message that is used by IKE to find the state of

an active main mode. A cookie in a received IKE message that cannot be matched with an

active main mode is invalid.

Total Acquire The total number of work requests submitted by IKE to the IPSec driver.

Total Get SPI The total number of requests submitted by IKE to the IPSec driver to obtain a unique Security

Parameters Index (SPI).

12 IP Security Monitor statistics http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/sag_ipsecmon_stats.mspx?mfr=true © 2007 Microsoft Corporation. All rights reserved.

76

IKE Statistic Description

Key Additions The number of outbound quick mode security associations (SAs) added by IKE to the IPSec

driver.

Key Updates The number of inbound quick mode SAs added by IKE to the IPSec driver.

Get SPI

Failures

The number of requests submitted by IKE to the IPSec driver to obtain a unique SPI that failed.

Key Addition

Failures

The number of outbound quick mode SA addition requests submitted by IKE to the IPSec driver

that failed.

Key Update

Failures

The number of inbound quick mode SA addition requests submitted by IKE to the IPSec driver

that failed.

ISADB List Size The number of main mode state entries, including negotiated main modes, main modes in

progress, and main modes that failed and have not been deleted.

Connection List

Size

The number of quick mode state entries.

Oakley Main

Mode

The total number of successful SAs created during main mode negotiations.

Oakley Quick

Mode

The total number of successful SAs created during quick mode negotiations. Because there are

typically multiple quick mode SAs created for each main mode SA, this number does not

necessarily match the main mode number.

Invalid Packets

Received

The number of received IKE messages that are invalid, including IKE messages with invalid

header fields, incorrect payload lengths, and incorrect values for the responder cookie (when it

should be set to 0). Invalid IKE messages are commonly caused by stale retransmitted IKE

messages or an unmatched preshared key between the IPSec peers.

Soft

Associations

The total number of negotiations that resulted in the use of plaintext (also known as soft SAs).

This typically reflects the number of associations formed with computers that did not respond to

main mode negotiation attempts. This can include both non-IPSec-aware computers and IPSec-

aware computers that do not have IPSec policy to negotiate security with this IPSec peer.

Although soft SAs are not the result of main mode and quick mode negotiations, they are still

treated as quick mode SAs.

The last container in the Main Mode is called Security Associations. This container lists the peer computers that use this specific IKE policy. It tells you that communications between the computers listed in this container are encrypted by the current IKE policy.

The Quick Mode

The Quick Mode provides information about IPSec protocol. The names of the containers in the Quick Mode are almost the same as in the Main Mode, but the

77

information they contain is different. Let’s take a look at the first container called Generic Filters.

In the first column, you can see the name of the policy. The source is set to Me, the destination IP address is listed as 192.168.16.88, but since the Quick Mode deals with the IPSec protocol, you can see the port numbers that are used by this filter. This filter applies the IPSec encryption to all SMTP traffic from the Exchange Server to the workstation with the IP address of 192.168.16.88. Since the SMTP is a connection oriented protocol, it requires the TCP protocol for the transport portion. You can also see that the inbound traffic is permitted.

The next container is called Specific Filters, but again we are dealing with the IPSec Protocol.

It is an expansion of the information listed in the Generic Filters container. It lists the inbound and outbound IPSec filters and the port numbers associated with them.

The next container is the Negotiation Policy.

78

This container lists the Filter Action of the IPSec rule. You have three default options:

• Permit • Request Security • Require Security

In this case, the Filter Action is set to Require Security. As I have mentioned before, the Require Security policy requires the IPSec protection for all traffic being sent or received (except initial inbound communication). The Secure Server (Require Security) policy includes stronger security methods. Unsecured communication with a non-IPSec-aware computer is not allowed.

If you right click on this policy and go to the Properties, you will be able to see four methods of the IPSec security. They are most likely the encryption algorithms, DES, 3DES, MD5, and SHA1. The custom algorithms will also be listed there.

The major difference between the Main Mode and Quick Mode is the statistics that are displayed in the Statistics container.

79

IPSec Statistics The following table describes the items in the list of IPSec statistics in the IP

Security Monitor snap-in:13

IPSec Statistic Description

Active Security

Associations

The number of active IPSec SAs.

Offloaded

Security

Associations

The number of active IPSec SAs offloaded to hardware.

Pending Key

Operations

The number of IPSec key operations in progress.

Key Additions The total number of successful IPSec SA negotiations.

Key Deletions The number of key deletions for IPSec SAs.

Re-Keys The number of rekey operations for IPSec SAs.

Active Tunnels The number of active IPSec tunnels.

Bad SPI Packets The total number of packets for which the Security Parameters Index (SPI) was incorrect. The

SPI is used to match inbound packets with SAs. If the SPI is incorrect, it might mean that the

inbound SA has expired and a packet using the old SPI has recently arrived. This number is

likely to increase if rekey intervals are short and there are a large number of SAs. Since SAs

expire under normal conditions, a bad SPI packet does not necessarily mean that IP security

13 IP Security Monitor statistics http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/sag_ipsecmon_stats.mspx?mfr=true © 2007 Microsoft Corporation. All rights reserved.

80

IPSec Statistic Description

is failing.

Packets Not

Decrypted

The total number of packets that failed decryption. This failure might indicate that a packet

arrived for which the SA had expired. If the SA expires, the session key used to decrypt the

packet is also deleted. This does not necessarily indicate that IP security is failing.

Packets Not

Authenticated

The total number of packets for which data could not be verified. This failure is most likely

caused by an expired SA.

Packets With

Replay Detection

The total number of packets that contained a valid Sequence Number field.

Confidential

Bytes Sent

The total number of bytes sent using the ESP protocol.

Confidential

Bytes Received

The total number of bytes received using the ESP protocol.

Authenticated

Bytes Sent

The total number of bytes sent using the AH protocol.

Authenticated

Bytes Received

The total number of bytes received using the AH protocol.

Transport Bytes

Sent

The total number of bytes sent using IPSec transport mode.

Transport Bytes

Received

The total number of bytes received using IPSec transport mode.

Bytes Sent in

Tunnels

The total number of bytes sent using IPSec tunnel mode.

Bytes Received

in Tunnels

The total number of bytes received using IPSec tunnel mode.

Offloaded Bytes

Sent

The total number of bytes sent using hardware offload.

Offloaded Bytes

Received

The total number of bytes received using hardware offload.

The Security Associations container lists the peer computers that use this specific IPSec policy. It tells you that communications between the computers listed in this container are encrypted by the current IPSec policy.

81

Network Monitor and IPSec

At some point, you will be required to set up an IPSec connection between two networks or multiple workstations. You will quickly realize that it is not a straightforward task. You will spend a lot of frustrating hours trying to establish an IPSec connection and yes, you will curse a lot. But you will also ask yourself a simple question which has a complicated answer: “Why does it not work?”.

I will not be able to cover all the possible issues with the IPSec protocol configuration in the Windows environment, but I can show you how the IPSec protocol is viewed in the Network Monitor.

The Network Monitor is a protocol analyzer. It allows you to capture the network traffic, view and analyze it. In other words. it is a packet sniffer. It is a very useful tool that helps in troubleshooting problems with the LAN and WAN traffic.

Before I proceed with analyzing captured packets, let’s send an SMTP message via telnet and then examine the network traffic between the client and the SMTP server. For reference, I’m including the steps to send the message.

• First, you need to open a connection from your computer to your mail server: telnet mail.domain.com 25

• You should receive a reply like: 220 mail.domaint.com Chicago cubs rock (version of the mail connector) (time and date)

This is basically a greeting message from the SMTP server; it is also called a banner. The banner can be customized. You should never put a word “Welcome” on the banner, because of the legal connotations of this word.

• After the greeting from the server, you should greet the server back in order to start the communication by typing the following command: EHLO mail.domain.com

• Your greeting will be acknowledged by the server with the message

250 OK

• Next, you should type the following command to tell the receiving SMTP

server who the message is from:

MAIL FROM:[email protected]

82

• You will get another acknowledgment:

250 OK- MAIL FROM [email protected]

• Type the following command to tell the receiving SMTP server who the message is to:

RCPT TO: [email protected]

• It will be acknowledged by:

250 OK - Recipient [email protected]

• You will have to inform the server that you are ready to send data by typing the command:

Data

• You receive the following response:

354 Send data. End with CRLF.CRLF

• You are now ready to start typing the text of the message. The user will see this part of the message in their inbox. Type the following command to add a subject line: Subject: test message

Press ENTER two times. You do not receive a response from this command.

• Now type:

Test test test

• Press enter and type period (.) to end the message and then press ENTER. You receive the following response:

250 OK

• Close the connection by typing the following command:

QUIT

• You receive the following response:

221 closing connection

83

Now let’s examine how the Network Monitor sees these steps, when there is no IPSec encryption.

Here is how the Network Monitor sees this process:

You can see the telnet exchanges between the client and the SMTP server. You can also notice that the text of the message is in clear text.

All e-mail messages are sent in a clear text unless they are encrypted.

Once you implement the IPSec filter on the e-mail server and the workstation, you should test it by sending an e-mail via a telnet session and sniff the packets with the Network Monitor. This is the only way to make sure that the IPSec filter is working. The IPSec Monitor may indicate that the tunnel has been established, but the sniffer may show you an unencrypted data payload and the entire authentication process. In this case, the IPSec filter is setup to accept all unencrypted traffic, but always asks for the secure channel. If the secure channel fails, then the traffic will be sent unencrypted.

84

This is how the Network Monitor sees the traffic between the MS Outlook and the Exchange 2003 server:

The TCP protocol flags the packets that are required to synchronize sequence numbers and the acknowledgments must be sent back. Once this process is completed, the entire communication between the workstation and the Exchange 2003 server is handled by the MSRPC protocol (Microsoft Remote Procedure Call).

MSRPC is derived from the Open Software Foundation RPC protocol, which has been implemented on other platforms for many years. You should keep in mind that MSRPC implements Microsoft-specific extensions that have historically separated it from other RPC implementations. Many of these interfaces have been in Windows since its inception, providing plenty of attack surface for buffer-overflow exploits and the like.14

Encrypted Telnet Session

Now let’s take a look at the unencrypted and encrypted telnet traffic between two PC. In this example I will try to telnet to a workstation with the IP address 192.168.1.100 to another workstation with the IP address 192.168.1.101. The authentication login ID is “administrator”. Telnet is a connection oriented

14 Hacking Exposed 5th Edition By Stuart McClure, Joel Scambray, George Kurtz; ISBN 0072260815 / 9780072260816

85

protocol, which requires the TCP protocol to provide reliable, in-order delivery of a stream of bytes. So let’s see how this delivery occurs in a real world.

In the first picture, you can see the telnet negotiations between two PCs after the connection has been established. The source PC (192.168.1.100) sends the first letter of the login ID, which in this case, is a letter “a”.

The source PC receives the acknowledgment from the destination PC (192.168.1.101) and sends the second letter “d”.

The entire sequence continues until the destination PC receives the entire login ID “administrator”.

86

The password is processed in exactly the same fashion, in other words, in a clear text without any security or encryption.

So what happens to this Telnet connection, when you apply the IPSec in a tunnel mode? Basically, the initial request for the Telnet session is initiated by the TCP protocol, but the immediately after the request has been issued, the IKE protocol kicks in and takes over the entire operation.

As I have mentioned before, IKE is a hybrid protocol that consists of ISAKMP, OAKLEY, SKEME protocols. I have already described their functionality. In this example, the IPSec filter is using two of these protocols: the ISAKMP and OAKLEY.

87

The ISAKMP protocol establishes SA and negotiates cryptographic keys. A Security Association (SA) is a combination of a destination address, a security protocol, and a unique identification value, called a Security Parameters Index (SPI).

ISAKMP can be implemented over any transport protocol or over IP itself. Implementations MUST include a send and receive capability for ISAKMP using the User Datagram Protocol (UDP) on port 500. UDP Port 500 has been assigned to ISAKMP by the Internet Assigned Numbers Authority (IANA). In the graphic above, you can clearly see that the ISAKAMP/IKE source and destination port is UDP port 500.

The Network Monitor can provide you with the wealth of the information about the IPSec connection. You can see the types of encryption algorithms that are being used. In this example, I am using the default algorithms. But you can also see the vendor ID, which is MS NT5 ISAKMPOAKLEY, WINDOWS 2003 FAMILY.

88

In addition, you can see what type of basic authentication has been used to establish this IPSec tunnel. Remember that I have used the preshared key. This authentication method is listed as Attribute: TV: basic Authentication method = XAUTHInitPreShared. I have also mentioned that the preshared key works like a password. It is stored in the registry of the computer in clear text! Let’s say you are a hacker, who is sniffing packets, and you notice that your victim is using a preshared key to secure the IPSec connection. And you know where to look for the key since it is located in computer’s registry.

As I mentioned earlier, OAKLEY is a protocol, which provides mechanism to exchange cryptographic keys between authenticated parties across insecure links.

Conclusion

The IPSec implementation is complex and should not be taken lightly. It can provide great security, but sloppy implementation can undermine all security benefits that are provided by the IPSec. There are a lot of components, such as servers, firewalls and other LAN/WAN devices that could have the undesirable effect on the implementation of this protocol. IPSec forces you to think about your network infrastructure as whole. You must not only think about how to secure communications between servers and workstations, but also about the type of data you are going to secure and the human factor. You also have to anticipate

89

hacker attacks. You are securing information for a reason. You do not want unauthorized people to have access to this data. However, these people may have a desire to access it at any cost and will spend time and money to hack it.

You should never assume that your secure links are flawless. Before the outbreak of the World War II, Germans started to use Enigma Machines to encrypt their secret communications. They were convinced that these codes could not be broken. Three young mathematicians from Poland, Marian Rejewski, Jerzy Różycki and Henryk Zygalski, broke the Enigma machines with sliding rulers, notepads, and perforated cards. They were all in their late twenties, when they accomplished this task. They started their work in 1932 and by September 1939, Polish military was able to intercept and read good portion of German military and diplomatic communications. Thanks to their early accomplishments, the British were able to build one of the first computers, called Ultra, and read everything that Germans encrypted. One of the main problems of the Enigma machines was the sloppiness of their operators, who forgot to reset the rotors. They were so convinced that the system was unbreakable that they stopped paying attention to crucial details. In essence, the arrogance of the operators was the Achilles’ heel of the Enigma machine. This human arrogance is the Achilles’ heel of all security systems.

90

Appendix

IPSec Standards

Request For Comments Articles

RFC 2401 - Security Architecture for the Internet Protocol

RFC 2402 - IP Authentication Header

RFC 2403 - The Use of HMAC-MD5-96 within ESP and AH

RFC 2404 - The Use of HMAC-SHA-1-96 within ESP and AH

RFC 2406 - IP Encapsulating Security Payload (ESP)

RFC 2408 - Internet Security Association and Key Management Protocol (ISAKMP)

RFC 2409 - The Internet Key Exchange (IKE)

RFC 2412 - The OAKLEY Key Determination Protocol

http://articles.techrepublic.com.com/5100-6345_11-5155749.html

http://www.microsoft.com/technet/network/sdiso/sdisofaq.mspx

http://technet.microsoft.com/en-us/library/bb877982.aspx