IPSec - Internet Protocol Security

IPSec is one of the new buzz words these days in the networking security area. It's becoming very popular and also a standard in most operating systems. Windows 2000 fully supports IPSec and that's most probably where you are likely to find it. Routers these days also support IPSec to establish secure links and to ensure that no-one can view or read the data they are exchanging.

When the original IP (Internet Protocol) specification was created, it didn't really include much of a security mechanisim to protect it from potential hackers. There were 2 reasons they didn't give IP some kind of security. First was because back then (we are talking around 30 years ago) most people thought that users and administrators would continue to behave fairly well and not make any serious attempts to compromise other people's traffic. Second reason was because the cryptographic technology needed to provide adequate security simply wasn't widely available and in most cases not even known about!

How IPSec Works

The Internet Security Agreement/Key Management Protocol and Oakley ( ISAKMP)

ISAKMP provides a way for two computers to agree on security settings and exchange a security key that they can use to communicate securely. A Security Association (SA) provides all the information needed for two computers to communicate securely. The SA contains a policy agreement that controls which algorithms and key lengths the two machines will use, plus the actual security keys used to securely exchange information.

There are two steps in this process. First, the two computers must agree on the following three things:

1) The encryption algorithm to be used (DES, triple DES)

2) Which algorithm they'll use for verifying message integrity (MD5 or SHA-1)

3) How connections will be authenticated: using public-key certificate, a shared secret key or Kerberos.

Once all that has been sorted out, they start another round of negotiations which cover the following:

1) Whether the Authentication Header (AH) protocol will be used

2) Whether the Encapsulating Security Payload (ESP) protocol will be used

3) Which encryption algorithm will be used for ESP

4) Which authentication protocol will be used for AH

IPSec has 2 mechanisms which work together to give you the end result, which is a secure way to send data over public networks. Keep in mind that you can use both or just one of these mechanisms together.

These mechanisms are:

• Authentication Header (AH)
  
• Encapsulating Security Payload - ESP

The Authentication Header (AH) Mechanism

The Authentication Header information is added into the packet which is generated by the sender, right between the Network (Layer 3) and Transport (Layer 4) Layer (see picture below) of the OSI model.

Authentication protects your network, and the data it carries, from tampering. Tampering might be a hacker sitting between the client and server, altering the contents of the packets sent between the client and server, or someone trying to impersonate either the client or server, thus fooling the other side and gaining access to sensitive data.

To overcome this problem, IPSec uses an Authentication Header (AH) to digitally sign the entire contents of each packet. This signature provides 3 benefits:

1) Protects against replay attacks. If an attacker can capture packets, save them and modify them, and then send them to the destination, then they can impersonate a machine when that machine is not on the network. This is what we call a replay attack. IPSec will prevent this from happening by including the sender's signature on all packets.

2) Protection against tampering. The signatures added to each packet by IPSec means that you can't alter any part of a packet undetected.

3) Protection against spoofing. Each end of a connection (e.g client-server) verifies the other's identity with the authentication headers used by IPSec.

The AH is computed on the entire packet, including payload (upper layers - 4,5,6,7) and headers of each layer. The following picture shows us a packet using AH :

Below is the analysis of the Authentication Header:

AH Algorithms

For point-to-point communication (e.g client to server), suitable authentication algorithms include keyed Message Authentication Codes (MACs) based on symmetric encryption algorithms (e.g DES) or on one-way hash functions (e.g MD5 or SHA-1).

For multicast communication (e.g between a group of routers), one-way hash algorithms combined with asymmetric signature algorithms are usually used, but they are also more cpu intensive.

The Encapsulating Security Payload - ESP

The Authentication Header (AH) we spoke about will protect your data from tampering, but it will not stop people from seeing it. For that, IPSec uses an encryption which provides the Encapsulating Security Payload (ESP). ESP is used to encrypt the entire payload of an IPSec packet (Payload is the portion of the packet which contains the upper layer data).

ESP is a bit more complex than AH because alone it can provide authentication, replay-proofing and integrity checking. It acomplishes this by adding 3 separate components:

• An ESP header
• An ESP trailer and
• An ESP authentication block.

Each of these components contains some of the data needed to provide the necessary authentication and integrity checking. To prevent tampering, an ESP client has to sign the ESP header, application data, and ESP trailer into one unit, of course ESP is used to encrypt the application data and the ESP trailer to provide confidentiality. The combination of this overlapping signature and encryption operation provides good security.

Let's have a look at a packet using IPSec - ESP:

IPSec can get very complicated and messy. I have tried keeping everything as simple as possible, but you should keep in mind that this topic can be studied in far greater depth than is presented here!