IPsec Simplified

What is IPsec?

What network security does IPsec provide? IPsec is an extension of the Internet Protocol (IP) designed to secure network communication through cryptography. It can provide all of the security that can be achieved through cryptography: confidentiality and integrity of the data, plus authentication of the endpoint hosts. The combination of integrity and authentication provides non-repudiation. IPsec also detects replay attacks. We need to look at how it provides that security, and how to set it up.

Since IPsec is a modification of the IP implementation within the TCP/IP protocol suite, that means a modification to the kernel of modern operating systems.

Don't worry. Unless you are using something truly antique, your operating system will support IPsec.

The following attempts to be a "just enough" explanation. If you want all the details, see the official documents:
RFC 4301, "Security Architecture for IP"
RFC 4302, "IP Authentication Header"
RFC 4303, "IP Encapsulating Security Payload (ESP)"
RFC 4306, "Internet Key Exchange (IKEv2) Protocol"

How do you pronounce its name? I've heard all of these:
ipsec as a two-syllable word that would rhyme with tripsec, zipsec, etc...
eye pee seck
eye pee security
secure eye pee

Architecture

IPsec secures network communication between pairs of hosts. IPsec can be used to establish VPN (or Virtual Private Network) connections between sites or between a remote user and the core business site. User issues such as authenticating a human as the owner of some user identity, restricting access to data by users, and so on, are outside the scope of IPsec. Those important security tasks are left to the operating systems run on the hosts.

Put another way, IPsec provides network communication security for operating systems. Those operating systems then do their own work to authenticate users and authorize their activities.

Two hosts interested in securely communicating with each other (and pardon the anthropomorphism, but it makes this story much easier to tell and understand) must establish a Security Association (SA) with each other. The details can be tremendously complicated, but the SA comes down to the following. See my "Just Enough Crypto" page for explanations of ciphers, hashes, and HMACs:

Which of the CIA goals are important? Confidentiality, integrity, and host authentication are possible. Computers are fast and the cryptographic work is easy, so the usual answer is "all of them", although subsets are possible.
Confidentiality requires a cipher and a key. If it is desired, which cipher is used (e.g., AES or Blowfish) and in which mode (e.g., AES-256-CBC-MAC, AES-256-GCM) and what is the key?
Data integrity and host authentication requires a hashed message authentication code (HMAC) based on some cryptographic hash function. If it is desired, which HMAC is used (e.g., HMAC-SHA-2-256) and what is the key?

The SA is negotiated by the Internet Key Exchange (IKE) protocol. There is a quick burst of IKE traffic before two hosts start communicating with IPsec. IKE simplifies to these major steps:

The hosts authenticate to each other using cryptography. This could be done with shared secrets:
- Host 1 sends a challenge known to be unique. The number of seconds elapsed since 00:00:00 1 January 1970 makes for a useful challenge.
- Host 2 encrypts the challenge using the shared secret key and returns this response.
- Host 1 decrypts the response using the shared secret key. If the resulting cleartext is the challenge, host 2 knows the secret and must be the real host 2.
The problem with that approach is that it does not scale. You can set up shared secrets between a few machines. But since every machine-to-machine pair needs a key, and the number of keys grows as the square of the number of machines, that does not scale to a large organization. Set up digital certificates instead:
- Generate a public/private key pair and digital certificate for each host. The digital certificate is an X.509 data structure including the public key wrapped inside a digital signature made by the trusted signing authority, or Certification Authority (CA).
- Host 1 sends a challenge known to be unique. The number of seconds elapsed since 00:00:00 1 January 1970 makes for a useful challenge.
- Host 2 encrypts the challenge using its private key and returns this response along with the digital certificate containing the public key for Host 2.
- Host 1 verifies the digital signature on the certificate using its trusted copy of the signing authority's public key. Host 1 then extracts the public key for Host 2 from the certificate and uses that to decrypt the response. If the resulting cleartext is the challenge, the responding host knew the private key for Host 2 and must be the real Host 2.
After a round of that in each direction, each host is confident in the other host's identity.
The two hosts negotiate a mutually acceptable combination of confidentiality (or not) and integrity plus authentication (or not). This is usually the complete CIA triple, but some host pairs may be configured to only use some.
The two hosts negotiate a mutually acceptable set of algorithms and keys. The algorithms will generally be the strongest cipher and hash supported by both ends. The needed shared secret keys will be agreed upon using Diffie-Hellman key negotiation.

The SA information is then passed to the IPsec module of the kernel.

Once the SA has been established, IKE is used from time to time to negotiate new session keys. You want to use a key only for a limited time and amount of data. IKE can quickly re-run Diffie-Hellman to agree on new keys. Any delay caused by periodic re-keying will tend to disappear into the background noise of routine network latency jitter.

Once the kernel has an SA for another host, it will enforce its use. Every packet sent to that host will be modified according to the SA. Every packet apparently from that host will be dropped if it cannot be handled according to the SA. That means that if one host of an IPsec pair suddenly quit using IPsec and simply sent plain IP datagrams, the other host would drop all those apparently bogus datagrams.

Authentication Header is an IPsec extension to IP to provide data integrity, source host authentication, and protection against replay attacks.

Encapsulating Security Payload is an IPsec extension to IP to provide data confidentiality, data integrity, source host authentication, and protection against replay attacks.

Authentication Header (AH)

Authentication Header inserts an extension to the original IP header. The AH header includes the Security Parameters Index (SPI), which is effectively an index number indicating which SA is used. It also includes the integrity check value (ICV), which is a hashed message authentication code of the payload, the non-changing IP and AH header fields, and appropriate key.

Encapsulating Security Payload (ESP)

I have no idea why the term is Encapsulating Security Payload when Encrypting Security Payload would better emphasize the main point....

ESP inserts an extension to the original IP header, and adds an authentication block to the end of the datagram. The ESP header includes the Security Parameters Index (SPI), which is effectively an index number indicating which SA is used. It also includes a sequence number, a number that should increase with each IPsec datagram sent. This allows a system to recognize and ignore duplicate received IPsec datagrams.

A "Next header" field is added to the end of the encrypted payload, along with padding to make the datagram an integer multiple of 32 bits. Then a final authentication data block contains the ICV, which is an HMAC of the payload, the non-changing IP and ESP header fields, and appropriate key.

Transport Mode and Tunnel Mode

Transport Mode encrypts only the payload. The IP header itself is left alone (beyond changing the Protocol Type field to 50 for ESP). The source and destination IP addresses are the actual endpoints. Transport Mode is used for host-to-host communications. The data contents are protected, but anyone observing network traffic can see traffic flow patterns.

Original:	`[ H1 → H2 \| payload ]`
IPsec:	`[ H1 → H2 \| SPI \| E{payload, key} ]`

Tunnel Mode is used to encrypt the entire original IP datagram, both payload and original header, and then encapsulate that inside a new IP header. You create a Virtual Private Network (VPN) between two remote sites by doing Tunnel Mode IPsec at the gateways. Let's think about sending a packet from site #1 to site #2. Sites #1 and #2 each have correspondingly numbered gateways connecting them to the Internet.

The original datagram's IP header has the IP addresses of the endpoints. That entire datagram is encrypted at the site #1 gateway, and encapsulated inside a header routing it from gateway #1 to gateway #2. When it gets there, gateway #2 decrypts its payload according to the SA specified by the SPI in the header. The result is an IP datagram which is identical to the one that disappeared into the encryption back at the other site. That cleartext datagram is routed onto the networks within site #2.

Original:	`[ H1 → H2 \| payload ]`
IPsec:	`[ GW1 → GW2 \| SPI \| E{[ H1 → H2 \| payload ], key} ]`

Packets moving the opposite direction are encrypted on gateway #2, encapsulated inside a GW2-to-GW1 IP header and routed to gateway #1. It is decrypted there, and the resulting cleartext datagram is routed onto the internal site #1 networks.

If GW1 and GW2 do not not have an SA established, they set one up via IKE when the first packet bound for the other site reaches one of the gateways.

If the end-point hosts shared an SA, the payload of the datagrams crossing the internal networks of the two sites would be ciphertext. There's certainly no harm in encrypting data that happens to already be ciphertext. And this would provide defense in depth and protect against hostile insiders at both sites.

Original:	`[H1→H2\| payload ]`
Within sites:	`[H1→H2\|SPI_H1,H2\| E{payload, key_H1,H2} ]`
Between sites:	`[GW1→GW1\|SPI_GW1,GW2\| E{[H1→H2\|SPI_H1,H2\| E{payload, key_H1,H2} ], key_GW1,GW2} ]`

Amazon
ASIN: 059600270X

Amazon
ASIN: 1907117040

How To Set Up Host Keys and Certificates

Do these steps on the CA host. This syntax shows the use of tools from the OpenSSL package.

1: Generate a new certificate signing request. At the prompt Common Name put the new hostname, otherwise the defaults should be fine:
# cd /etc/ssl
# /etc/pki/tls/misc/CA.pl -newreq

2: Sign the new certificate:
# /etc/pki/tls/misc/CA.pl -sign

3: Decrypt the private key to cleartext, as the IPsec racoon daemon can't very well decrypt it on its own at boot time:
# openssl rsa -in newkey.pem -out newkey.pem
# chmod 400 newkey.pem

4: Rename the files:
# mv newkey.pem newhostname_key.pem
# mv newcert.pem newhostname_cert.pem
# mv newreq.pem newhostname_req.pem

5: Install the appropriate key pair, digital certificate, and public key of the trusted signing authority on each host. Those will be the three files named newhostname_* plus the CA public key cacert.pem. Copy them to the IPsec host and install them in the correct location. On a Linux machine with the racoon daemon managing the SAs, those files go in the directory /etc/racoon/certs/.

How To Configure IPsec on Linux

Assuming that you have generated and distributed certificates as shown above, and you want to turn on IPsec between the two hosts 192.168.0.1 and 10.1.2.3:

Create a policy in /etc/setkey.conf on each host:
Put the following in /etc/setkey.conf on host 192.168.0.1:

# Out our 192.168.0.0/24 connection to 10.1.2.3:
spdadd 192.168.0.0/24 10.1.2.3/32 any -P out ipsec
	esp/transport//require
	ah/transport//require;
# In from 10.1.2.3 on our 192.168.0.0/24 connection:
spdadd 10.1.2.3/32 192.168.0.0.0/24 any -P in ipsec
	esp/transport//require
	ah/transport//require;

Then put the converse in /etc/setkey.conf on host 10.1.2.3:

# Out our 10.1.2.0/24 connection to 192.168.0.1:
spdadd 10.1.2.0/24 192.168.0.1/32 any -P out ipsec
	esp/transport//require
	ah/transport//require;
# In from 192.168.0.1 on our 10.1.2.0/24 connection:
spdadd 192.168.0.1/32 10.1.2.0/24 any -P in ipsec
	esp/transport//require
	ah/transport//require;

Configure the racoon daemon to run IKE on each host:
Modify /etc/raccoon/racoon.conf to include two stanzas like this, replacing thishostname with the local host name in each case:

sainfo anonymous
{
	lifetime time 1 hour ;
	encryption_algorithm blowfish, twofish, rijndael, aes ;
	authentication_algorithm hmac_sha1, hmac_sha256, hmac_sha384, hmac_sha512 ;
	compression_algorithm deflate ;
}

remote anonymous
{
	exchange_mode main;
	doi ipsec_doi;
	situation identity_only;
	my_identifier asn1dn;
	# Change these host names to match yours!
	certificate_type x509 "thishostname_cert.pem" "thishostname_key.pem";
	verify_cert off;

	nonce_size 16;
	initial_contact on;
	proposal_check obey;    # obey, strict, or claim

	proposal {
		encryption_algorithm aes;
		hash_algorithm sha1;
		authentication_method rsasig;
		dh_group 2;
	}
}

Start racoon and configure it to start after each boot:

# /etc/init.d/racoon start
# chkconfig racoon on

Opportunistic Encryption

Most people very reasonably say that it makes no sense to arrange a method for whispering secrets to strangers. Data encryption implies data sensitivity, and so you should only send sensitive data to a host of known identity. Well, sure, but....

There is an interesting thing called opportunistic encryption. This is an attempt to establish encrypted communication with any host before falling back to plaintext communication if needed. This means you can routinely encrypt some of your network traffic without setting up any Security Associations, just by asking your operating system to attempt opportunistic encryption and then communicating with other hosts supporting the same thing.

What does this accomplish? Not much meaningful security. But, as I said, computers are fast, and the idea is why not encrypt anyway despite the lack of real security?

I have taught networking and information security to people from various U.S. Government Three-Letter Agencies. The topic of opportunistic encryption takes them through these stages:

Polite attempts to correct my error, surely I don't really mean what I seem to be saying.
Derision at the foolish people who would even consider such a silly thing as whispering secrets to strangers.
Utter horror when they are finally convinced that some people really do use this for things like routine web browsing, IRC, electronic mail, peer-to-peer file sharing, and more.

Yes, this would allow the opportunistic encryption subset of Internet users to communicate with each other while frustrating attempts to routinely eavesdrop on them. See the Wikipedia page for pointers to the details for doing this on Linux and routers with Openswan and other FreeS/WAN derivatives, on Windows, to protect e-mail and VoIP, and more.