IPsec VPN Fundamentals

In the previous lesson, you saw that a public WAN sends your traffic across the Internet, and that this traffic needs to be encrypted.
Take a company with a headquarters in New York and a branch office in San Francisco: R1 and R2 exchange traffic through the Internet every day.

Figure 1 – Enterprise sites network topology

But there is a problem.

The Internet provides zero encryption. Every packet traveling between the two sites is in cleartext.
Anyone capturing traffic between the two ISP routers can read your data.

IPsec solves this.
It wraps every packet in an encrypted envelope before sending it across the untrusted network.
Only the two IPsec peers hold the keys to decrypt the content.

Figure 2 – IPsec encrypted tunnel concept

IPsec is not a single protocol. It is a framework that combines several components. Let's go through each one:

• IKE (Internet Key Exchange): Negotiates encryption parameters and exchanges keys between peers.

• ESP (Encapsulating Security Payload): Encrypts and authenticates the data payload.

• AH (Authentication Header): Provides authentication and integrity without encryption. Rarely used in modern deployments because ESP can do both.

• SA (Security Association): A one-way agreement between two peers that defines the encryption algorithm, keys, and lifetime for a specific traffic flow.

In real-world deployments, the combination you will see most often is IKE + ESP.

Now that you know what IPsec is, let's see how it actually establishes a secure tunnel.

IPsec uses a two-phase process. Before any of your traffic is encrypted, the two peers must agree on security parameters and exchange cryptographic keys.

This negotiation happens through IKE (Internet Key Exchange).
Here is the high-level packet processing flow when IPsec is active:

Figure 3 – IPsec packet processing flow

The process works in three stages:

1. Interesting traffic detection: The router checks outgoing packets against an ACL. If the packet matches, it triggers the IPsec process.

2. IKE Phase 1: The two peers establish a secure management channel (ISAKMP SA) to protect the negotiation itself.

3. IKE Phase 2: Using the secure channel from Phase 1, the peers negotiate the actual IPsec SA that will encrypt user traffic.

IKE Phase 1

IKE Phase 1 creates a bidirectional ISAKMP Security Association between the two peers.
Think of it as the secure meeting room where both sides sit down and agree on how they will protect the actual data.

Figure 4 – IKE Phase 1 tunnel establishment

During Phase 1, the peers negotiate five parameters.
You need to know each one of them:

Both peers must agree on identical parameters for Phase 1 to succeed.
If there is a mismatch on any parameter, the negotiation fails.

IKE Phase 2

Once Phase 1 is complete, the peers use the secure ISAKMP channel to negotiate the IPsec SA in Phase 2.
This SA defines how your actual user traffic will be encrypted.

Figure 5 – IKE Phase 2 IPsec SA establishment

Phase 2 negotiates:

• The transform set (encryption + integrity algorithms for user traffic, e.g., ESP-AES + ESP-SHA-HMAC)

• The interesting traffic definition (which traffic flows will be protected)

• The IPsec SA lifetime (how long before rekeying)

When IPsec is configured on a Cisco router, the transform set contains a mode tunnel command.

What does that mean? And what is the alternative?

IPsec supports two encapsulation modes: Transport mode and Tunnel mode.
The mode determines how much of the original packet is protected and whether a new IP header is added.

Transport Mode

In transport mode, IPsec encrypts only the payload of the original IP packet. The original IP header is preserved and used for routing. ESP is inserted between the original IP header and the payload.

Figure 6 – IPsec transport mode packet structure

You would typically use transport mode in this scenario:

• Host-to-host communication between two servers on the same network

Because the original IP header stays intact, transport mode has less overhead than tunnel mode. No additional IP header is added.

However, keep in mind the limitation: since the original IP header is visible, an attacker can see the true source and destination IP addresses of the communicating hosts.

Tunnel Mode

In tunnel mode, IPsec encrypts the entire original IP packet, header and payload, and wraps it inside a new IP header. This new outer header contains the IPsec peers' WAN addresses and is used for routing across the Internet.

Figure 7 – IPsec tunnel mode packet structure

Tunnel mode is the default mode for site-to-site VPNs. It completely hides the original source and destination addresses. An attacker capturing packets on the Internet sees only the two router WAN IPs. They cannot determine which internal hosts are communicating.

The tradeoff is additional overhead. The new outer IP header adds 20 bytes, which reduces the effective MTU: on a Cisco router it drops to about 1438 bytes instead of 1500.

Comparison Table

Here is a side-by-side comparison to help you remember the differences:

Tunnel mode is the default and is used in virtually all site-to-site VPN configurations. Transport mode is reserved for specific scenarios where the original IP header must stay visible.