OSPF MD5 Authentication

In plaintext authentication, routers pass the password directly inside Hello packets.
Anyone opening Wireshark on that link sees the cleartext password immediately.

How MD5 Protects the Password

MD5 fixes this vulnerability by keeping the secret key strictly on the router.
Instead of sending the password over the wire, routers process it through MD5 and transmit a cryptographic hash instead.

Figure 1 - Plaintext shows the key, MD5 shows a hash

Compare the two captures: plaintext gives away the exact string you typed, while MD5 shows a 32-character hexadecimal hash and nothing else.

How MD5 Authentication Works

1. R1 combines its secret key (PMN_KEY) with the OSPF Hello packet payload.

2. It hashes the combination using the MD5 algorithm.

3. R1 attaches the resulting hash inside the OSPF packet header.

Figure 2 - The key goes in, the hash comes out

When R2 receives the packet, it runs the exact same calculation using its locally stored key:

• Match? The packet is accepted.

• Mismatch? The packet is dropped immediately.

Even if an attacker captures the frame, an MD5 hash is a one-way function and cannot be reversed to reveal the original key.

Enabling MD5 alters how authentication data is carried within the OSPF frame:

• Auth Type: Set to Type 2 (Cryptographic).

• Auth Crypt Data: The 32-character MD5 hash is appended to the end of the OSPF packet.

Figure 3 - Both sides match, the adjacency forms

Under the hood, MD5 introduces two extra fields inside the authentication header:

• Key ID: Identifies which key is being used. Any integer works, but it must match on both sides.

• Cryptographic Sequence Number: A counter that increments with every packet to prevent replay attacks.

When both routers use matching Auth Types, secret keys, and Key IDs, the hashes match and the adjacency forms smoothly.

What the Attacker Sees

If an attacker tries to inject Hellos with Auth Type 0, both R1 and R2 drop those packets immediately due to the type mismatch.

Figure 4 - The rogue Hello is still ignored

Opening Wireshark on an MD5-secured link still displays the raw OSPF packets, but the plaintext key is completely gone, replaced by the Auth Crypt Data hash.

Figure 5 - The attacker captures a hash, not a key

Because MD5 is a one-way hashing function, an attacker cannot reverse this string back to the original password. Without the key configured locally on their device, the captured packet is useless.

Key Takeaway: For an OSPF MD5 adjacency to form, three parameters must match on both routers:

1. Auth Type (Type 2)

2. Key ID

3. Key / Password

Just like plaintext, you can enable MD5 per interface or across an entire area.
Let's start with the interface approach using the same topology: R1 and R2 in Area 0 over the 10.0.12.0/30 link.

Figure 6 - Lab topology

Applying the Interface Commands

Configuring MD5 requires two interface commands: one to enable MD5 authentication, and one to set the Key ID and secret key.