MTU Troubleshooting on Cisco IOS

Maximum Transmission Unit (MTU) is the largest size in bytes that a certain layer can forward. The MTU is different for each protocol and medium that we use. Ethernet for example has a MTU of 1500 bytes by default.

This means that a single Ethernet frame can carry up to 1500 bytes of data. On top of this data we add the Ethernet header. Typical header sizes are 14 bytes for Ethernet (add another 4 bytes if you use 802.1Q tagging).

Below is a picture with the different MTU fields that you might encounter:

You can see that a typical Ethernet header is 14 bytes, IP is 20 bytes and TCP is also 20 bytes. The maximum amount of payload that TCP can use is called the TCP MSS (Maximum Segment Size). This MSS value is the largest amount of data that a host can receive in a single TCP segment. This value is used to set a limit on the payload in order to prevent fragmentation and is sent in the SYN packet during the 3 way handshake. The MSS value isn’t synchronized between hosts, it can be different for each direction.

So why is all of this important to know? Let’s imagine we have an IP packet that is sent on our LAN. The size of the Ethernet frame will be like this:

• 1460 bytes of payload for TCP.
• 20 bytes for the TCP header.
• 20 bytes for the IP header.
• 14 bytes for the Ethernet header.

1460 (PAYLOAD) + 20 (TCP)  + 20 (IP) = 1500 bytes + 14 (ETHERNET) = 1514 bytes in total.

Sending 1514 bytes is no problem for Ethernet but some other mediums might have issues with large MTU values. Often problems arise when you add additional headers because of PPPoE, GRE Tunneling or IPSEC since this reduces the available bytes for our MTU. To demonstrate this problem (and how to solve it!) I will use a simple network with a reduced MTU. Here’s what it looks like:

The network above has two routers, a webserver (S1) behind R1 and a client (H1) behind R2. Here’s what the configuration looks like:

Configuration

First we’ll configure some IP addresses:

R1(config)#interface fastEthernet 0/0
R1(config-if)#ip address 192.168.12.1 255.255.255.0

R1(config)#interface fastEthernet 0/1           
R1(config-if)#ip address 192.168.1.254 255.255.255.0

R2(config)#interface fastEthernet 0/0
R2(config-if)#ip address 192.168.12.2 255.255.255.0

R2(config)#interface fastEthernet 0/1
R2(config-if)#ip address 192.168.2.254 255.255.255.0

I’ll add some static routes for connectivity:

R1(config)#ip route 192.168.2.0 255.255.255.0 192.168.12.2

R2(config)#ip route 192.168.1.0 255.255.255.0 192.168.12.1

Here’s what the default MTU values look like:

R2#show interfaces fastEthernet 0/0 | include MTU
  
MTU 1500 bytes
, BW 100000 Kbit/sec, DLY 100 usec, 

R2#show ip interface fastEthernet 0/0 | include MTU
  
MTU is 1500 bytes

The first MTU value is the interface MTU, it’s 1500 bytes by default for Ethernet. The second one is the IP MTU which is also 1500 bytes. Once you get above 1500 bytes your router will start fragmenting the IP packets.

Is this limit of 1500 bytes really working? There’s an easy way to find out. Let’s do a ping with the DF-bit (Don’t Fragment) between the routers:

R2#ping
Protocol [ip]:   
Target IP address: 192.168.12.1
Repeat count [5]: 1
Datagram size [100]: 
Timeout in seconds [2]: 
Extended commands [n]: y
Source address or interface: 
Type of service [0]: 
Set DF bit in IP header? [no]: y
Validate reply data? [no]: 
Data pattern [0xABCD]: 
Loose, Strict, Record, Timestamp, Verbose[none]: v
Loose, Strict, Record, Timestamp, Verbose[V]: 
Sweep range of sizes [n]: y
Sweep min size [36]: 1495
Sweep max size [18024]: 1505
Sweep interval [1]: 
Type escape sequence to abort.
Sending 11, [1495..1505]-byte ICMP Echos to 192.168.12.1, timeout is 2 seconds:
Packet sent with the DF bit set
Reply to request 0 (1 ms) (size 1495)
Reply to request 1 (4 ms) (size 1496)
Reply to request 2 (1 ms) (size 1497)
Reply to request 3 (4 ms) (size 1498)
Reply to request 4 (1 ms) (size 1499)
Reply to request 5 (4 ms) (size 1500)
Request 6 timed out (size 1501)
Request 7 timed out (size 1502)
Request 8 timed out (size 1503)
Request 9 timed out (size 1504)
Request 10 timed out (size 1505)
Success rate is 54 percent (6/11), round-trip min/avg/max = 1/2/4 ms

In the ping above you can see that the largest packet that I can send is 1500 bytes. The second packet with 1501 bytes can’t be sent because it is too large and we set the DF-bit.

Let’s look at an actual packet between the client and the webserver, see what these values look like in an actual frame:

Above you can see the TCP MSS which is 1460 bytes. What else can we see in Wireshark?

Above you see that the total length of the IP packet is 1500 bytes (1460 bytes for TCP MSS + 40 bytes for the TCP/IP header). Ethernet adds another 14 bytes which is how we get to 1514 bytes in total.

To simulate a network with a lower MTU I will reduce the MTU of the FastEthernet 0/0 interface of R2:

R2(config)#interface fastEthernet 0/0
R2(config-if)#mtu 1400

By reducing the MTU to 1400 bytes, the largest TCP segment size will be only 1360 bytes (1400 – 40 = 1360).  This is a scenario where users often complain that sending a ping is no problem but accessing a website or something else doesn’t work. Why? Let’s look at the example below:

Above you see a ping between the client and webserver. As you can see the total length is only 74 bytes…no problem to send this because our MTU allows 1400 bytes. Now we will try to connect to the webserver from the client using HTTP:

This is where things go wrong…as you can see above the total length of the IP Packet is 1500 bytes which is 100 bytes above our MTU of 1400 bytes. It’s impossible to communicate between the client and webserver now.

How do we fix this? There are two solutions for this:

• Set the correct IP MTU value so the router knows when to fragment IP packets.
• Reduce the TCP MSS value for outgoing connections so there is less payload.

Here’s how to configure the correct IP MTU value: