GRE tunneling and routing

Generic Routing Encapsulation, or GRE, is a protocol for encapsulating data packets that use one routing protocol inside the packets of another protocol. "Encapsulating" means wrapping one data packet within another data packet, like putting a box inside another box. GRE is one way to set up a direct point-to-point connection across a network, for the purpose of simplifying connections between separate networks. It works with a variety of network layer protocols.

In my case the GRE tunnel was needed to connect 2 networks together, our network AS206275 and a partner network which name I'm not allowed to specify. This GRE tunnel was ensuring that our servers were communicating with their servers at the lowest possible latency, I've configured this GRE tunnel to be using the OSI layer 2 adapter so we can bridge it to other interfaces for further expansion of our network.

Our GRE tunnel didn't have any internal IPs as they were not needed, I've also configured some static routes and some ARP announcings on that GRE tunnel so we can announce our IP addresses behind the GRE tunnel on the partner company network.

What is MTU and why is it affecting our setup?

In networking, maximum transmission unit (MTU) is a measurement representing the largest data packet that a network-connected device will accept. Imagine it as being like a height limit for freeway underpasses or tunnels: Cars and trucks that exceed the height limit cannot fit through, just as packets that exceed the MTU of a network cannot pass through that network.

However, unlike cars and trucks, data packets that exceed MTU are broken up into smaller pieces so that they can fit through. This process is called fragmentation. Fragmented packets are reassembled once they reach their destination.

MTU is measured in bytes — a "byte" is equal to 8 bits of information, meaning 8 ones and zeroes. 1,500 bytes is the maximum MTU size.

The MTU was affecting our setup because the GRE encapsulation was adding another 24 bytes to our IP header which had the MTU of 1500 which was maxing out the recommended MTU size to use.

My preferred solution for this issue

To fix this issue I've had to do some MSS clamping and change my static routes to advertise the correct MTU size, the MSS clamping was done via iptables and the option used for the static routes was "advmss MTU-SIZE-HERE".