Migrations were successful, DR was working and we were able to ping machines over the tunnel. However, I had a situation with a customer who was unable to use services such as SSH or RDP over the HCX L2 extension VLAN tunnels.
After testing between two machines on either side of the tunnel, we were able to manually adjust the MTU in the test machines and was able to start using SSH between these two devices. Unfortunately, this isn't going to be a solution for the remaining workloads from on-prem to target sites across the stretched VLANs. This was narrowed down to an MTU mismatch on the physical network underlay on one of the firewalls at Azure.
To mitigate this issue, we have two options, lower the MTU for the Network Extension in the Uplink profiles, or try and solve the MTU issue on the network.
The following commands should help you understand the discovered MTU path at each hop. The MTU for the Uplink Network Profile should have the network underlay in consideration. MTU that is too low may affect HCX and will need to be tuned in the Network Profile.
CLI commands
Show the IP addresses of tunnel peers
>ccli
>show transport status
Discovered MTU path
>ssh
>Traceroute -I <target hcx appliance IP> --mtu
vNic adapter config, shows IP address of each adapter, MTU and mac address
>ifconfig
>ping -s <mtu 1200> <target peer IP> -M do
Fast ping using 10,000 packets to check for packet loss
>ping -I <source HCX appliance IP> -i 0.001 -c 10000 <target IP>
Fast ping using 10,000 packets to check for packet loss using a MTU value
>ping -I <source HCX appliance IP> -i 0.001 -c 10000 <target IP> -s <mtu> -M do
Routing table
>route -n
Shows the stretched L2 network extensions
>show netext networks
Packet capture on port number from vNic
>tcpdump -I vNic_<x> port <port number> -n
Packet capture from two vNics on two source appliances
>tcpdump -I vNic_<x> host <source IP> and host <source IP> -n
For the test machines at either end of the tunnel, we had Windows at on-prem and a Linux VM at the target. After you have manually changed the MTU on the windows server for your tests, the following commands were able to check for MTU fragmentation.
Testing on Windows VM
On Windows the flags -f and -l will be used; -f flag tells ping to NOT fragment the packets and -l sets the size of the packet. Assuming an MTU setting of 9000, subtract 28 to get the packet size to use, the command will be:
ping REMOTE_HOSTNAME -f -l 8972