VMware HCX Troubleshooting
Check in app.log for application events
Check in web.log for web service events
/var/log/vmware/hbrsrv*.log on IX appliances
Check Firewall Port Connectivity
Use this command to check whether the port is open between HCX Manager and ESXi hosts or Appliances
curl -vv telnet://<esxihost>:<port>
tcpdump -ni any port 4500
Nc -vu destination-IP 4500
Traceroute -U -p 4500 -s local and remote IP
Verify Network Connection for Download
tcpdump port 443
Perftest gives a report on the available bandwidth for the HCX tunnels. Hitting tab should give you a list of available commands.
In order to run "perftest"
from HCM manager, run ccli by calling "ccli" as "root" user
inside ccli context, list deployed appliances by command "list"
from the list shown, select the appliance to run test. Each appliance has a "Id" field. select appliance by command "go <Id>".
run perftest by calling "perftest site"
run perftest with multiple flows with parameter "-P <num_of_flows>"
perftest all allows all tests to be run.
perftest uplink will test the links. use the -T flag to constrain by length of test, i.e. -T 1 will check one minute in each direction.
Health Check Commands
hc probe -d
Select an appliance and run a unit health check to check the state of the resources, services and tunnels.
go 2 (example)
hc unit all -d
HCX Connectivity Troubleshooting
Check the connectivity status from the HCX Interconnect:
Check the Security associations are up: IPsec status
Confirm the routing is appropriate: IP route
Get the remote side IP: IP tunnel
Try pinging the remote side (only if ICMP is allowed): ping <remote side IP>
Confirm traffic on UDP 4500 is sent and received: tcpdump -s0 -n -i vNic_# udp
# tcpdump -c 10 -ni vNic_0 udp and port 4500
(note the vNic_# might be vNic_0 or another vNic, you can get the info via "ip route" and use that in the tcpdump command above)
[admin@HCX-MGR] go 0 Switched to node 0. [admin@HCX-MGR] ssh Welcome to HCX Central CLI [root@<appliance0>] ip tun <status of gre tunnel> [root@<appliance0>] tcpdump -c20 -i vNic_0 -n -e host <ip address of remote side>
1. Start the capture
You can just run tcpdump command to start the capture. As you can see below, by default traffic on the eth0 interface is being captured. You will have to terminate the capture with CTRL + C once you have taken enough captures.
root@hcx-mgr:/home# tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:40:07.641494 ARP, Request who-has 10.10.0.1 (dc:a6:32:04:d2:56 (oui Unknown)) tell 10.10.0.10, length 46 12:40:07.641560 ARP, Reply 10.10.0.1 is-at dc:a6:32:04:d2:56 (oui Unknown), length 28 12:40:10.159445 IP 10.10.0.1.58760 > 18.104.22.168.1900: UDP, length 101 12:40:11.487595 IP 10.10.0.1.52766 > 10.10.255.255.32414: UDP, length 21 12:40:11.488865 IP 10.10.0.1.56177 > 10.10.255.255.32412: UDP, length 21 12:40:11.604242 IP 10.10.0.10.1900 > 22.214.171.124.1900: UDP, length 518 12:40:11.891715 IP 10.10.0.10.1900 > 126.96.36.199.1900: UDP, length 504 12:40:12.025215 IP 10.10.0.10.1900 > 188.8.131.52.1900: UDP, length 502 12:40:12.280478 IP 10.10.0.10.1900 > 184.108.40.206.1900: UDP, length 438 12:40:12.725948 IP 10.10.0.10.1900 > 220.127.116.11.1900: UDP, length 447
2. Capture from a specific interface
You can pass -i flag to specify the interface. The below example shows how to capture the packets from the wlan0 interface. You can also use any argument to capture packets on all the interfaces tcpdump -i any
root@hcx-mgr:/home# tcpdump -i wlan0
💡 You can pass -n flag to instruct tcpdump not to convert IP addresses to names.
root@hcx-mgr:/home# tcpdump -i wlan0 -n
3. Capture only a specific number of packets
There are thousands of packets that traverse the interface at any given moment so, you might get overwhelmed by the number of packets on the screen. You can pass -c flag to capture only a specific number of packets. Let's say I only want to capture 10 packets.
root@hcx-mgr:/home# tcpdump -i wlan0 -n -c 10
4. Capture traffic based on specific IP addresses
You can pass src or dst arguments to specify the IP addresses. For example, let's say you only want to capture the traffic destined to HCX IX appliance in target (10.26.99.101)
root@hcx-mgr:/home# tcpdump -i wlan0 -n dst 10.26.99.101
If you want to specify both source and destination IPs then ensure to pass the and keyword, for example tcpdump -i wlan0 -n src 192.168.0.5 and dst 10.26.99.101
5. Capture traffic based on ports
Using the previous example, you can pass port argument to specify ports and protocols. If you capture only ICMP traffic:
root@hcx-mgr:/home# tcpdump -i wlan0 -n icmp
You can also capture only UDP based traffic as shown below
root@hcx-mgr:/home# tcpdump -i wlan0 -n udp
You can also go wild and capture traffic based on packet flags. For example, TCP-SYN, TCP-ACK or ICMP echo-request or echo-reply.
root@hcx-mgr:/home# tcpdump -i wlan0 -n 'icmp[icmptype] = icmp-echoreply'
6. View and export the captures
You can pass -w flag to save the captures as a PCAP file. Use -s 0 to capture full-sized packets.
root@hcx-mgr:/home# tcpdump -i wlan0 -s 0 -n udp -w /home/pi/udp_capture.pcap
You can use -r flag to view the file locally.
root@hcx-mgr:/home# tcpdump -r /home/hcx/udp_capture.pcap You can also export the capture file (using FTP/SCP etc) and later view it on Wireshark.