top of page
  • Writer's pictureRichard

VMware HCX Troubleshooting

  • Log Locations:

  • HCX: /common/logs/admin

  • Check in app.log for application events

  • Check in web.log for web service events

  • IX:

  • /var/log/vmware/hbrsrv*.log on IX appliances


 

Check Firewall Port Connectivity


Use this command to check whether the port is open between HCX Manager and ESXi hosts or Appliances


  • curl -vv telnet://<esxihost>:<port>

  • tcpdump -ni any port 4500

  • Nc -vu destination-IP 4500

  • Traceroute -U -p 4500 -s local and remote IP


Verify Network Connection for Download

  • tcpdump port 443


 


Performance


Perftest gives a report on the available bandwidth for the HCX tunnels. Hitting tab should give you a list of available commands.

In order to run "perftest"

  1. from HCM manager, run ccli by calling "ccli" as "root" user

  2. inside ccli context, list deployed appliances by command "list"

  3. from the list shown, select the appliance to run test. Each appliance has a "Id" field. select appliance by command "go <Id>".

  4. run perftest by calling "perftest site"

  5. run perftest with multiple flows with parameter "-P <num_of_flows>"

perftest all allows all tests to be run.


perftest uplink will test the links. use the -T flag to constrain by length of test, i.e. -T 1 will check one minute in each direction.



 

Health Check Commands


hc probe -d


Select an appliance and run a unit health check to check the state of the resources, services and tunnels.


go 2 (example)

hc unit all -d



 

HCX Connectivity Troubleshooting


Check the connectivity status from the HCX Interconnect:

  1. Check the Security associations are up: IPsec status

  2. Confirm the routing is appropriate: IP route

  3. Get the remote side IP: IP tunnel

  4. Try pinging the remote side (only if ICMP is allowed): ping <remote side IP>

  5. Confirm traffic on UDP 4500 is sent and received: tcpdump -s0 -n -i vNic_# udp


# tcpdump -c 10 -ni vNic_0 udp and port 4500


(note the vNic_# might be vNic_0 or another vNic, you can get the info via "ip route" and use that in the tcpdump command above)


[admin@HCX-MGR] go 0
Switched to node 0.
[admin@HCX-MGR] ssh
Welcome to HCX Central CLI
[root@<appliance0>] ip tun            
<status of gre tunnel>
[root@<appliance0>] tcpdump -c20 -i vNic_0 -n -e host <ip address of remote side>                                                                        



1. Start the capture


You can just run tcpdump command to start the capture. As you can see below, by default traffic on the eth0 interface is being captured. You will have to terminate the capture with CTRL + C once you have taken enough captures.

root@hcx-mgr:/home# tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:40:07.641494 ARP, Request who-has 10.10.0.1 (dc:a6:32:04:d2:56 (oui Unknown)) tell 10.10.0.10, length 46
12:40:07.641560 ARP, Reply 10.10.0.1 is-at dc:a6:32:04:d2:56 (oui Unknown), length 28
12:40:10.159445 IP 10.10.0.1.58760 > 239.255.255.250.1900: UDP, length 101
12:40:11.487595 IP 10.10.0.1.52766 > 10.10.255.255.32414: UDP, length 21
12:40:11.488865 IP 10.10.0.1.56177 > 10.10.255.255.32412: UDP, length 21
12:40:11.604242 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 518
12:40:11.891715 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 504
12:40:12.025215 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 502
12:40:12.280478 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 438
12:40:12.725948 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 447

2. Capture from a specific interface

You can pass -i flag to specify the interface. The below example shows how to capture the packets from the wlan0 interface. You can also use any argument to capture packets on all the interfaces tcpdump -i any

root@hcx-mgr:/home# tcpdump -i wlan0

💡 You can pass -n flag to instruct tcpdump not to convert IP addresses to names.

root@hcx-mgr:/home# tcpdump -i wlan0 -n

3. Capture only a specific number of packets

There are thousands of packets that traverse the interface at any given moment so, you might get overwhelmed by the number of packets on the screen. You can pass -c flag to capture only a specific number of packets. Let's say I only want to capture 10 packets.

root@hcx-mgr:/home# tcpdump -i wlan0 -n -c 10

4. Capture traffic based on specific IP addresses

You can pass src or dst arguments to specify the IP addresses. For example, let's say you only want to capture the traffic destined to HCX IX appliance in target (10.26.99.101)

root@hcx-mgr:/home# tcpdump -i wlan0 -n dst 10.26.99.101

If you want to specify both source and destination IPs then ensure to pass the and keyword, for example tcpdump -i wlan0 -n src 192.168.0.5 and dst 10.26.99.101

5. Capture traffic based on ports

Using the previous example, you can pass port argument to specify ports and protocols. If you capture only ICMP traffic:

root@hcx-mgr:/home# tcpdump -i wlan0 -n icmp

You can also capture only UDP based traffic as shown below

root@hcx-mgr:/home# tcpdump -i wlan0 -n udp

You can also go wild and capture traffic based on packet flags. For example, TCP-SYN, TCP-ACK or ICMP echo-request or echo-reply.

root@hcx-mgr:/home# tcpdump -i wlan0 -n 'icmp[icmptype] = icmp-echoreply'

6. View and export the captures

You can pass -w flag to save the captures as a PCAP file. Use -s 0 to capture full-sized packets.

root@hcx-mgr:/home# tcpdump -i wlan0 -s 0 -n udp -w /home/pi/udp_capture.pcap

You can use -r flag to view the file locally.

root@hcx-mgr:/home# tcpdump -r /home/hcx/udp_capture.pcap 
You can also export the capture file (using FTP/SCP etc) and later view it on Wireshark.
72 views0 comments

Recent Posts

See All
bottom of page