Log Locations:
HCX: /common/logs/admin
Check in app.log for application events
Check in web.log for web service events
IX:
/var/log/vmware/hbrsrv*.log on IX appliances
Check Firewall Port Connectivity
Use this command to check whether the port is open between HCX Manager and ESXi hosts or Appliances
curl -vv telnet://<esxihost>:<port>
tcpdump -ni any port 4500
Nc -vu destination-IP 4500
Traceroute -U -p 4500 -s local and remote IP
Verify Network Connection for Download
tcpdump port 443
Performance
Perftest gives a report on the available bandwidth for the HCX tunnels. Hitting tab should give you a list of available commands.
In order to run "perftest"
from HCM manager, run ccli by calling "ccli" as "root" user
inside ccli context, list deployed appliances by command "list"
from the list shown, select the appliance to run test. Each appliance has a "Id" field. select appliance by command "go <Id>".
run perftest by calling "perftest site"
run perftest with multiple flows with parameter "-P <num_of_flows>"
perftest all allows all tests to be run.
perftest uplink will test the links. use the -T flag to constrain by length of test, i.e. -T 1 will check one minute in each direction.
Health Check Commands
hc probe -d
Select an appliance and run a unit health check to check the state of the resources, services and tunnels.
go 2 (example)
hc unit all -d
HCX Connectivity Troubleshooting
Check the connectivity status from the HCX Interconnect:
Check the Security associations are up: IPsec status
Confirm the routing is appropriate: IP route
Get the remote side IP: IP tunnel
Try pinging the remote side (only if ICMP is allowed): ping <remote side IP>
Confirm traffic on UDP 4500 is sent and received: tcpdump -s0 -n -i vNic_# udp
# tcpdump -c 10 -ni vNic_0 udp and port 4500
(note the vNic_# might be vNic_0 or another vNic, you can get the info via "ip route" and use that in the tcpdump command above)
[admin@HCX-MGR] go 0
Switched to node 0.
[admin@HCX-MGR] ssh
Welcome to HCX Central CLI
[root@<appliance0>] ip tun
<status of gre tunnel>
[root@<appliance0>] tcpdump -c20 -i vNic_0 -n -e host <ip address of remote side> 1. Start the capture
You can just run tcpdump command to start the capture. As you can see below, by default traffic on the eth0 interface is being captured. You will have to terminate the capture with CTRL + C once you have taken enough captures.
root@hcx-mgr:/home# tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:40:07.641494 ARP, Request who-has 10.10.0.1 (dc:a6:32:04:d2:56 (oui Unknown)) tell 10.10.0.10, length 46
12:40:07.641560 ARP, Reply 10.10.0.1 is-at dc:a6:32:04:d2:56 (oui Unknown), length 28
12:40:10.159445 IP 10.10.0.1.58760 > 239.255.255.250.1900: UDP, length 101
12:40:11.487595 IP 10.10.0.1.52766 > 10.10.255.255.32414: UDP, length 21
12:40:11.488865 IP 10.10.0.1.56177 > 10.10.255.255.32412: UDP, length 21
12:40:11.604242 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 518
12:40:11.891715 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 504
12:40:12.025215 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 502
12:40:12.280478 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 438
12:40:12.725948 IP 10.10.0.10.1900 > 239.255.255.250.1900: UDP, length 4472. Capture from a specific interface
You can pass -i flag to specify the interface. The below example shows how to capture the packets from the wlan0 interface. You can also use any argument to capture packets on all the interfaces tcpdump -i any
root@hcx-mgr:/home# tcpdump -i wlan0💡 You can pass -n flag to instruct tcpdump not to convert IP addresses to names.
root@hcx-mgr:/home# tcpdump -i wlan0 -n3. Capture only a specific number of packets
There are thousands of packets that traverse the interface at any given moment so, you might get overwhelmed by the number of packets on the screen. You can pass -c flag to capture only a specific number of packets. Let's say I only want to capture 10 packets.
root@hcx-mgr:/home# tcpdump -i wlan0 -n -c 104. Capture traffic based on specific IP addresses
You can pass src or dst arguments to specify the IP addresses. For example, let's say you only want to capture the traffic destined to HCX IX appliance in target (10.26.99.101)
root@hcx-mgr:/home# tcpdump -i wlan0 -n dst 10.26.99.101If you want to specify both source and destination IPs then ensure to pass the and keyword, for example tcpdump -i wlan0 -n src 192.168.0.5 and dst 10.26.99.101
5. Capture traffic based on ports
Using the previous example, you can pass port argument to specify ports and protocols. If you capture only ICMP traffic:
root@hcx-mgr:/home# tcpdump -i wlan0 -n icmpYou can also capture only UDP based traffic as shown below
root@hcx-mgr:/home# tcpdump -i wlan0 -n udpYou can also go wild and capture traffic based on packet flags. For example, TCP-SYN, TCP-ACK or ICMP echo-request or echo-reply.
root@hcx-mgr:/home# tcpdump -i wlan0 -n 'icmp[icmptype] = icmp-echoreply'6. View and export the captures
You can pass -w flag to save the captures as a PCAP file. Use -s 0 to capture full-sized packets.
root@hcx-mgr:/home# tcpdump -i wlan0 -s 0 -n udp -w /home/pi/udp_capture.pcapYou can use -r flag to view the file locally.
root@hcx-mgr:/home# tcpdump -r /home/hcx/udp_capture.pcap
You can also export the capture file (using FTP/SCP etc) and later view it on Wireshark.