After a bit of research, here is what I've come up with. If anyone has anything to add or correct, please feel free...
These heartbeat errors have nothing to do with vCenter. This can be seen by powering off the vCenter server and noting that these errors are still generated with the same random intervals.
These heartbeat errors are generated in ESXi and sent to OpenMange via the SNMP service in ESXi.
Most of the web info about heartbeat errors is all about vCenter heartbeat - that seems to be a different issue than what I’m seeing. No indication of these is in the vmware.log files within the VM. Everything I have found on the web about heartbeat timeouts seems completely irrelevant to my issue because it has nothing to do with vCenter.
I think this heartbeat lies between VMware tools running on the VM, and ESXi. Stopping VMware tools on the client will generate this trap. I also found that doing snapshots on VMs will often trigger this trap - which explains why I sometimes see it during Veeam backups. I could find no information on how to change the timeout value of this heartbeat.
The SNMP trap is called: vmHBLost
.1.3.6.1.4.1.6876.4.1
Generic = 6
Specific = 3
Sample: Message:Virtual machine detects a loss in guest heartbeat: vmID=3, vmConfigFile=/vmfs/volumes/StoragePath/VMpath/vm.vmx , vmDisplayName=servername
Some info on the web suggests this trap is deprecated. The only solution I can come up with is to ignore this trap.
I chose to change the e-mail alert settings in OpenManage so that it doesn’t e-mail when this trap is sent.
- Sign in to OpenManage Essentials as administrator
- Manage > Alerts > Alert Actions > E-mail
- Edit the action for “E-Mail for OpenManage Alerts”.
- Step through the config pages, and on the “category and sources association” page, deselect VMWare ESX Server > vmHBLost
Alternatively, it looks like the trap could be disabled in ESXi: