Now this is actually one of those obvious and helpful error messages that you get every now and then and just by looking at the error message I could see what had caused this issue.
We use Backup Exec with the avvi agent to perform backups of some of our production vms.
The avvi agent allows us to perform SAN to Tape backups off host which means we don't need to do anything special with regards to backup configuration on any of our ESXi hosts. The configuration process is a simple of case enable the option within the Backup Exec media servers and present the ESX Datastores to them (with the same LUN IDs etc) and that's pretty much it. Most of our vms are also running the Backup Exec Remote Agent for it's OS (Windows or Linux) which then allows us to have granular file recovery from our image based backups which is a nice feature...although not as useful when doing your backups to tape and not disk as the recovery process still needs to extract the full vmdk off of the tape before recovering the individual files to be restored to the vm or elsewhere.
A good guide for setting this configuration up can be found on Symantec's website here:
Now what usually happens when a backup job is run on a vm using this method is this:
· The BE job starts on the media server and talks to vCenter to take a snapshot of the vms vmdk
· Once completed the vm is now running from the snapshot and the original vmdk is static and only read by the vm
· BE then gets the ESXi host and guest virtual machine information from vCenter it needs to backup
· BE then opens a connection with the ESXi server to ask for the virtual machine metadata
· BE then informs vCenter to disable Storage vMotion for that VM to ensure that the backups can complete successfully.
· Using vStorage APIs, Backup Exec then opens a direct data connection to the ‘unknown’ SAN volumes which have been presented to it and the virtual machine data is offloaded directly to the media server for backup
· Once the backup process has completed the snapshot is deleted and BE disconnects from the ESXi host and informs vCenter to enable Storage vMotion again for the vm
· Backup job then completes.
The error above is caused by the Storage vMotion being disabled by Backup Exec to run the backups. After the backup job completes the call to vCenter does not get made or fails and so the vm is stuck with it's Storage vMotion disabled.
The trouble with this is that you often don't know this is an issue until you go to perform a Storage vMotion or unless you have vms inside an SDRS cluster and they fail to migrate to other datastores.
You can however identify these vms though by performing a lookup within the vCenter database as described in this VMware KB article:
Luckily this is a known issue and there are two very easy ways to address this if you have this issue.
The first, and often easiest way, is to shutdown the vm and remove it from the inventory. Then browse thedatastore where it resides, locate the vmx file and add it to the inventory again.
This approach basically gives the vm a new id within vcenter and thus gets any customised settings removed allowing it to SvMotion again.
This does pose an issue however in that you will need downtime on your vm, although very short, in order to resolve this.
The other approach, as detailed by VMware in the KB above, is to manually edit the settings within the vCenter DB for the vm affected. Whilst this does not require a vm outage to work, it does require vCenter to be stopped whilst you access the DB and in some instances (Environments with vCloud Director, SRM, LabManager etc) this is more impacting than 1 vm being shutdown for a couple of mins and so finding a quiet evening or weekend to shut the vm down is my preferred approach and this can be easily scripted anyway to save those long hours from building up!
This is not restricted to Symantec btw. I have seen this issue with VEEAM backup software also and as yet I'm not aware of any definitive solution to prevent this from happening from time to time. It pays to keep an eye on this if you are running a similar backup technology in your environment.