For all users who back up VMware ESXi in version 6.0.x it is vitally important to know that a serious bug has unfortunately been introduced by VMware in this version, a bug that has to do with a major problem on Changed Block Tracking (CBT). Here is the link to the official VMware article: https://kb.vmware.com/s/article/2136854.
To better understand the problem, here is a brief explanation of what happens when Iperius Backup performs a backup of an ESXi virtual machine with the CBT mode :
- The virtual machine configuration is saved.
- A snapshot of the virtual machine is done.
- The ESXi host is asked what are the sectors used by the virtual machine disks (for the first backup), or the different sectors (from the previous backup).
- The download of disk sectors, returned by the previous step, is started.
As stated by VMware, in version 6.0.x of ESXi it may happen that sometimes, when a software asks about the disk sectors to be saved, the host responds incorrectly.
This means that the third phase of the backup generates an inconsistent result. Clearly this is not a Iperius bug, but a VMware bug, which therefore affects any software backing up virtual machines using incremental / differential (CBT) mode.
The final result of the backup then will be a non-coherent disk image and therefore the impossibility to restore the virtual machine correctly.
To overcome this serious problem, you must then download and install the patch made available by VMware: https://kb.vmware.com/s/article/2137545
To check if your backups have been affected by this bug, an important verification is to try a restore. You can do that with Iperius, restoring a VM backup to a new test machine, to see if the restore works fine. If yes, you can consider your backup consistent and correct. Otherwise, you immediately need to make a new backup from the scratch after you installed the VMware patch.
An in-depth look at the Changed Block Tracking (CBT):
Changed Block Tracking (CBT) is a feature that allows you to perform incremental and differential backups of VMware virtual machines.
Virtual machines running on ESX / ESXi hosts can keep track of the disk sectors that have changed. This function is called Changed Block Tracking (CBT). On many file systems, CBT identifies changed disk sectors between two sets of change IDs. In VMFS partitions, CBT can also identify all sectors of the disk in use.
When a software makes a backup, it may request transmission only of the blocks modified since the last backup or blocks actually used on the virtual disk. The CBT functionality is accessible through any third-party applications as part of the vSphere Data Protection API (VADP). Applications call VADP to request that the VMkernel return data blocks that have changed on a virtual disk since the last backup snapshot.
To make sure the CBT can identify the disk sectors modified since the last change ID, the following requirements are needed:
- The host must be ESX / ESXi 4.0 or later.
- The virtual machine that owns the disks to be traced must be hardware version 7 or later.
- I/O operations must go through the ESX / ESXi storage stack. So NFS is supported, like RDM in virtual compatibility mode, but not RDM in physical compatibility mode. VMFS is supported, whether supported by SAN, iSCSI or local disk.
- CBT must be enabled for the virtual machine (see below).
- Virtual machine storage does not have to be an independent (persistent or non-persistent) disk, because it is not possible to take a snapshot of independent disks.
In order for the CBT to identify the sectors of the disk in use with the special change ID ‘*’, the following requirements are needed:
- The virtual disk must be on a VMFS volume, SAN, iSCSI, or local disk.
- The virtual machine must have zero (0) snapshots when CBT is enabled, for a clean start.
In some cases, such as when a power outage or forced shutdown of virtual machines occurs, the CBT could be reset to zero and lose track of incremental changes. In vSphere 4.1 and earlier, Cold Migration (but not Storage vMotion) could restore but not disable CBT. In vSphere 5.x versions prior to vSphere 5.5 Update 2, vMotion storage restores the CBT.
To check if a virtual disk has the CBT enabled:
Open the vSphere client and select a virtual machine. The machine must be turned off.
Right-click the virtual machine and select Edit Settings> Options> Advanced / General> Configuration Parameters.
The configuration file (.vmx) of the virtual machine contains the entry:
ctkEnabled = ‘TRUE’
Note: set the value to False to disable CBT.
For each virtual disk, the .vmx file contains the entry:
scsix: x.ctkEnabled = ‘TRUE’
A .ctk file exists for each virtual disk and snapshot disk.
For example:
vmname.vmdk
vmname-flat.vmdk
vmname-ctk.vmdk
vmname-000001.vmdk
vmname-000001-delta.vmdk
vmname-000001-ctk.vmdk
Note: After a successful backup and full consolidation of snapshots, there should not be any .ctk files for snapshots left in the data store. For example: vmname-000001-ctk.vmdk.
PLEASE NOTE: if you need technical support or have any sales or technical question, don't use comments. Instead open a TICKET here: https://www.iperiusbackup.com/contact.aspx
**********************************************************************************
PLEASE NOTE: if you need technical support or have any sales or technical question, don't use comments. Instead open a TICKET here: https://www.iperiusbackup.com/contact.aspx
*****************************************