A colleague was moving a VM from one datastore to another datastore. Somehow in this process something has gone wrong.
All files are there except the snapshot information .vmsd file which i recreated. I checked the logs and can see that there are 2 snapshot delta files which i have.
Without the snapshots the machine starts normally but i need to revert to the latest snapshot.
After recreating the config vmdk files and snapshot (chain) info the snapshots show up in ESXI but reverting to a snapshot gives the following error
2016-12-27T09:38:09.269Z [33BC1B70 verbose 'Vmsvc.vm:/vmfs/volumes/58621528-51cb7ea2-6b52-d89d6729d8dc/Development new/Development new.vmx'] Power On message: Input/output error
--> Cannot open the disk '/vmfs/volumes/58621528-51cb7ea2-6b52-d89d6729d8dc/Development new/Development new-000002.vmdk' or one of the snapshot disks it depends on.
File structure looks like this
Developmentnew-000001-delta.vmdk 2.4G
Developmentnew-000001.vmdk 352
Developmentnew-000002-delta.vmdk 26.1G
Developmentnew-000002.vmdk 339
Developmentnew-flat.vmdk 64.0G
Developmentnew.nvram 8.5K
Developmentnew.vmdk 476
Developmentnew.vmsd 1.5K
Developmentnew.vmx 2.9K
Developmentnew.vmxf 269
If i look at the binary delta files they just look normally and also ends normally with zero's.
Snapshot 001 Binary headers
COWD € w)¥
Snapshot 002 Binary headers
COWD € Ë<D
Things i tried to recover the snapshot
- Cloning (using vmkfstools -i) the snapshot returns same Input/output error on both snapshots.
- Repair/fix using vmkfstools returns no errors
- Adding another snapshot and revert returns same errors (cloning still not possible)
- Deleting snapshot also returns error
- Deleting all snapshots needs snapshot consolidation but this fails.
- Using VMWare Workstation to start snapshot. But VMWare crashes with memory issues.
- Using vmware-vdiskmanager to repair returns the following
vmware-vdiskmanager.exe -R "g:\Dev3\Development new-000002.vmdk"
VixDiskLib: Invalid configuration file parameter. Failed to read configuration file.
DISKLIB-LEGCHK: Ignoring holes in extent. --Ok
No errors were found on the virtual disk, 'g:\Dev3\Development new-000002.vmdk'.
vmware-vdiskmanager.exe -R "g:\Dev3\Development new-000001.vmdk"
VixDiskLib: Invalid configuration file parameter. Failed to read configuration file.
DISKLIB-LEGCHK: Ignoring holes in extent. --Ok
The virtual disk, 'g:\Dev3\Development new-000001.vmdk', is corrupted and cannot be repaired.
vmware-vdiskmanager.exe -R "g:\Dev3\Development new.vmdk"
VixDiskLib: Invalid configuration file parameter. Failed to read configuration file.
No errors were found on the virtual disk, 'g:\Dev3\Development new.vmdk'.
- Using DiskInternals VmfsRecovery which is able to read the snapshot partially but no path information.
- Trying to recover lost data on the ESXI machine filesystem but failed as there was no diskspace already on it.
- Using mount tools to mount the snapshot fails with read error.
Following is situation after adding more snapshots but 1 and 2 are still the same.
vmkfstools -t0 -v10 "Development new-000004.vmdk"
DISKLIB-VMFS : "/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000004-delta.vmdk" : open successful (14) size = 135168, hd = 499227139. Type 8
DISKLIB-DSCPTR: Opened [0]: "Development new-000004-delta.vmdk" (0xe)
DISKLIB-LINK : Opened '/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000004.vmdk' (0xe): vmfsSparse, 134217728 sectors / 64 GB.
DISKLIB-VMFS : "/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000002-delta.vmdk" : open successful (14) size = 28068417536, hd = 491887108. Type 8
DISKLIB-DSCPTR: Opened [0]: "Development new-000002-delta.vmdk" (0xe)
DISKLIB-LINK : Opened '/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000002.vmdk' (0xe): vmfsSparse, 134217728 sectors / 64 GB.
DISKLIB-VMFS : "/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000001-delta.vmdk" : open successful (14) size = 2599952384, hd = 646945285. Type 8
DISKLIB-DSCPTR: Opened [0]: "Development new-000001-delta.vmdk" (0xe)
DISKLIB-LINK : Opened '/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000001.vmdk' (0xe): vmfsSparse, 134217728 sectors / 64 GB.
DISKLIB-VMFS : "/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-flat.vmdk" : open successful (14) size = 68719476736, hd = 671259142. Type 3
DISKLIB-DSCPTR: Opened [0]: "Development new-flat.vmdk" (0xe)
DISKLIB-LINK : Opened '/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new.vmdk' (0xe): vmfs, 134217728 sectors / 64 GB.
DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 4, numSubChains = 1
DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 671259142, extentType = 2
DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 646945285, extentType = 0
DISKLIB-CHAINESX : ChainESXOpenSubChain:(2) fid = 491887108, extentType = 0
DISKLIB-CHAINESX : ChainESXOpenSubChain:(3) fid = 499227139, extentType = 0
DISKLIB-CHAINESX : ChainESXOpenSubChainNode: can't create deltadisks node 48423dbe-Development new-000004-delta.vmdk failed with error Input/output error (0xbad000a, I/O error)
DISKLIB-CHAIN : "/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000004.vmdk" : failed to open (Input/output error).
DISKLIB-VMFS : "/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000004-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000002-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/574dbe3f-e8ed28ab-bab5-a02bb8317488/Development/Development new-flat.vmdk" : closed.
DISKLIB-LIB : Failed to open 'Development new-000004.vmdk' with flags 0xe Input/output error (327689).
Failed to open virtual disk: Input/output error 327689
So as you can see i tried many things to recover/repair the snapshota but I'm out of options. Are there any other things i can try?
Or can i conclude that the moving operation corrupted the binary data in the process.