Windows Server 2012 VM created from Snapshot fails to Boot on GCE

Question

I am not a Windows expert and stuck while recovering Windows Server 2012 from Snapshot. I am trying to create a new VM instance on Google Compute Engine - GCE from a Snapshot which I created from Windows Server 2012 VM Instance couple of weeks ago . Whenever I create a new VM I am never able to do RDP then after reading GCE Troubleshooting guide I determined that may be Windows is not booting up properly. I was able to view the Serial Port Output as following

SeaBIOS (version 1.8.2-20161003_105447-google)
Total RAM Size = 0x00000003c0000000 = 15360 MiB
CPUs found: 4     Max CPUs supported: 4
found virtio-scsi at 0:3
virtio-scsi vendor='Google' product='PersistentDisk' rev='1' type=0 removable=0
virtio-scsi blksize=512 sectors=314572800 = 153600 MiB
drive 0x000f3120: PCHS=0/0/0 translation=lba LCHS=1024/255/63 s=314572800
Booting from Hard Disk 0...

Its stuck at Booting from Hard Disk 0...

I dig further into it and read the Serial Port 2 log as following

Windows Boot Manager
Windows failed to start. A recent hardware or software change might be the cause. To fix the problem: 

Insert your Windows installation disc and restart your computer.
Choose your language settings, and then click "Next."
Click "Repair your computer.
If you do not have this disc, contact your system administrator or computer manufacturer for assistance. 

Status: 
A required device isn't connected or can't be accessed.  

I attached the disk to another machine and I can see all files on the disk. But can't modify any thing as the disk is write-protected.

Original VM from which I created Snapshot is still there so I tried to attach the snapshot to original VM instance as Boot Disk in order to have same hardware configuration but I can't detach the disk associated with that instance and it gives me following error.

Hot-remove of the root disk is not supported.

I tried creating VM with same Machine Type as before taking the Snapshot. But failed as well.

Any suggestions on how I can create new VM instance from Snapshot and boot Windows properly?

Answer 1

After going through number of articles, forums and user guides, finally I am able to spin the Windows VM Instance successfully from Snapshot.

The issue was with bootloader and Boot Configuration Data (BCD). Ideally you use Images for OS Disk. In my case OS and data were in same disk and we just had a Snapshot. GCE allows creating new VM instance from Snapshot but in my case the instance was not booting up.

Follow the step by step guide to recover your OS/Data from snapshot.

Summary:

Create Disk from Snapshot and fix the disk BCD using new temporary VM Instance.

Detailed Steps

Step 1: Create Recovery VM Instance and Start it

This instance is temporary instance and you may delete it after you recover your OS / data.

From Google Cloud Console

Select Compute Engine > VM Instances and select CREATE INSTANCE

Make sure you select the same OS as the Snapshot. Once started, make sure you can do remote desktop and login to new VM instance.

Note down instance name and zone in which the instance is running.

Step 2: Create New Disk from Snapshot

From Google Cloud Console

Select Compute Engine > Disks and select CREATE DISK

Make Sure you select the same Disk size / Disk Type as on/before taking snapshot else Windows may throw boot error. Also make sure the disk is in same zone as your recovery instance. If your disk is not in same zone as in your instance then you won't be able to attach it.

Step 3: Attach the Disk to Recovery Instance

In this step you attach the disk you (created in Step-2) to VM instance (created in Step-1)

Open Google Cloud Shell and type the following command

gcloud compute instances attach-disk [INSTANCE-NAME] --disk [DISK-NAME] --zone [ZONE]

Replace the variables with your Instance name, Disk name and Zone in which you are running the instance.

Step 4: Mount Disk and assign Drive letter in Windows

Go to Start > Search and type diskmgmt.msc to open the Disk Management tool. If the disk you just attached, shows up as Offline, right-click on it and select Online.

After making sure the disk is Online, verify that each volume on the disk has a drive letter assigned. The specific drive letters assigned is not important. If any of the volumes do not have a drive letter assignment, right-click the volume and select Change Drive Letter and Paths, then Add. Select Assign the following drive letter, let it choose the next available drive letter, then click OK. Again, the actual drive letters used doesn’t matter.

Note down the Drive letter. For me its D: drive.

Step 6: Remove write-protection from disk

Try creating a new folder in your attached drive. If your disk is write-protected and you are not able to create any file or folder in the drive then you need to turn-off the write-protection, otherwise you can skip this step.

Open Elevated Command Prompt (run as Administrator) and type

diskpart 

and you will get DISKPART> prompt

Type:

list volume 

System will display all volumes with number. Next you need to select the volume by typing:

select volume # 

where # is the volume number. For me it's 1.

Then type following commands remove write-protection

attr disk clear readonly 
attr volume clear readonly
attr volume clear hidden
attr volume clear shadowcopy

Exit diskpart by typing exit or closing command prompt window. Open the drive in Windows Explorer. You should be able to see all your data and Windows system files. Create a new folder in the drive to make sure disk is not write-protected.

Step 7: Fix Boot Configuration Data (BCD)

If you are familiar with Windows bcedit command, then by all means use bcedit, but I used EasyBCD for fixing the boot configuration data.

Download and install EasyBCD on your Recovery VM Instance from https://neosmart.net/EasyBCD

Once installed open EasyBCD and click on

File > Select BCD Store

and file selection dialog under filename enter D:\Boot\BCD or whatever drive letter you assigned in Step 5. The system will show you Boot Configuration Data for your drive.

Click on Edit Boot Menu button and select Skip the boot menu and click on Save settings.

Click on Advanced Settings button and under the Basic tab click on Drive: menu and select the Drive letter of the disk.

Please note: The drive letter should be the same as Step-5

Click on BCD Backup/Repair button and under the BCD management options select Re-create/repair boot files and click on Perform Action button.

Make the disk offline by Opening Disk Management again and right-click the disk and select Offline.

Now minimize your RDP window and in Google Cloud Shell type following command to detach the disk from Recovery instance

gcloud compute instances detach-disk [INSTANCE-NAME] --disk [DISK-NAME] --zone [ZONE]

Now you have fixed the Boot Configuration Data of the Disk created from Snapshot.

We are now ready to spin the VM Instance and boot it using this disk. Let's create instance from disk

Step 8: Create new VM Instance

From Google Cloud Console select Compute Engine > VM Instances and select CREATE INSTANCE

In order to avoid any problem at first run, make sure you select the same Machine Type as on/before the snapshot creation.

At this point you should be able to have working VM instance and you should be able to do RDP login.

In case you are still facing any issues, then have a look at your serial port log by clicking the VM instance in Google Cloud Console and then scroll down to bottom of the page or you can type the following command in Google Cloud Shell.

gcloud compute instances get-serial-port-output [INSTANCE-NAME] --zone [ZONE]

Lesson Learned

• For OS disks with / without data, use images instead of snapshots.
• Do not keep your data in same disk as OS even if it's a test machine and you are doing some temporary work.

Answer 2

Thanks for the detailed the answer. The key issue here is that Windows will mark the volume as read-only/snapshot when you take a VSS initiated snapshot. GCE Persistent Disk Snapshot will preserve that flag. All the data is correct and consistent. All you need to do is to remove that flags as step1-step6 outlined.

A small suggestion for step7. You can simply use bulit-in tool bcdboot.exe to fix it.

bcdboot.exe D:\Windows /s D:

(BCD Hive is actually all right, however, the disk signature in MBR is modified by Windows when you manually online a disk. That's why we need to run the command here to bring it back to be consistent with boot database).

Answer 3

@Shaheryar's answer is awesome!! If you need to automate / script some of the steps, you can use the following:

Step 4 and 6

The disk can be made online and read flags cleared by putting the following in a mount.txt file:

rem This script is mean to be run with diskpart /s mount.txt
rem It mounts the snapshot volume to D:/ and makes it writeable

list disk

rem The snapshot disk should be on disk #1
rem Use `list disk` to find it if it changed - also update unmount.txt

select disk 1
online disk

rem The partition should be on volume #2
rem Use `list volume` to find it if it changed - also update unmount.txt

select volume 2
assign letter=D

attr disk clear readonly
attr volume clear readonly
attr volume clear hidden
attr volume clear shadowcopy

exit

From an admin console/shell, run this file using

diskpart /s mount.txt

Step 7

The GUI steps to fix the BCD can be replaced with the following commands in an admin Powershell session:

attrib -h -s D:\Boot\BCD
Remove-Item -Path D:\Boot\BCD

# Recreate BCD
bcdedit.exe /createstore D:\Boot\BCD
bcdedit.exe /store D:\Boot\BCD /create "{bootmgr}" /d "Windows Boot Manager"
bcdedit.exe /store D:\Boot\BCD /set "{bootmgr}" device partition=D:
bcdedit.exe /store D:\Boot\BCD /set "{bootmgr}" displaybootmenu No
bcdedit.exe /store D:\Boot\BCD /timeout 0

$out = bcdedit.exe /store D:\Boot\BCD /create /d "Microsoft Server" /application osloader
$id = Select-String -Input $out -Pattern '{.+}'  -AllMatches | % { $_.Matches } | % { $_.Value }

# Output from above
bcdedit.exe /store D:\Boot\BCD /set "$id" device partition=D:
bcdedit.exe /store D:\Boot\BCD /set "$id" osdevice partition=D:
bcdedit.exe /store D:\Boot\BCD /set "$id" path \Windows\system32\winload.exe
bcdedit.exe /store D:\Boot\BCD /set "$id" systemroot \Windows
bcdedit.exe /store D:\Boot\BCD /set "$id" nx OptOut
bcdedit.exe /store D:\Boot\BCD /set "$id" allowedinmemorysettings 0x15000075
bcdedit.exe /store D:\Boot\BCD /set "$id" inherit "{6efb52bf-1766-41db-a6b3-0ee5eff72bd7}"
bcdedit.exe /store D:\Boot\BCD /set "$id" resumeobject "{45e743a4-934e-11e9-a872-8c17a054efa7}"

bcdedit.exe /store D:\Boot\BCD /displayorder "$id"

# Print result
bcdedit.exe /store D:\Boot\BCD /enum /v

These steps are adopted from the creators of EasyBCD. The missing steps were found by trial-and-error to match the resulting file the EasyBCD GUI will create.

Missing unmount step

You might also want to unmount the volume before detaching it. This can be done by putting the following in an unmount.txt file:

rem This script unmounts the snapshot after its bootloader has been fixed

rem See mount.txt for why volume 2
select volume 2
remove

rem See mount.txt for why disk 1
select disk 1
offline disk

exit

Finally, run it using an admin console/shell

diskpart /s mount.txt