Website completely down after resizing boot disk VM (Google Cloud)

Question

I had to resize the boot disk of my Debian Linux VM from 10GB to 30GB because it was full. After doing just so and stopping/starting my instance it has become useless. I can't enter SSH and i can't access my application. The last backups where from 1 month ago and we will lose A LOT of work if i don't get this to work.

I have read pretty much everything on the internet about resizing disks and repartitioning tables, but nothing seems to work.

When running df -h i see:

Filesystem      Size  Used Avail Use% Mounted on
overlay          36G   30G  5.8G  84% /
tmpfs            64M     0   64M   0% /dev
tmpfs           848M     0  848M   0% /sys/fs/cgroup
/dev/sda1        36G   30G  5.8G  84% /root
/dev/sdb1       4.8G   11M  4.6G   1% /home
overlayfs       1.0M  128K  896K  13% /etc/ssh/keys
tmpfs           848M  744K  847M   1% /run/metrics
shm              64M     0   64M   0% /dev/shm
overlayfs       1.0M  128K  896K  13% /etc/ssh/ssh_host_dsa_key
tmpfs           848M     0  848M   0% /run/google/devshell

when running sudo lsblk i see:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda       8:0    0   40G  0 disk
├─sda1    8:1      35.9G  0 part /var/lib/docker
├─sda2    8:2    0   16M  1 part
├─sda3    8:3    0    2G  1 part
├─sda4    8:4    0   16M  0 part
├─sda5    8:5    0    2G  0 part
├─sda6    8:6       512B  0 part
├─sda7    8:7    0  512B  0 part
├─sda8    8:8        16M  0 part
├─sda9    8:9    0  512B  0 part
├─sda10   8:10   0  512B  0 part
├─sda11   8:11        8M  0 part
└─sda12   8:12   0   32M  0 part
sdb       8:16   0    5G  0 disk
└─sdb1    8:17   0    5G  0 part /home
zram0   253:0    0  768M  0 disk [SWAP]

Before increasing the disk size i did try to add a second disk and i even formatted and mounted it following the google cloud docs, then unmounted it again. (so i edited the fstab and fstab.backup etc..)

Nothing about resizing disks / repartition tables on the google cloud documentation worked for me.. The growpart, fdisk, resize2fs and many other StackOverflow posts did neither.

When trying to access through SSH i get the "Unable to connect on port 22" error as stated here on the google cloud docs

When creating a new Debian Linux instance with a new disk it works fine.

Anybody that can get this up and running for me without losing any data gets 100+ and a LOT OF LOVE......

Answer 1

I have tried to replicate the case scenario, but it didn't get me any VM instance issues.

I have created a VM instance with 10 GB of data and then Stopped it, increased the disk size to 30 GB and started the instance again. You mention that you can't ssh to the instance or access your application. After my test, I could ssh though and enter the instance. So there must be an issue of the procedure that you have followed that corrupted the VM instance or maybe the boot disk.

However there is a workaround to recover the files from the instance that you can't SSH to. I have tested it and it worked for me:

1. Go to Compute Engine page and then go to Images
2. Click on '[+] CREATE IMAGE'
3. Give that image a name and under Source select Disk
4. Under Source disk select the disk of the VM instance that you have resized.
5. Click on Save, if the VM of the disk is running, you will get an error. Either stop the VM instance first and do the same steps again or just select the box Keep instance running (not recommended). (I would recommend to stop it first, as also suggested by the error).
6. After you save the new created image. Select it and click on + CREATE INSTANCE
7. Give that instance a name and leave all of the settings as they are.
8. Under Boot Disk you make sure that you see the 30 GB size that you set up earlier when was increasing the disk size and the name should be the name of the image you created.
9. Click create and try to SSH to the newly created instance.
10. If all your files were preserved when you were resizing the disk, you should be able to access the latest ones you had before the corruption of the VM.

UPDATE 2nd WORKAROUND - ATTACH THE BOOT DISK AS SECONDARY TO ANOTHER VM INSTANCE

In order to attach the disk from the corrupted VM instance to a new GCE instance you will need to follow these steps :

1. Go to Compute Engine > Snapshots and click + CREATE SNAPSHOT.
2. Under Source disk, select the disk of the corrupted VM. Create the snapshot.
3. Go to Compute Engine > Disks and click + CREATE DISK.
4. Under Source type go to Snapshot and under Source snapshot chooce your new created snapshot from step 2. Create the disk.
5. Go to Compute Engine > VM instances and click + CREATE INSTANCE.
6. Leave ALL the set up as defult. Under Firewall enable Allo HTTP traffic and Allow HTTPS traffic.
7. Click on Management, security, disks, networking, sole tenancy
8. Click on Disks tab.
9. Click on + Attach existing disk and under Disk choose your new created disk. Create the new VM instnace.
10. SSH into the VM and run $ sudo lsblk
11. Check the device name of the newly attached disk and it’s primary partition (it will likely be /dev/sdb1)
12. Create a directory to mount the disk to: $ sudo mkdir -p /mnt/disks/mount
13. Mount the disk to the newly created directory $ sudo mount -o discard,defaults /dev/sdb1 /mnt/disks/mount
14. Then you should be able to load all the files from the disk. I have tested it myself and I could recover the files again from the old disk with this method.