Remove directory from Google Cloud Compute Engine using serial console

1
votes

The storage on my Google Cloud Compute Engine Instance (inadvertently) got completely used up. I can no longer SSH into my instance. I have increased the space on my instance, but it still will not allow me to SSH. My VM operating system is Ubuntu.

I have tried two things:

1.) I have tried creating a start script to remove a large directory to clear up some space. Under "custom metadata" I have set the key as "startup-script" and the value as 

#! /bin/bash
rm /home/myusername/dir-to-rm

I save and reset the instance, click "connect to serial console", but I still cannot SSH.

2.) I have tried to use a startup script to set a login username and password for the serial console by using the same method, but with the following script:

#! /bin/bash
sudo useradd tmpuser
echo 'tmpuser:password' | chpasswd
3
sshgoogle-cloud-platformgoogle-compute-engine
The easiest solution is to resize the disk in the Cloud Console. Then reboot the instance. The root file system will be resized on reboot automatically. Another option is to disconnect the boot drive, attach to another instance, mount to a directory and the cleanup space. - John Hanley
@JohnHanley Thanks! I have tried resizing the disk and reseting the VM, but I still have the same problem. To follow your second suggestion, are there any good resources/tutorials to do that, I am out of my depth trying to figure out how to do all that. - snarble
I wrote an article that explains the process for Debian. The same for Ubuntu. jhanley.com/google-cloud-debian-9-resize-root-file-system Follow my document. If you still have problems capture the console output that I show in my article so that I can tell you what is wrong. - John Hanley
Note: reboot the instance. Capture all of the serial port output. If you can provide me a link to download the output, I will look at the console trace. Do this if you still have problems after my article. - John Hanley

3 Answers

1
votes

I donated my time to help this person since he is a student. Total time to recover was about an hour.

The problem was caused by a corrupted file system. Repairing the file system corrected the problem. I then resized the disk larger to provide more free space.

During startup, the network interfaces failed to load. This, of course, prevents any form of connectivity. This is why SSH did not work.

1
votes

You can mount the disk in another instance, remove the directory and mount back.

Here are the details

  1. Stop your current instance
  2. Edit the instance, remove the disk from the instance.
  3. Create a new instance, and attach above disk as a second disk
  4. Start the new instance
  5. Ssh into the new instance
  6. Mount second disk mount /dev/sdb1 /mnt
  7. Remove the directory: rm /mnt/home/myusername/dir-to-rm
  8. umount the disk and stop the instance
  9. Attach the disk back to original instance as the boot disk
  10. Start
0
votes

I would suggest to create a snapshot from your VM instance [1] and then create a disk using the snapshot you created. After that, create a new instance using the newly created disk and try to ssh. Check if you can access all your files in the newly created instance via SSH.

[1] https://cloud.google.com/compute/docs/disks/create-snapshots

Hope this helps.