Unable de connect / start Gitlab EE on Google Compute Engine

Question

I deployed a gitlab EE instance omnibus on a Google Compute Engine instance following these steps: https://docs.gitlab.com/ee/install/google_cloud_platform/

Everything OK. I have updated my VM on ubuntu 18.04 and I restarted the instance, everything OK. But, when I've upgraded my VM to increase his capacity, a problem occured at system boot :

dockerd[1286]: time="2020-09-28T15:25:42.251633392+02:00" level=info msg="Starting up"
dockerd[1286]: time="2020-09-28T15:25:42.256850354+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
dockerd[1286]: time="2020-09-28T15:25:42.257123624+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
dockerd[1286]: time="2020-09-28T15:25:42.257177289+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
dockerd[1286]: time="2020-09-28T15:25:42.257196202+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
dockerd[1286]: time="2020-09-28T15:25:42.258723872+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
dockerd[1286]: time="2020-09-28T15:25:42.258801514+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
dockerd[1286]: time="2020-09-28T15:25:42.258922665+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
dockerd[1286]: time="2020-09-28T15:25:42.258947831+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
dockerd[1286]: time="2020-09-28T15:25:42.264233013+02:00" level=error msg="failed to mount overlay: no such device" storage-driver=overlay2
dockerd[1286]: time="2020-09-28T15:25:42.264281115+02:00" level=error msg="[graphdriver] prior storage driver overlay2 failed: driver not supported"
dockerd[1286]: failed to start daemon: error initializing graphdriver: driver not supported
systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: docker.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Docker Application Container Engine.

And, I am not unable to connect to the instance via SSH anymore (Error 4003 because back-end is not working, and then I got a timeout when I use SSH on port 22)...

I suppose that the Gitlab instance used Docker to work...

Do you hava any idea for this issue ?

Thank's a lot

Answer 1

I have followed the guide you have shared, and I was able to install a Gitlab instance without problems: After my installation I resized my instance and it continued working well.

Also, I can see that my Gitlab installation is working without Docker. You can install Gitlab with Docker but the guide you are following is not for this purpose.

But the problem you are having with your docker; looks like your docker is trying to use overlay2 without the proper backing filesystem, you could check this documentation for further information.

Now regarding the problem with your SSH, the error code 4003, this error could have many explanations:

1. Your instance can’t start the ssh service.
2. Your firewall is not configured properly to allow the port 22.
3. You have modified something in your instance and your metadata stopped working.

If your instance is running well,we could check points 1 and 2 using a port scanner, we could check tough port 22 for ssh if it is reachable. If it is not in green it means that it could be a problem in the firewall or with the SSH service.

If you have problems with your firewall, please check this documentation to create the proper rules.

If the problem is with your metadata it could be necessary to Install the guest environment in-place, you could validate if the guest environment is working by inspecting system logs emitted to the console while an instance starts up, please check this document of Validating the guest environment for more information.

In your case you should see something like Started Google Compute Engine Startup Scripts. because you are using Ubuntu, if your metadata is working properly.

Since you are not able to connect to the instance via SSH button, you have to enable the serial console of this vm instance to get into the command line of the vm instance.

Please try to create a startup-script to create a user and password and with this login in the VM from serial ports, the script would be like this:

1. Go to the VM instances page in Google Cloud Platform console.
2. Click on the instance for which you want to add a startup script.
3. Click the Edit button at the top of the page.
4. Click on Enable connecting to serial ports
5. Under Custom metadata, click Add item.
6. Set 'Key' to 'startup-script' and set 'Value' to this script:

#! /bin/bash 
useradd -G sudo <user>
echo '<user>:<password>' | chpasswd

7. Click Save and then click RESET on the top of the page. You might need to wait for some time for the instance to reboot.
8. Click on ”Connect to serial port” on the page.
9. In the new window, you might need to wait a bit and press Enter on your keyboard once; then, you should see the login prompt.
10. Login using the USERNAME and PASSWORD you provided and you will be logged.

If your metadata is not working properly, you should to install the guest environment in your VM:

1. Ensure that the version of your operating system is supported .
2. Enable the Universe repository. Canonical publishes packages for its guest environment to the Universe repository.

sudo apt-add-repository universe

3. Update package lists:

sudo apt update

4. Install the guest environment packages:

sudo apt install -y gce-compute-image-packages

5. Restart the instance and inspect its console log to make sure the guest environment loads as it starts back up.
6. Verify that you can connect to the instance using SSH.

I hope this information would be useful to you.