Use gcloud metadata-from-file shutdown-script to stop docker container gracefully

Question

I have created gcloud compute instance from docker image and configured it to launch shutdown script which should call docker stop in order to shut down the app in the container gracefully.

gcloud beta compute instances create-with-container mycontainername \
    --container-image ypapax/trap_exit \
    --metadata-from-file shutdown-script=./on_shutdown.sh

And here is my initital on_shutdown.sh:

#!/usr/bin/env bash
docker stop $(docker ps -a -q)

Although, I added more debugging lines to it and now on_shutdown.sh looks like:

#!/usr/bin/env bash
# https://cloud.google.com/compute/docs/shutdownscript
curl -X POST -d "$(date) running containers on $(hostname): $(docker ps)" http://trap_exit.requestcatcher.com/test
docker stop $(docker ps -a -q)
result=$?
curl -X POST -d "$(date) $(hostname) stop status code: $result" http://trap_exit.requestcatcher.com/test

When I reboot the google compute instance:

sudo reboot

The script on_shutdown.sh is launched (I see it checking requrest listener). But when it tries to stop docker container, there is nothing to stop yet, docker ps shows empty line.

So this line:

curl -X POST -d "$(date) running containers on $(hostname): $(docker ps)" http://trap_exit.requestcatcher.com/test

gives me

Thu Jul 12 04:29:48 UTC 2018 running containers on myinstance:

Before calling sudo reboot I checked docker ps and saw my container running:

$ docker ps     
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
bbaba30e65ff        ypapax/trap_exit    "/entrypoint.sh"    7 seconds ago       Up 7 seconds                            myinstance

So looks like docker container is killed between calling reboot and launching on_shutdown.sh. The problem is that killing doesn't call trap cleanup EXIT in my entrypoint. It needs to be stopped in order to call the cleanup. Here is my entry point:

#!/usr/bin/env bash
set -ex

cleanup(){
    echo cleanup is called
    curl -X POST -d "cleanup is called on $(hostname) $(date)" http://trap_exit.requestcatcher.com/test
}

trap cleanup EXIT
curl -X POST -d "container is started on $(hostname) $(date)" http://trap_exit.requestcatcher.com/test
sleep 10000

So I would like to run my container's cleanup on gcloud compute instance reboot or shutdown but flag --metadata-from-file shutdown-script=./on_shutdown.sh doesn't help to do it. I also tried other methods to call a script on reboot like this. But my script hadn't been launched at all.

Here is my Dockerfile if it could help.

VonC VonC · Accepted Answer · 2018-07-15T04:32:57

First, there are limitations coming with this approach:

Create and run shutdown scripts that execute commands right before an instance is terminated or restarted, on a best-effort basis.

Shutdown scripts are especially useful for instances in a managed instance group with an autoscaler.

The script runs during the limited shutdown period before the instance stops

As you have seen, docker might already have stopped by the time the shutdown script run: check with docker ps -a (instead of docker ps) to see the status of all exited containers. Try adding a supervisor (as in this example) a docker image itself, in order to see if the supervisor, or at least use the docker run --init option: the goal is to check if the containers themselves do use their supervisor scripts.

Use gcloud metadata-from-file shutdown-script to stop docker container gracefully

1 Answers