Apache Ignite docker container with persistent volume on gcloud is not working well

Question

I am having difficulties with Apache ignite docker instance with persistent storage in gcloud.

The way I want to deploy an ignite docker instance in gcloud is when I push my code to github then it triggers a build and update new docker instance, it mounts and stores data in persistent volume automatically in Google cloud from what I created before. By the way I am not going to use kubernetes service for now because of some reasons with gcloud price.

I could deploy persistence-volume manually in console.

docker run -d -v persistence-volume:/persistence -e IGNITE_WORK_DIR=/persistence ignite-database:version

[What is working] Push code to github to trigger to build new instance and update in gcloud.

[What is not working] Automatically mount the persistent volume created before when I push the code to github.

I feel like gcloud command or cloud.yaml file are the ones I have to go through to fix it. It is not clear in what I have to do.

[cloud.yaml] (github triggers to build new update)

steps:
- name: 'gcr.io/kaniko-project/executor:v0.20.0'
  args:
  - --destination=gcr.io/$PROJECT_ID/database:latest
  - --cache=true
  - --cache-ttl=168h
- name: gcr.io/cloud-builders/gcloud
  args: [ compute, instances, update-container, database, --container-restart-policy=always, --zone=us-east1-a ]
timeout: 1200s

[Dockerfile]

# Start from a Java image.
FROM openjdk:8


# Ignite version
ENV IGNITE_VERSION 2.8.0
ENV JVM_OPTS -Xms512m -server -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -Djava.net.preferIPv4Stack=true


# Ignite home
ENV IGNITE_HOME /opt/ignite/apache-ignite-${IGNITE_VERSION}-bin

RUN apt-get update && apt-get install -y --no-install-recommends \
        unzip \
        wget \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /opt/ignite

RUN wget https://archive.apache.org/dist/ignite/2.8.0/apache-ignite-2.8.0-bin.zip \
    && unzip apache-ignite-${IGNITE_VERSION}-bin.zip \
    && rm apache-ignite-${IGNITE_VERSION}-bin.zip

COPY ./config/run.sh $IGNITE_HOME/
#COPY ./config/default-config.xml $IGNITE_HOME/apache-ignite@default-config.xml
#COPY ./config/persistence-config.xml $IGNITE_HOME/apache-ignite@persistence-config.xml
COPY ./config/control.sh $IGNITE_HOME/
COPY ./config/run_and_activate.sh $IGNITE_HOME/

#RUN sysctl start apache-ignite@default-config.xml
#RUN sysctl enable apache-ignite@persistence-config.xml

RUN chmod +x $IGNITE_HOME/run.sh
RUN chmod +x $IGNITE_HOME/control.sh
RUN chmod +x $IGNITE_HOME/run_and_activate.sh

CMD $IGNITE_HOME/run_and_activate.sh
EXPOSE 11211 47100 47500 49112

I am looking forward your advice.

Could you please share the errors you are getting when trying to automatically mount the persistent volume created before when I push the code to github? — pessolato
Actually there is no error appear with upper yml and dockerfile. It works without persistent volume mounted. — KeiTheNoop

DazWilkin DazWilkin · Accepted Answer · 2020-07-13T19:20:08

I'm unfamiliar with Kaniko and assume you're confident that your step works under Cloud Build (instructions here). You can confirm this by checking Container Registry for updated images.

I recommend not using :latest but perhaps your git commit (see e.g. ${COMMIT_SHA}).

- name: "gcr.io/kaniko-project/executor:v0.20.0"
  args:
  - --destination="gcr.io/$PROJECT_ID/database:${COMMIT_SHA}"
  ...

Changing "container-restart-policy` changes how the VM will react when the container fails. What you actually want to do is force the restart (with your new image). I recommend you try:

- name: gcr.io/cloud-builders/gcloud
  args:
  - compute
  - instances
  - update-container
  - database
  - --container-image="gcr.io/${PROJECT_ID}/database:${COMMIT_SHA}"
  ...

This should force the VM to restart the container and, because the SHA will change with every change to your git repo, this should always pull the updated image.

When you create the instance (the first time), you will:

mount persistent volumes (see link)
configure the env varz (see link).

When you update the VM you do not need to reapply these mounts or recreate the env variables as they will be preserved. You can change them if you wish (but I think you don't need to).

So, something of the form:

PD="persistence"
gcloud compute instances create-with-container ${INSTANCE} \
--container-image docker.io/busybox:1.27 \
--disk name=${PD} \
--container-mount-disk=mode="rw",mount-path="/disks/${PD}",name="${PD}" \
--container-env=IGNITE_WORK_DIR="/disks/${PD}" \
...

NOTE these are only needed during the VM create not update

Apache Ignite docker container with persistent volume on gcloud is not working well

1 Answers