Kubernetes Camel Spring applciation graceful shut down

Question

I have a spring boot camel application that processes messages on activeMq. however every once in a while, the ready probe fails and restarts the pod. Now thats fine, but then i get the applciation loggin out Inflight messages constantly and it gets into a restart cyclem due to ready probe constantly failing.

Is it possible to have kubernets allow the spring/camel app to gracefully shut down or consume any messages on the queue?

Smells more like a problem with your readiness check, why would it report a failure if Camel is still processing. And also maybe you need to tweak your probes to have higher timeouts or the likes to not be as sensitive. — Claus Ibsen
The readiness probe doesn't restart the application, the liveness probe does. The readiness probe puts the pod out of service. — dev_in_prog

Crou Crou · Accepted Answer · 2018-10-31T13:53:58

Using Container hooks, you can setup prestop that can look as follows:

apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-demo
spec:
  containers:
  - name: lifecycle-demo-container
    image: nginx
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
      preStop:
        exec:
          command: ["/usr/sbin/nginx","-s","quit"]

This hook is called immediately before a container is terminated. It is blocking, meaning it is synchronous, so it must complete before the call to delete the container can be sent. No parameters are passed to the handler.

You can follow the flow on Termination of Pods

User sends command to delete Pod, with default grace period (30s)

The Pod in the API server is updated with the time beyond which the Pod is considered “dead” along with the grace period.

Pod shows up as “Terminating” when listed in client commands

(simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.
If the pod has defined a preStop hook, it is invoked inside of the pod. If the preStop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.

The processes in the Pod are sent the TERM signal.

(simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.

When the grace period expires, any processes still running in the Pod are killed with SIGKILL.

The Kubelet will finish deleting the Pod on the API server by setting grace period 0 (immediate deletion). The Pod disappears from the API and is no longer visible from the client.

Kubernetes Camel Spring applciation graceful shut down

1 Answers