Pod receives traffic even Kubernetes readiness probe fails

Question

I have one application which servers for REST request and also is listening on a Kafka topic. I deployed the application to Kubernetes and configure the readiness probe like this

readinessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

basically following the instruction from [configure-liveness-readiness-startup-probes]

After deployment is done, I can see the pod readiness probe fails

Readiness probe failed: cat: can't open '/tmp/healthy': No such file or directory

That is expected. Then I sent a kafka message to the topic . I observed that

1) the kafka message has been consumed by my application and saved to database.
2) the rest api can't be accessed.

I assumed if the pod's readiness probe is failed, the application can neither receive kafka message nor the rest request. But why in my test, the REST request and Kafka message are handled differently.

According to the Kubernete documentation:

The kubelet uses readiness probes to know when a Container is ready to start accepting traffic

But it doesn't say clearly what kind of traffic it really means. Does kubernetes only restrict the http traffic to the pod if readiness probe failes but not restrict tcp traffic (as Kafka is working over tcp)?

My actual intention is to make my service application (kafka consumer) able to control when to receive kafka messages (and REST request as well). E.g. if there is heavy opertion, my service will delete the /tmp/healthy file and thus make the pod not ready for recieving kafka message and Rest request. When the heavy operation is finished, the app write the healthy file to make the pod ready for receiving message.

Some more information, in my test, the kubernetes version is v1.14.3 and kafka broker is running in a separated vm outside of kubernetes.

Answer 1

This is two very different things:

• Receiving requests: An external service is sending a request and expect a response.
• Sending requests: Your service is sending a request and waiting for a response.

ReadinessProbe

When a ReadinessProbe fails, no new requests will be routed to the pod.

Kafka consumer

If your pod is a Kafka consumer, then your pod is initializing requests to Kafka, to retrieve messages from the topic.

Check for required directory

can't open '/tmp/healthy': No such file or directory

If the directory /tmp/healthy is needed for your service to work correctly, your service should check for it on startup, and exit(1) (crash with an error message) if the required directory isn't available. This should be done before connecting to Kafka. If your application uses the directory continually, e.g. writing to it, any operations error codes should be checked and handled properly - log and crash depending on your situation.

Consuming Kafka messages

My actual intention is to make my service application (kafka consumer) able to control when to receive kafka messages ( and REST request as well). E.g. if there is heavy opertion, my service will delete the /tmp/healthy file and thus make the pod not ready for recieving kafka message and Rest request.

Kafka consumers poll Kafka for more data, whenever the consumer want. In other words, the Kafka consumer ask for more data whenever it is ready for more data.

Example consumer code:

 while (true) {
     ConsumerRecords<String, String> records = consumer.poll(100);
     for (ConsumerRecord<String, String> record : records) {
         // process your records
     }
 }

Remember to commit the records that you have processed so that the messages aren't processed multiple times e.g. after a crash.