Google Cloud Pub/Sub push subscription retries the message delivery during the message handling

Question

I'm running a Java 8 servlet based application as a Google Cloud Pub/Sub push endpoint. The ack deadline is 600 seconds. The server is embedded Jetty 9.4.

Say I have a message M that I know to be published to the topic with my push subscription.

My endpoint takes a long time to handle the message before acking it. It executes a script (as a subprocess using Java ProcessBuilder class) that sometimes can run for a long time. Let's say I receive the message M at 8:00:00 AM. My script starts running, and the request sent by the push subscription is still active. Then, at 8:00:30 AM I receive that same message M, even though I have not returned a 200 or 500 response to a previous request.

When I test my app locally, I am able to post a 200 response after 40 or more seconds.

How can I fix this? I know I probably should be using a pull subscription, but the behavior that I observe is not documented, or at least I couldn't find it.

Answer 1

The ack deadline is not a guarantee that the message will not be redelivered in that time period; it is best effort. With long-standing push requests, the message can be redelivered if the server thinks the outgoing request has terminated, which is more likely the longer the request is active. It could be that the connection to your server is flaky or perhaps be some kind of server-side deadline that is responding to the request at a higher level than your code for processing.

You could look at the Stackdriver metrics for the push subscription, in particular, look at subscription/push_request_count, which you can break down by response and see if Cloud Pub/Sub thinks that error responses are being returned.

Answer 2

Turns out, the GCP load balancer that was between my instance that served the push endpoint had a 30 sec timeout.