Google Pub/Sub + Cloud Run spawning multiple containers

Question

I'm looking for some help! I'm working on a small project that involves publishing a message to a Google Pub/Sub Topic and using a 'push' subscription to trigger a Cloud Run HTTPS service. The problem I'm having is that the Ack deadline on the 'push' subscription does not seem to be respected. I'm seeing the same message spawn multiple containers in the Cloud Run service even though the Ack deadline or request timeout has not been reached on the first run. I only want 1 message to spawn 1 container once to completion (return 200 OK).

Current Flow (Issue):

A message is published to pubsub topic
A 'push' subscription (with 600 second Ack deadline) forwards the message to a Cloud Run HTTPS service endpoint (with 600 second request timeout)
The Cloud Run Service starts processing this message, this work takes 120 seconds
While in processing, the same message re-triggers a new Cloud Run service container (every 10 seconds until first container completes), spawning multiple containers for the same message.
Once the first Cloud Run container completes, no new Cloud Run containers are launched and the message stops being received every 10 seconds.

Screenshot

Expected flow (What I want):

A message is published to pubsub topic
A 'push' subscription (with 600 second Ack deadline) forwards the message to a Cloud Run HTTPS service endpoint (with 600 second request timeout)
The Cloud Run Service processes this message for 120 seconds with only 1 container. Any NEW messages should launch on a separate container.
The Cloud Run service finishes processing within the 600 second Ack deadline and does not spawn new containers for the same message.

Subscription Description:

gcloud pubsub subscriptions describe myBuildSubscription
ackDeadlineSeconds: 600
expirationPolicy: {}
messageRetentionDuration: 87000s
name: projects/myproject-12345/subscriptions/myBuildSubscription
pushConfig:
  oidcToken:
    serviceAccountEmail: [email protected]
  pushEndpoint: https://my-project-example-ue.a.run.app
topic: projects/myproject-12345/topics/my_build_queue

Additional Info: I'm working based off this tutorial: https://cloud.google.com/run/docs/tutorials/pubsub

I added a sleep function to the Pub/Sub tutorial code in order to simulate the 120 seconds of work:

https://github.com/cvasq/golang-samples/commit/fa2286d4395d31b5eca12d73dd24187042211124

Current workaround: Set the maximum number of containers and max requests per container to 1. This allows processing only 1 message from the topic at a time.

Can anyone provide some pointers on why this might be happening? The config seems correct but I may be missing something.

There is no code in your question, so we can only guess. Make sure that your logs show the same message id being resent every 10 seconds. The most likely problem is that you are accidentally nack'ing the message. — John Hanley
Keep in mind with push subscriptions, there are no explicit ack/nacks. A success code is considered an ack, whereas a 4xx/5xx error code will result in the message being redelivered. I recommend checking the response of your invocations. — Alex Hong
Hi @JohnHanley, thanks for taking a look. There is a Github link to the code I was using to replicate the issue.This is it — cvasq

cvasq cvasq · Accepted Answer · 2020-03-03T00:22:26

I was able to resolve my issue by re-creating the pubsub subscription and the cloud run service at the command line with the desired deadline and concurrency configuration.

The first time I created the subscription and cloud run service at the command line and later changed the Ack deadline in the Google Cloud UI. I suspect the change didn't take on the Google Cloud backend for some reason.

gcloud pubsub subscriptions create mySubscription --topic projects/myproject-123456/topics/build_queue --push-endpoint=https://myservice-ue.a.run.app --push-auth-service-account=cloud-run-pubsub-invoker@myproject-123456.iam.gserviceaccount.com --ack-deadline 600


gcloud run deploy --memory 2Gi site-build --image gcr.io/myproject-123456/site-builder --concurrency 1 --max-instances 10

Google Pub/Sub + Cloud Run spawning multiple containers

1 Answers