gcp cloud function pub/sub topic dead letter

Question

I am trying to achieve the following on gcp using terraform.

A cloud function listens to messages added to a pub/sub topic
Once a message is added the cloud function is triggered
If there is an error in processing the message the message is put onto a corresponding dead letter queue.

I'm not sure how to create the subscription for the cloud function which also contains the dead letter policy. At the moment I'm doing the following for the cloud function,

resource "google_cloudfunctions_function" "brw-user-function-item-registered" {
 // details

  event_trigger {
    event_type = "google.pubsub.topic.publish"
    resource   = "brw-messages"
    failure_policy {
      retry = false
    }
  }

 // details     
}

However there is no option to specify the dead-letter policy in the event_trigger section. If I create a separate google_pubsub_subscription I'm not sure how to give the endpoint to the cloud function. I had a look at what gets created and its actually an endpoint, however I'm not sure how to specify those details in the google_pubsub_subscription

Even though you can use terraform import, to import the google_pubsub_subscription automatically created by your cloud function and then assign it a dead_letter_policy with the same push_config.push_endpoint, the dead letter policy will have no effect because you Cloud Function will ACK acknowledge the pubsub message as soon as it receive it and even if it crashes. So HTTP cloud function a better bet. Otherwise you catch any error in your code and publish the message programatically in the deadletter topic :) — MBHA Phoenix
if I use HTTP Cloud function then if the function crashes will the message be put into the dead letter topic? — tmp dev

guillaume blaquiere guillaume blaquiere · Accepted Answer · 2021-08-13T08:09:38

You can't plug a function directly on a PubSub topic and define a deadletter topic. (same issue if you want to define filters on your subscription).

The solution is to create an HTTP functions, and to create separately a push subscription with a dead letter topic. 2 points of attention:

Take care to the security part, perform secure calls
The PubSub message format is slightly different from PubSub triggered cloud function and HTTP cloud function with PubSub push message.

In addition, keep in mind that the dead letter topic is used when at least 5 failures occur on the same message. The dead letter topic is not used at the first error returned by your Cloud Functions.

gcp cloud function pub/sub topic dead letter

1 Answers