Cloud function triggering on same file 2x?

Question

I have a cloud function that is based off of google's function that triggers a dag when a new file arrives in a gcs bucket. For some reason, about 1 in every 50 or 100 files is triggered twice, even though there is only one of each file. Does anyone know why this is happening?

here's google's link to the cloud function https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/e93672b2828609b5008ff6a33f2393daed5cacbf/functions/composer-storage-trigger/index.js

I have the function set to retry every 60 seconds because if i dump a lot of files in at once a bunch will need to keep retrying

Answer 1

For background functions, such as Cloud Storage triggers, Cloud Functions guarantees "at-least-once" delivery of the event payload. The implication here is that, for reliability, the event may actually get delivered more than once in some rare circumstances. Your functions needs to expect this and be ready for it. It's your responsibility to make sure your functions are idempotent if it's not OK to process the event twice.