Google PubSub / Gmail Webhook: Consistently Receiving Multiple POST Requests From PubSub When Sending An Email

Question

I have a webhook set up that is listening to and handling changes in my INBOX and SENT folders in Gmail using Google Cloud PubSub API and the Gmail API in Java.

The problem I'm seeing is that when I send a message to another user, PubSub seems to PUSH to my endpoint twice within a single second with a slightly different history_id and message_id but the same subscription name and user email.

I understand that PubSub guarantees at-least-once delivery so it's not unsual to be receiving a duplicate message, but because it's consistently happening and the message_id is different, I'm thinking there might be multiple push requests based on the PubSub documentation below:

Cloud Pub/Sub assigns a unique message_id to each message, which can be used to detect duplicate messages received by the subscriber. This will not, however, allow you to detect duplicates resulting from multiple publish requests on the same data.

What I've tried:

Ensured that there is only a single Topic/Subscription on my Google Cloud console.
Set the Ack deadline different values between 10 and 600 seconds.
Called service.users().stop() to ensure that I haven't been calling watch() multiple times then begin watch() again.

I've looked into PubSubIO to ensure exactly-once-delivery, but I figure if I'm consistently getting multiple PubSub messages, there must be something fundamentally wrong with how I've set up my webhook.

Edit: Here is the code I have to watch for changes in my Gmail account. I am using a service account with domain-wide authority in order access accounts in the entire domain

public static Map<String, String> watchInbox(Gmail service) throws IOException {
    Map<String, String> watchInboxResponse = new HashMap<>();
    List<String> labelsToWatch = Arrays.asList("INBOX", "SENT");
    String topicName = "projects/subscription-name/topics/topic-name";

    WatchRequest request = new WatchRequest();
    request.setLabelIds(labelsToWatch);
    request.setTopicName(topicName);

    WatchResponse response = service.users().watch("me", request).execute();

    watchInboxResponse.put("historyId", response.getHistoryId().toString());
    watchInboxResponse.put("expiration", response.getExpiration().toString());
    return watchInboxResponse;
}

I take insert the historyid and expiration into a database and use that to check, upon receiving a webhook call, do I need to call watch() again if more than 24 hours has gone since last calling watch (as recommended by Google).

Can you add any details like: code used, error problem encountered? How do I ask a good question?, How to create a Minimal, Complete, and Verifiable example Show the community what you have tried. — MαπμQμαπkγVπ.0
Please enhance your code. The example that you have included is extremely minimal and does not help with a solution. — John Hanley
Did you find a solution to this? I'm having a similar issue: stackoverflow.com/questions/53619495/… — mclovine

jainam shah jainam shah · Accepted Answer · 2018-12-13T22:20:14

I had similar behaviour when I was implementing Google pub/sub watch request.

What Gmail does while you are composing mail is that it create the system labels "Send and Draft" and keeps constantly saving to draft with new messageId and label "Send and draft" you are subscribed to any changes to "Inbox and Send" so you will be hit on your webhook for twice or maybe more time!!

The message from Gmail always contains labels you would have to filter those message that has label draft.

I use Gmail .net SDK for my code and had to handle

//Explicitly avoid further processing
 bool isdraft = y.Message.LabelIds.Contains("DRAFT");

and filter those regardless.

Google PubSub / Gmail Webhook: Consistently Receiving Multiple POST Requests From PubSub When Sending An Email

1 Answers