Gmail API getMessage method returns 404 for message gotten from listHistory method

Question

A short description of the issue:

Our sync is set up as following:

Perform full sync on user's Inbox by calling messages.list method. Save highest historyId from the response Upon Push Notification to our webhook endpoint, call history.list with startHistoryId we previously saved Using message ID's from response's messagesAdded section (we exclude messages in messagesDeleted section and messages with label DRAFT), call messages.get method on Gmail API. On random basis, Gmail API returns 404 error ("Requested entity was not found"). This often happens in bursts (but not always), where 30-50 supposedly added messages return 404. I've verified the users don't have automatic filters set up that could cause it. It also has happened outside of working hours, which implies the user could not have deleted the messages. Example:

listHistory was called with historyId 463077 Following message was returned in the response:

{
  id: '1762a36a2ab4ab29',
  threadId: '175e5e4219f86042',
  labelIds: [ 'UNREAD', 'IMPORTANT', 'CATEGORY_PERSONAL', 'INBOX' ]
}

getMessage returned 404 (Requested entity was not found) for that particular message. Timestamp 2020-12-03T20:06:42.392Z A small code sample that reliably reproduces the issue. The sample should run as-is or with minimal setup, without external dependencies.

We're using googleapis 47.0.0 Node package.

  let addedMessageIds: string[] = [];
  let deletedMessageIds: string[] = [];
  let nextPageToken: undefined | string;
  const numbers: number[] = [];
  while (true) {
    const result = await gmail.users.history.list({
      startHistoryId: '664689',
      pageToken: nextPageToken,
      userId: 'me',
      labelId: 'INBOX',
      historyTypes: ['MESSAGE_ADDED', 'MESSAGE_DELETED'],
    });

    for (const historyMessage of result.data.history || []) {
      for (const deleted of historyMessage.messagesDeleted || []) {
        if (!deleted.message?.id) continue;
        deletedMessageIds.push(deleted.message.id);
      }

      for (const added of historyMessage.messagesAdded || []) {
        if (!added.message?.id) continue;
        addedMessageIds.push(added.message.id);
      }
    }

    if (!result.data.nextPageToken) break;
    nextPageToken = result.data.nextPageToken;
  }
  
  for (const id of addedMessageIds) {
    if (deletedMessageIds.includes(id)) continue;
    try {
      await gmail.users.messages.get({
        id: id,
        userId: 'me',
      });
    } catch (e) {
      console.error('DID NOT GET MESSAGE', {
        messageId: id,
        error: e.message,
      });
    }    
  }

Metrics at Google Cloud Platform report the error rate for getMessage method at ~76%. By comparing message ID's in our system, we still manage to get most of the messages, but not all. Since the error rate is always above 0, we have no certainty whether we got the messages or not.

Why we would receive a 404 error when using getMessage to retrieve a message ID returned by the listHistory call?

Answer 1

This appears to be a bug!

I was unable to reproduce this issue myself, but there is already a report on Google's Issue Tracker which detail the same kind of behaviour (though this is possibly your report):

• Gmail API getMessage method returns 404 for message gotten from listHistory method

Google does seem to know about this issue from their responses, so I would suggest following that issue thread and providing additional information there to help along the process.

You can also hit the ☆ next to the issue number in the top left on the aforementioned pages which lets Google know more people are encountering this and so it is more likely to be seen to faster.

I hope this is helpful to you!