Google Cloud PubSub: Previously valid subscription suddenly unauthorized

Question

I'm having some trouble with the Google Cloud PubSub API. Recently, I started using Cloud PubSub to queue messages for a chat server I'm working on. Messages coming in from a PubSub subscription are forwarded to users using the socket.io node library. I have no problems getting this set up - I run my node.js server, open up a couple browser windows, and I can chat away without any problems.

I've noticed, however, that often after the server has been running for a few hours, that it starts spitting out the following error:

(node:2525) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project. (node:2525) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): Error: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

...

(node:2525) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1253): Error: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential...

This error repeats and repeats (you can see the incrementing rejection id in the error message) until stopping a few minutes later and things resume working. While I'm getting the error messages, I can't send any messages through Cloud PubSub.

Here's the code that sets up the listener:

function listenForMessages(handler)
{
    var pubsub = require('@google-cloud/pubsub')({
        projectId: config.pubsub_project_id,
        keyFilename: config.pubsub_keyfile
    });

    pubsub.subscribe(config.pubsub_topic, 'test-subscription', {autoAck: true}, function(err, subscription){
        subscription.on('message', function(message) {
            handler(message.data);
        });
    });
}

The credentials come from an external config file - I'm pretty sure they're ok, especially given that there's no trouble setting up the listener when I initially run the server.

TL;DR: I start getting an "invalid credentials" error repeatedly a few hours after I start running a node server that uses Google Cloud PubSub to queue messages.

I also have this issue lately. The weird part is that I have about 10 different microservices pulling from 10 different queues, but this only ever happens on one specific queue. And yeah, it usually takes a few days to a week for it to show up, but seems to happen regularly. — static416
Also, I would change the title if you can to more accurately summarize the issue. The promise rejection is only incidental. The main problem is that it's losing authorization randomly after some period of time. EDIT: I attempted to edit the title myself. — static416

vizsatiz vizsatiz · Accepted Answer · 2018-10-19T08:02:12

I was facing the same issue and finally found the solution. The issue was google token getting expired and thus the pubsub subscriber throwing the exception. There was already a bug in github for a similar oauth issue. (Link)

The issue was fixed in node-pubsub version 0.20.0 and upgrading to this version will fix your issue. The actual fix was in google-auth-library-nodejs version 2.0 and node-pubsub:v0.20.0 uses google-auth-library-nodejs:2.0.

From github release notes of google-auth-library-nodejs:2.0:

The OAuth2.refreshAccessToken method has been deprecated. The getAccessToken, getRequestMetadata, and request methods will all refresh the token if needed automatically. There is no need to ever manually refresh the token.

Hope this helps !!

Google Cloud PubSub: Previously valid subscription suddenly unauthorized

1 Answers