"Error: Could not load the default credentials." in App Engine production environment

Question

I have been using Datastore in AppEngine since few weeks ago and there was no such access issue in production. Today 1pm SGT, my service was suddenly returning 500 error with this error message although I never deploy to production.

Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.

This error probably happened when accessing Datastore with my GCP's default credential:

const {Datastore} = require('@google-cloud/datastore');
const datastore = new Datastore();
const [shop] = await datastore.get(
    datastore.key(['Shop', Number(phone)])
)

My stacks:

Standard AppEngine with nodejs10
Koa, Next, Datastore

In App Engine dashboard, as a random solution, I changed to the old version. Then, it suddenly started working. Then, I changed back to the original version. It worked well too. Any clue?

Suspect 1: Around the time, I was working locally. Although I never deploy, I was executing few GCP commands. These should not affect production tho

export GOOGLE_APPLICATION_CREDENTIALS="[my local credencial json file]"

gcloud config set project [project-name]

This looks like an runtime issue a kind of transient error, if you need a root cause analysis maybe is [better file a support case][1] in order to know more about this and avoid issues on the future [1]: console.cloud.google.com/support — Jan Hernandez
Thanks for the comment. I contacted chat support before but they redirected me to Stack Overflow or higher plan to get support — youminkim
I have no idea what is happening, but I think usually after my GAE app starts up after being dormant for a couple of days it will fail and then restart and then work. Strange and looking for a way to reduce false alarms. Did you find any more information? — ahong

ahong ahong · Accepted Answer · 2020-12-22T19:49:59

Seems like this problem has to do with App Engine/Cloud Function cold starts. In this scenario, the environment would not have loaded the credentials yet when the client library is being initialized.

Apparently it is fixed in newer versions of the client libraries: https://github.com/googleapis/gapic-generator-typescript/issues/287

But if not or if you do not want to upgrade your @google-cloud/ client libraries, this is the suggested workaround:

const {Datastore} = require('@google-cloud/datastore');
let datastore;

// If you are using it in a request handler
exports.handler = async (req, res) {
  if (!datastore) {
    datastore = new Datastore();
  }
  const [shop] = await datastore.get(
    datastore.key(['Shop', Number(phone)])
  );
}

See also: https://github.com/googleapis/google-auth-library-nodejs/issues/798#issuecomment-591622283

"Error: Could not load the default credentials." in App Engine production environment

1 Answers