User Rate Limit Exceeded for Google Cloud Storage OAuth2 API

1
votes

I use the Google API PHP Client (http://code.google.com/p/google-api-php-client/) to make OAuth requests - to get a new Access Token.

I have cached the Refresh Token and use this to generate a new Access Token. I have gone over the documentation (https://developers.google.com/accounts/docs/OAuth2, https://developers.google.com/storage/docs/developer-guide) and it only talks about limits on the Refresh Token (one limit per client/user combination, and another per user across all clients) but nothing about Access Token limits (except for the fact that an Access Token is only valid for an hour).

I'm trying to calculate bucket size usage across thousands of buckets. I'm trying to parallelize this task to cut down on time - I do this by spawning a new process for each bucket and each process requests a new Access Token. I do this because of my assumption that there is no limit on the number of Access Tokens issued and because, for a bucket with lots and lots of objects, the calculation time + potential exponential backoff time could theoretically exceed the lifetime of the Access Token.

But when I try to do this, I see this error:

Error No: 1
Error on Line: 242
Error Message: Uncaught exception 'apiAuthException' with message 'Error refreshing the OAuth2 token, message: 
<HTML>
<HEAD>
<TITLE>User Rate Limit Exceeded</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>User Rate Limit Exceeded</H1>
<H2>Error 403</H2>
</BODY>
</HTML>

Is this because I'm spawning a lot (16 at the moment) of Access Tokens?

If not, what then, is causing this error? What's the best way to get around this error?

Is there a Google Documentation page that documents the User Rate Limits?

2
oauth-2.0google-cloud-storage

2 Answers

7
votes

There are no limits on access tokens based on a refresh token. There's however, a limit on the rate at which access tokens can be requested. You can request thousands of access tokens based on a single refresh token where they are all simultaneously valid, but if you exceed a few qps of simultaneous access tokens requests you will get rate limit exceeded.

As mentioned above, a single access_token can be re-used in parallel in multiple requests, if it's valid for all of these requests.

The limits on access_tokens are not published as they are subject to change. The correct client implementation is to engage in exponential backoff to ensure correctness in the presence of rate limit changes. However, in your case, since all tokens share the same scope and usage context, you should be able to re-use the same token successfully.

1
votes

There is a limit on the number of access tokens. When you request a new access token using a refresh token, it may overwrite a previously assigned access token, even if the time limit is not exceeded (yes, there's a maximum number of access tokens that you can have active at a single time).

Also, there's a rate limit on how often you can use the refresh token to request a new access token.

The best way around this is to have a central key management process that is responsible for getting a new access token every x minutes (if the timeout is an hour, make it 30 minutes, for example). The parallelized processes then request the single access token from the central key management process.

As per https://developers.google.com/accounts/docs/OAuth2 :

Access tokens have a limited lifetime and, in some cases, an application needs access to a Google API beyond the lifetime of a single access token. When this is the case, your application can obtain what is called a refresh token. A refresh token allows your application to obtain new access tokens.

Note that there are limits on the number of refresh tokens that will be issued; one limit per client/user combination, and another per user across all clients. You should save refresh tokens in long-term storage and continue to use them as long as they remain valid. If your application requests too many refresh tokens, it may run into these limits, in which case older refresh tokens will stop working.