StackDriver User Quotas

0
votes

I'm using the Java client library's MetricServiceClient for getting StackDriver timeseries. I am authenticating using a user oauth token (this user has access to multiple projects), but there seems to be some kind of global quota across multiple projects because when I fetch only one or two projects at a time I have no throttling, but when I fetch four or five different projects at a time, I start getting throttled with errors like the following:

io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Insufficient tokens for quota 'DefaultGroup' and limit 'USER-100s' of service 'monitoring.googleapis.com' for consumer 'project_number:764086051850'.

I have confirmed this by alternating which projects are being fetched so that I can say it is not any single project -- they all start to get rate limited. Another strange thing is, that project_number in the error message doesn't correspond to any project I am fetching, or even have access to -- it is meaningless to me.

This appears to be the quota for # of requests per 100 seconds, but I have that set to 10,000 on all projects and I'm not doing nearly that many requests, as the quota historical chart in the web console confirms.

Is there really some global quota that applies across multiple projects and if so, is there some way to work around it? It is much simpler to me to have a single user with access to multiple projects instead of having to make service account tokens for them all.

1
stackdrivergoogle-cloud-monitoring
Are you still seeing this quota error? Can you attach the quota historical chart in the web console while you got this error? - JL-HaiNan
I stopped using a single user account and used separate service account keys for each project. I don't have any charts left from back then but I can assure you that the project charts for API quotas was much lower per project than what the stated quota was (I was nowhere near the 10,000 req/100 seconds that it stated). When I did the same request behavior using service accounts I didn't get any throttling, despite making the same number of requests per project. It appeared that the throttling was taking place cross-project since I was using a single user account. Any ideas? - keitwb

1 Answers

1
votes

The token quota you're hitting is for users using Application Default Credentials (which uses a shared gcloud project for billing), while it exists to get users up and run quickly but it's not recommended for actual production use. Therefore using a proper service account bound to the user's project is highly recommended and the solution to the issue.