Scaling Authorization for Google Cloud Storage Buckets

Question

I want to grant access to Google Cloud Storage buckets to thousands, maybe millions of users. My original plan was to grant access via IAM roles using their REST api. However, on this Quotas & limits page, it says

There is a limit of 100 members holding legacy IAM roles per bucket and a limit of 1500 members holding all IAM roles per bucket. Examples of members include individual users, groups, and domains. See IAM identities.

Does that mean, I can only offer read access to 1500 people? If so, what would be a way to grant access for thousands or millions of people?

Answer 1

You cannot directly add millions of people to the bucket IAM permissions. Instead at this scale consider creating a google group and and adding the mass of people to that group. According to this doc Workspace groups are unlimited in size.

However you may want to consider other options than directly adding users to the IAM roles, e.g.

1. Write your own backend that understands your user identity, and then returns signed urls that the clients can use to do the upload.
2. Use firebase storage, which is tightly integrated with GCS but provides end user authentication/authorization.

These approaches have some advantages:

• You don't have to require your users to have Google Accounts
• You can have more control over exactly what each user can do (e.g. prevent them reading/writing each others data, limit file sizes, add rate limits or even just validate the data they send)
• You don't need to expose your cloud infrastructure to them.
• Option (1) could be written in a way that is portable so that you don't get locked in to Google.