GCP VM Instance is not able to access secrets from Secret Manager despite of appropriate Roles

Question

I created few secrets in GCP's Secret Manager service. Then to access those secrets on my local machine I created a service account and a JSON key to authenticate that service account from my local machine. I also granted the role Secret Manager Secret Accessor to that service account to enable it to access the values of secrets from Secret Manager. Now, it is working perfectly fine on my local machine.

Further, I want to deploy this code to the GCP Compute Instance. Hence, I created one and shipped the source code to that instance. I also granted the same permission Secret Manager Secret Accessor to the default service account of compute instance. Now, when I run try this code on the instance it returns me an error of permission denied as stated below.

The above exception was the direct cause of the following exception:
ibdax            | 
ibdax            | Traceback (most recent call last):
ibdax            |   File "manage.py", line 22, in <module>
ibdax            |     main()
ibdax            |   File "manage.py", line 18, in main
ibdax            |     execute_from_command_line(sys.argv)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
ibdax            |     utility.execute()
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 363, in execute
ibdax            |     settings.INSTALLED_APPS
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 82, in __getattr__
ibdax            |     self._setup(name)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 69, in _setup
ibdax            |     self._wrapped = Settings(settings_module)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 170, in __init__
ibdax            |     mod = importlib.import_module(self.SETTINGS_MODULE)
ibdax            |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
ibdax            |     return _bootstrap._gcd_import(name[level:], package, level)
ibdax            |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
ibdax            |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
ibdax            |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
ibdax            |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
ibdax            |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
ibdax            |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ibdax            |   File "/ibdax/ibdax/settings.py", line 19, in <module>
ibdax            |     from ibdax.constants import (
ibdax            |   File "/ibdax/ibdax/constants.py", line 30, in <module>
ibdax            |     DEV_DATABASE_HOST=secrets.get_secrets("dev-database-host")
ibdax            |   File "/ibdax/ibdax/gcp_secret_manager.py", line 23, in get_secrets
ibdax            |     response = self.client.access_secret_version(request)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1155, in access_secret_version
ibdax            |     response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
ibdax            |     return wrapped_func(*args, **kwargs)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
ibdax            |     on_error=on_error,
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
ibdax            |     return target()
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
ibdax            |     six.raise_from(exceptions.from_grpc_error(exc), exc)
ibdax            |   File "<string>", line 3, in raise_from
ibdax            | google.api_core.exceptions.PermissionDenied: 403 Request had insufficient authentication scopes.

I checked the IAM role for Compute Instance's service account and it has some message which I am not able to understand. Here is the screenshot of that -

How do I fix this?

sethvargo sethvargo · Accepted Answer · 2021-04-18T17:57:34

First, please don't download a JSON service account key file to your local machine. You can install the gcloud command line tool at https://cloud.google.com/sdk and then authenticate via your user account:

$ gcloud auth login && gcloud auth application-default login

This will prompt you to login to your Google account via the web - no service account required. This is much better from a security and auditing perspective.

Similarly, when running on GCE (or any "compute-based" platform like GKE, Cloud Functions, Cloud Run, etc), you should create a service account and run the instance as that service account. Do not use the Default Compute Engine service account!. Furthermore, do not give the Default Compute Engine service account permission to access all secrets, as this poses a significant security risk. Please see more information at Secret Manager Best Practices.

The default OAuth scopes for GCE do not include cloud-platform. You need to update the scopes to include cloud-platform:

$ gcloud compute instances set-service-account "my-instance" --service-account "...@..." --scopes "cloud-platform"

GCP VM Instance is not able to access secrets from Secret Manager despite of appropriate Roles

1 Answers