Google Cloud Function 403 for internal authenticated requests

Question

I am calling a cloud function from within my GCP project.

I receive 403 (Permission Denied) when the function is configured with Allow internal traffic only, see https://cloud.google.com/functions/docs/networking/network-settings#ingress_settings

When removing the ingress control there is no issue, the function responds with status 200. The function does not allow un-authenticated access, IAM policies are configured.

Following the example from https://cloud.google.com/functions/docs/securing/authenticating#function-to-function:

# main.py
import requests

# TODO<developer>: set these values
# REGION = None
# PROJECT_ID = None

RECEIVING_FUNCTION = 'hello-get'

# Constants for setting up metadata server request
# See https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature
function_url = f'https://{REGION}-{PROJECT_ID}.cloudfunctions.net/{RECEIVING_FUNCTION}'

metadata_server_url = \
    'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience='
token_full_url = metadata_server_url + function_url
token_headers = {'Metadata-Flavor': 'Google'}


def hello_trigger(request):
    token_response = requests.get(token_full_url, headers=token_headers)
    jwt = token_response.text

    function_headers = {'Authorization': f'bearer {jwt}'}
    function_response = requests.get(function_url, headers=function_headers)

    function_response.raise_for_status()

    return function_response.text


def hello_get(req):
    return 'Hello there...'

Deploying the function and the triggering function with desired ingress settings:

gcloud functions deploy hello-get --trigger-http --entry-point hello_get --runtime python37 --ingress-settings internal-only
gcloud functions deploy hello-trigger --trigger-http --entry-point hello_trigger --runtime python37 --ingress-settings all --allow-unauthenticated

Calling hello-trigger returns 403.

Changing ingress of hello-get solves the issue:

gcloud functions deploy hello-get --trigger-http --entry-point hello_get --runtime python37 --ingress-settings all

Now calling hello-trigger returns 200.

The service account used for Cloud Functions is given the Functions Invoker Role for this setup.

Google Cloud Functions is not a resource in your VPC. This means that your triggering function cannot call your hello-get function when internal-only is configured. Notice the wording "Only requests from VPC networks in the same project". A Cloud Function is located in Google's network. — John Hanley
@JohnHanley thanks. Can you give an example of a setup in which I would be able to call hello-get? If you don't mind spelling it out, I clearly struggle to understand the details of the networking here. — Boris Kjaer
You can call the function from a Compute Engine instance. For function to function calling, use authorization and not networking. cloud.google.com/functions/docs/securing/authenticating — John Hanley

guillaume blaquiere guillaume blaquiere · Accepted Answer · 2020-12-02T09:25:57

When you set the ingress traffic to internal-only, only the traffic coming from your VPC or from the VPC SC (Service Control) is accepted.

Here, in your trigger function, you don't come from YOUR vpc, but from another one (a serverless VPC, managed by Google, the land where the Cloud Functions are deployed). Therefore, the ingress setting isn't respected and you get a 403.

So, for this you have 2 solutions:

Use only IAM service to filter who can invoke or not your function, and let "public" your function with an ingress=all. (Solution proposed by John in his 2nd comment). It's already a high level of security.

However, sometime, for regulatory reason (or for old fashion security team design) network control is preferred.

If you want to pass through your VPC, you need to

Create a serverless VPC connector in the same region as your trigger function
Deploy your trigger function with this serverless VPC connector
Set the egress traffic to all --egress-settings=all

Like this, all the outgoing traffic of your trigger function will pass through the serverless VPC connector, thus, the traffic is routed in your VPC before trying to reach your "ingress-internal" cloud functions. And it will be accepted.

If your function use ingress=all settings, anyone can reach it from internet.

However, if you don't make the function publicly accessible, I mean, authorized to unauthenticated user, only the valid requests (authenticated AND authorized with the role cloudfunctions.invoker) will be processed by your Cloud Functions

In fact, there is a common layer to any Google service name GFE: Google Front End. This layer is in charge of many things (expose your service in HTTPS, manage your certificates, discard DDoS attack OSI layer 4,...) whom the check of the authentication header and the authorization check against the IAM service.

Therefore, in case of DDoS attack on the layer 4, GFE filter by default these attacks. In case of attack of layer 7, only the authorized request (valid) are allowed and you will pay only for them. The filter performed by GFE is free.

Google Cloud Function 403 for internal authenticated requests

1 Answers