Connecting two services on Google Cloud Run (gRPC/Python)

0
votes

I've just wasted most of today trying to connect two microservices deployed on Google Cloud Run, using gRPC with Python. Turns out if I open the Trigger ingress for the service I'm trying to call to 'Allow all Traffic' and 'Requires Authentication' I can connect to it from the other service.

However the moment I set ingress to 'Allow internal traffic only' (with 'Requires Authentication' still set) my service can no longer connect to it and throws a Permissions Denied error in the logs:

status = StatusCode.PERMISSION_DENIED details = "Received http2 header with status: 403"

I can't see any logs on the service being called to indicate it's blocking or receiving anything, and as the only thing that's changed is the ingress setting, I'm assuming the authentication with IAM users that I have working when the service is public isn't the problem.

In the ingress docs (https://cloud.google.com/run/docs/securing/ingress) I saw a note that it only allows VPC traffic, so I went about creating a VPC network and a Serverless VPC Connector and attaching them to both services (and updating the IAM service account to allow access to the connector) - and still get the same issue.

I'm giving up now. I don't like the idea of having my internal service public, but turning on internal ingress doesn't seem to work.

Has anyone managed to connect two services on Google Cloud Run, with one internal traffic only, and if so what do I need to do to get it working??

Thanks!

1
google-cloud-run
On which service did you add the serverless VPC connector? – guillaume blaquiere
I added the VPC Connector to both. So my architecture is: * Webapp Service - All traffic, no authentication, with VPC Connector 'internal' * Backend Service - Internal Only, with authentication, with the same VPC Connector 'internal' attached to same VPC. Hope that helps! – dgildeh

1 Answers

1
votes

In your frontend Cloud Run service, you need to put a VPC connector and to route ALL the traffic to it (egress = all). On the backend service, you don't need the serverless VPC connector.

To go a step deeper, Cloud Run is always accessible from the internet even if you set the ingress to internal. In fact, there is an internal rule that check the request metadata (internal to Google internal network) to validate if the traffic is coming from YOUR project VPC or not. And because it's publicly accessible, you need to route ALL the egress traffic to the VPC, else the public access don't go through your VPC and you get a 403.