Cloud Run static outbound IP address does not go through Google App Engine firewall

1
votes

I have a python (flask) application running on Google App Engine (flex); the application is protected by the GAE firewall where:

  • Default rule is 'Deny' all ingress
  • There is a whitelist of IP addresses from which traffic is allowed.

I have some microservices deployed on Cloud Run (fully managed) which:

  • Receive requests from the GAE app (e.g. for heavy duty tasks)
  • Send the results of whatever they process as http requests back to handlers/endpoints in the GAE app

Thus the GAE app is the main point of interaction with clients and a dispatcher of heavy tasks, while the processing of those tasks is carried out by the microservices. I have set up a static outbound IP address of the Cloud Run hosted service which verfiedly works and traffic is routed through the NAT gateway as required in the documentation. The respective NAT IP address is on the firewall whitelist.

The problem is that the firewall still does not let in the Cloud Run >>> GAE app requests which bounce back with 403 statuses (of course, if I change the default firewall rule to 'Allow', traffic goes through). If I host the same microservice in a docker container on a GCE VM with a static IP address like this everything works flawlessly. This makes me hypothesize that albeit Cloud Run outbound traffic is indeed routed through the static IP address when traffic is towards addressees outside GCP, when I try to ping an internal (project-wise) asset it still goes though some dynamically selected IP (i.e. the static IP solution simply does not work). Unfortunately the logs don't show the 403-ed attempt so I can't see from what IP addresses those request seem to come (from a GAE standpoint).

I would be very grateful for ideas how this can be fixed as it greatly diminishes the value of the otherwise wonderful idea to have static outbound IP addresses for Cloud Run.

3
google-app-enginegoogle-cloud-platformfirewallgoogle-cloud-runstatic-ip-address
What is your egress traffic param value? run the command gcloud run services describe <service name> and share the Egress: value – guillaume blaquiere
All the traffic is routed through the Cloud NAT - `Egress: All, VPC connector: the_name_of_my_connector' – Kiril

3 Answers

2
votes

First, thank you both for your help and suggestions, they are very helpful. I found the solution with some kind help from Google:

  • When the Cloud Run microservice and the GAE app are hosted in the same project traffic is still routed through internal channels and appears to come from IP address 0.0.0.0 which can be whitelisted (so it would work) as long as one considers this address encompasses GCP assets which are parts of other projects too (to the best of my understanding)
  • A more robust solution seems to be setting up an externally facing load balancer as described here and putting it in front of the GAE app; in such a case, Cloud Run will indeed consistently use its static outbound IP address as described in the documentation
1
votes

You are correct saying that the static IP is not honoured when packets are routed internally to GCP.

I think this is what you want. You have to allow in the firewall one of the IPs mentioned there (not sure which one right now).

1
votes

Just as you and @Ema mentioned, this is an expected behavior having in mind that the traffic from Cloud Run to App Engine is intern.

When you use Cloud Nat to send all traffic there, it does happen. If you create a container and ping, let's say to www.github.com. You will find that the traffic goes through the IP you set. On the other hand, if you ping to www.google.com, given that the traffic is intern, and the site to reach out is in the same infrastructure, the request doesn't even goes through public internet.

Additionally, just to keep in mind Static outbound IP address is still in Beta and it is not recommended to use Beta features/products in production environments.

As you mentioned and as it is stated in Allowing requests from your services:

Creating a rule for IP 0.0.0.0 will apply to all Compute Engine instances with Private Google Access enabled, not only the ones you own. Similarly, allowing requests from 0.1.0.40 or 10.0.0.1 will allow any App Engine app to make URL Fetch requests to your app.

This questions might be of your interest: