How to configure Firewall Rules and DNS settings on GCP with VPC Service Controls when using Cloud Composer and Cloud Functions in the same project?

0
votes

We have troubles with the configuration of two GCP components (the GCP Cloud Composer, the GCP Functions) under VPC Service Controls for a single project. We can configure them separately with VPC Service Controls, but as soon as we add the second component we experience troubles.

Once we have configured the Firewall rules and DNS settings for the Cloud Function, the Composer starts to fail. The Composer monitoring tool (for webserver, SQL server, etc.) shows that the composer is unhealthy.

What are we doing wrong?

The docs we've used:
https://cloud.google.com/functions/docs/securing/using-vpc-service-controls
https://cloud.google.com/composer/docs/configuring-vpc-sc

For the Composer we use these settings:

  • 3 node
  • standard machine type
  • image version: composer-1.16.3-airflow-1.10.15
  • python: v3
  • private IP Composer

The APIs what we have enabled for VPC Service Controls:

  • Google Cloud Dataproc API
  • Cloud Functions API
  • Google Cloud Pub/Sub API
  • Cloud SQL API
  • Google Cloud Storage API
  • Google Compute Engine API
  • Google Kubernetes Engine API
  • Google Container Registry API
  • Cloud Monitoring API
  • Cloud Composer API
  • Artifact Registry API

The FireWall settings (based on the documentation):

Name Type Targets Filters Protocols / Ports Action Priority
composer-egress-gke-cluster-all-port Egress Apply to all IP ranges: 10.10.0.0/16 tcp, udp Allow 1000
composer-egress-healthcheck Egress Apply to all IP ranges: 130.211.0.0/22, 35.191.0.0/16 tcp:80,443 Allow 1000
composer-egress-port53 Egress Apply to all IP ranges: 0.0.0.0/0 tcp:53, udp:53 Allow 1000
composer-egress-webserver Egress Apply to all IP ranges: 172.31.251.0/24 tcp:3306,3307 Allow 1000
restricted-google-apis Egress Apply to all IP ranges: 199.36.153.4/30 tcp:443 Allow 1000
composer-deny-all Egress Apply to all IP ranges: 0.0.0.0/0 all Deny 2000
composer-ingress-helthcheck Ingress Apply to all IP ranges: 130.211.0.0/22, 35.191.0.0/16 tcp:80,443 Allow 1000
ingress-iap Ingress Apply to all IP ranges: 35.235.240.0/20 tcp Allow 1000
gke-europe-west1-composer-dns-xxx-xxx-all Ingress gke-europe-west1-composer-dns-xxx-xxx-node IP ranges: 10.124.0.0/14 tcp;udp;esp;ah;sctp;icmp Allow 1000
gke-europe-west1-composer-dns-xxx-xxx-master Ingress gke-europe-west1-composer-dns-xxx-xxx-node IP ranges: 172.16.8.0/28 tcp:10250,443 Allow 1000
gke-europe-west1-composer-dns-xxx-xxx-vms Ingress gke-europe-west1-composer-dns-xxx-xxx-node IP ranges: 10.10.0.0/16 tcp:1-65535;udp:1-65535;icmp Allow 1000

The DNS settings (based on the documentation):

DNS name: googleapis.com. Type: Private

DNS name Type TTl(Seconds) Data
*.googleapis.com. CNAME 300 restricted.googleapis.com.
googleapis.com. SOA 21600 ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
googleapis.com. NS 21600 ns-gcp-private.googledomains.com.

DNS name: cloudfunctions.net. Type: Private

DNS name Type TTl(Seconds) Data
*.cloudfunctions.net. A 300 199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7
cloudfunctions.net NS 21600 ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
cloudfunctions.net. SOA 21600 ns-gcp-private.googledomains.com.

DNS name: pkg.dev. Type: Private

DNS name Type TTl(Seconds) Data
*.pkg.dev. CNAME 300 pkg.dev.
pkg.dev. SOA 21600 ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
pkg.dev. NS 21600 ns-gcp-private.googledomains.com.
pkg.dev. A 300 199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7

DNS name: gcr.io. Type: Private

DNS name Type TTl(Seconds) Data
*.gcr.io. CNAME 300 gcr.io.
gcr.io. SOA 21600 ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
gcr.io. NS 21600 ns-gcp-private.googledomains.com.
gcr.io. A 300 199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7

Thank you for any idea or help what you can give.

1
google-cloud-platformgoogle-cloud-functionsgoogle-cloud-composergoogle-cloud-dnsgoogle-vpc
Can you provide some logs or actual error messages that you're getting ? From your description it's hard to say what's causing the issue here. – Wojtek_B
The problem with this that it's hard to get relevant log... The Composer just starts to fail according to it's monitoring UI after setting up the firewall rules and DNS settings. This UI seems to sum up the results in a non-consistent way. After changing the time resolution, sometimes it shows the failure, sometimes everything is green. After enabling the logging on the Composer's subnet the StackDriver is dumped by too many records to debug it properly. Meanwhile we've figured out the problem, I'll describe it below. – elaspog

1 Answers

1
votes

Meanwhile we've figured out the problem:

  • The most problematic is that an A type DNS record was missing from googleapis.com. which contains the IP addresses.
  • The second problem was that a firewall rule coming from the documentation of VPC SC settings of the Cloud Function has blocked the communication between:
    • the nodes of the Composer on the subnet
    • between the Worker nodes and the Master node (of the Kubernetes cluster created for composer) which is created automatically in a different project and connected by network peering to the subnet of the Composer workers

The two documentations obviously does not cover this special case when two of these services should be used together. After fixing the issues above it has started to work.

The problems what we still experience:

  • it's hard to debug this kind of problem, because the monitoring UI of the composer seems to show the information in non-consistent way
  • it's hard to use the StackDriver logs for debugging after enabling it on the subnet, because it's simply too much information and they are logged on "info" level, not on "warning" or "error" level