Google Kubernetes private cluster: Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork

Question

We are attempting to make several private Kubernetes clusters. We can find limited documentation on specific settings for the private cluster, therefore we are running into issues related to the subnetwork IP ranges.

Say we have 3 clusters: We set the Master Address Range to 172.16.0.0/28, 172.16.0.16/28 and 172.16.0.32/28 respectively.

We leave Network and Subnet set to "default". We are able to create 2 clusters that way, however, upon spin-up of the 3rd cluster, we receive the error of "Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork: 5." We suspect that we are setting up the subnetwork IP ranges incorrectly, but we are not sure what we are doing wrong, or why there is more than 1 secondary range per subnetwork, to begin with.

Here is a screenshot of the configuration for one of the clusters:

We are setting these clusters up through the UI.

Answer 1

This cluster has VPC-native (alias IP) enabled, which use 2 secondary ranges per cluster.

See https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips#secondary_ranges

According to

Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork: 5.

the max is 5. That's why the 3rd one failed to create.

Answer 2

The best approach is to create a new subnetwork for each cluster. This way, each subnetwork only requires 2 secondary ranges, and you won't hit the limit of 5.

Answer 3

For anyone who lands here from Google and is wondering how to list / see the subnet names that have been created using GKE as described in OP's question:

To list subnets for a region (and potentially modify or delete a Subnet, since you won't know the name) use the beta gcloud command:

gcloud beta container subnets list-usable

I landed here while looking for the answer and figured others trying to determine the best way to structure their subnets / ranges might be able to use the above command (which took me forever to track down).

Answer 4

(Can't comment on Alan's answer due to low reputation)

You can create a new subnetwork:

1. go to the "VPC network"
2. click on "default" (under name)
3. click on "Add subnet"
4. define the subnet range / zone

Then on GKE when you create a new cluster, select your new subnetwork. This should allow you to create more clusters without running into the error Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork: 5.

Answer 5

You can create clusters via gcloud and add --create-subnetwork "".

See: https://cloud.google.com/sdk/gcloud/reference/container/clusters/create#--create-subnetwork

This will create a new subnet with each new cluster so the "5 Secondary IP ranges per subnet" quota won't be reached.