HPA + Cluster Autoscaler + OPA within Federated Kubernetes cluster on GKE

2
votes

I'm setting up a federated kubernetes cluster with kubefed on the Google Container Engine (GKE) 1.8.3-gke.0.

And it seems like for a good HPA and cluster autoscaler I have to use Open Policy Agent as a kubernetes Admission Controller because of this:

By default, replicas are spread equally in all the underlying clusters. For example: if you have 3 registered clusters and you create a Federated Deployment with spec.replicas = 9, then each Deployment in the 3 clusters will have spec.replicas=3.

But in my case, the load would be dynamically changed in every region and every cluster should have dynamic pods number.

I can't find (or just can't see) examples or manuals regarding cases like mine. So, the question is:

What scenario should a policy have, if I have three clusters in my federated context, one for every region of GKE:

  • eu (1000 rps, nodes labeled with "region=eu")
  • us (200 rps, nodes labeled with "region=us")
  • asia (100 rps, nodes labeled with "region=asia")

It should be a single deployment to dynamically spread pods across those three clusters.

One pod should:

  • serve 100 rps
  • request 2 vCPUs + 2Gb RAM
  • be placed on a node solely (with anti-affinity)

How can I configure OPA to make that schema work, if this is possible?

Thanks in advance for any links to corresponding manuals.

1
kubernetesgoogle-kubernetes-engineopen-policy-agent

1 Answers

0
votes

What you are trying to do should be achivable through "Federated Horizontal Pod Autoscalers", one of their main use cases is exactly your scenario:

Quoting from the Requirements & Design Document of the Federated Pod Autoscaler:

Users can schedule replicas of same application, across the federated clusters, using replicaset (or deployment). Users however further might need to let the replicas be scaled independently in each cluster, depending on the current usage metrics of the replicas; including the CPU, memory and application defined custom metrics.

And from the actual documentation this passage from the conclusion describe the behaviour:

The use of federated HPA is to ensure workload replicas move to the cluster(s) where they are needed most, or in other words where the load is beyond expected threshold. The federated HPA feature achieves this by manipulating the min and max replicas on the HPAs it creates in the federated clusters. It actually relies on the in-cluster HPA controllers to monitor the metrics and update relevant fields [...] The federated HPA controller, on the other hand, monitors only the cluster-specific HPA object fields and updates the min replica and max replica fields of those in cluster HPA objects, which have replicas matching thresholds.

Therefore If I didn't misunderstood your needs, there is no reason to use a third product like Open Policy Agent or create policies.