Composer auto scaling?

7
votes

Given that GCP composer is running with GKE/GCE, is it auto scaling? Now I have 3 nodes in the cluster to support say, 100 DAGs.

Later, if I have about 300 DAGs, will it scale up itself (w/ celery workers)?

3
google-cloud-composer
Do you have any idea whether cloud composer supports auto scaling or not? I was creating infrastructure using Terraform but I don't see any such option of auto-scaling there. Is it still not there? - Ashish Kumar

3 Answers

5
votes

We currently don't support auto-scaling but it's on our roadmap. You could however manually scale up/down the GKE cluster by updating the nodeCount value.

5
votes

I just published an article showing how to enable autoscale in its underlying kubernetes https://link.medium.com/AMUlwUIkD0.

Basically:

  1. Enable autoscale on the node level

  2. Apply HorizontalPodAutoscaler to airflow-worker deployment

  3. Increase some airflow config parameter to remove the bottleneck

This has been tested in composer-1.7.2 and composer-1.7.5 but might be applicable to other version as well

Do check it out

1
votes

No autoscaling at this point, but I expect it's in the roadmap.

Astronomer is working on an autoscaling Airflow service using Kubernetes, it'll probably launch around the time Google Composer launches autoscaling... so good times are coming :)