How to connect to Google Cloud Composer Metadata DB (Airflow metadata DB - Cloud SQL) from App Engine Application

Question

I have a App Engine Application written in Flask Python 3.7

My usecase is to get information from Composer Metadata DB (dag runs, dag success, dag failures etc) from Composer metadata DB and show as a dashboard inside App Engine Application (few charts).

Homework Done so far - I was able to run sql queries on Composer metadata after logging in to one of the worker nodes (as worker nodes already have Cloud SQL proxy running which connects to Cloud SQL running in other container). This was done after creating a Compute engine under same VPC as of Composer and then doing ssh from Compute engine to one of the worker nodes.

Now the question is how to connect to Composer metadata DB that is under VPC from App Engine application using Cloud SQL Proxy ?

Answer 1

I would look at Serverless VPC connector, although it designed mainly for App Engine and Cloud Functions, you may consider implement this connector on a Flask app side that gives you opportunity to unify network between App engine and Compute engine nodes parties, thus you would be able to reach Cloud SQL proxy as well.

The setup is fairly much simple, though you just have to attach connector to the specific VPC network and region in the particular GCP project. The IP addresses pool must be in CIDR /28 range, reserved for a connector usage.