Why does the App engine (python3 - standard environment) generate aborted connections when connecting to a cloud sql -2nd Gen Mysql instance?

1
votes

I'm currently developing a project with python 3.7, Django 2.1, Mysql as database.

I'm deploying it in google cloud app engine standard environment and for the database I'm using a cloud SQL - MySql 2nd gen instance.

The application works well, however when I analyze the logs I see these errors:

"aborted connection - Got an error reading communication packets"

In this case the connection is being closed by my app (django).If I configure my app to have persistent connections and I put wait_timeout (i.e 60) in the config of the cloud sql, the error is:

"aborted connection - Got timeout reading communication packets".

I just determined that it's not a problem with sql cloud, or with the configuration of my application, but that it's an app engine problem. I came to this conclusion in the following way:

  • if I connect to the sql cloud instance through Mysql workbench, no connection is aborted
  • Similarly if I run my application on a local server, but connecting to cloud sql (through the cloud_sql_proxy), no error is generated and everything works perfect.

So my conclusion is that it is a problem of how the app engine connects to the cloud sql instance.

Why does this happen? How could it be solved?

2
google-app-enginegoogle-cloud-platformgoogle-cloud-sql

2 Answers

2
votes

The "Aborted connection" messages you're seeing, are usually triggered when a connection is closed improperly or there is a networking anomaly between the server and the client.

  • Sometimes Cloud SQL instances and GAE have long-live idle connections. In order to address this issue, it is recommended to set "wait_timeout' flag below 600 seconds - as you've already attempted to do so.

  • Another possible solution, is to implement application-level keepalives. SQLAlchemy provides “pre-ping” for this. Otherwise, generate activity on all open connections by sending a simple SQL statement such as "SELECT 1;" regularly, at least once every 5 minutes. Also consider using statements in your code like “with db.connect() as conn:” to control the connection’s lifetime.

1
votes

I believe this is because requests from App Engine applications to Cloud SQL are subject to the following time and connection limits:

  • For apps running in the App Engine standard environment, all database requests must finish within the HTTP request timer, around 60 seconds. For apps running in the flexible environment, all database requests must finish within 60 minutes.
  • Offline requests like cron tasks have a time limit of 10 minutes.
  • Requests to Cloud SQL have limitations based on the scaling type of the App Engine module and how long an instance can remain in memory (residence).
  • Each App Engine instance running in a standard environment cannot have more than 60 concurrent connections to a Cloud SQL instance. For applications written in Java 8 or Go 1.8, the limit is 100.
  • Connection Issues: If you see errors containing "Aborted connection nnnn to db:", it usually indicates that your application is not terminating connections properly. It could also be caused by network issues. This error does not mean that there are problems with your Cloud SQL instance.