App Engine Standard, Serverless VPCs, Cloud Memorystore giving significant amount of timeouts

Question

We configured our App Engine Standard python 3 service to connect to Cloud Memorystore via the Serverless VPC service (per the documentation, and other stack overflow threads). (I've included the app.yaml config below). This all worked well, unless an instance went idle for a little while. Over time we saw a high volume of:

Long unexplained hangs when making calls to Memorystore, even though they eventually worked
redis.exceptions.ConnectionError: Error 110 connecting to 10.0.0.12:6379. Connection timed out.
redis.exceptions.TimeoutError: Timeout reading from socket

These happened to the point where I had to move back to App Engine Flexible, where the service runs great without any of the above problems.

My conclusion is that Serverless VPC does not handle the fact that the redis client tries hard to leave the connection to redis open all the time. I tried a few variations of timeout settings, but nothing that helped. Has anyone successfully deployed App Engine Standard, Memorystore, and Serverless VPC?

env_variables:
  REDISHOST: <IP>
  REDISPORT: 6379

network:
  name: "projects/<PROJECT-ID>/global/networks/default"

vpc_access_connector:
  name: "projects/<PROJECT-ID>/locations/us-central1/connectors/<VPC-NAME>

Code used to connect to Memorystore (using redis-py):

REDIS_CLIENT = redis.StrictRedis(
    host=REDIS_HOST, 
    port=REDIS_PORT, 
    retry_on_timeout=True, 
    health_check_interval=30
)

(I tried various timeout settings but couldn't find anything that helped)

Could you please share the code that you are using to connect to Memorystore? Which Client Library are you using? — Nahuel Varela
@NahuelVarela added the code that instantiates the client, thx — jcjones1515

Jose V Jose V · Accepted Answer · 2019-10-11T14:26:44

I created a Memorystore instance and a Serverless VPC Access connector as stated in the docs (https://cloud.google.com/vpc/docs/configure-serverless-vpc-access), then deployed this sample (https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard_python37/redis) from Google Cloud Platform Python doc samples repo to App Engine Standard after making some modifications:

This is my app.yaml:

runtime: python37

# Update with Redis instance details
env_variables:
  REDIS_HOST: <memorystore-ip-here>
  REDIS_PORT: 6379

# Update with Serverless VPC Access connector details
vpc_access_connector:
  name: 'projects/<project-id>/locations/<region>/connectors/<connector-name>'
# [END memorystore_app_yaml_standard]

I edited the code on main.py and used the snippet that you use to connect to the memorystore instance. It ended up like this:

redis_client = redis.StrictRedis(
    host=redis_host, port=redis_port,
    password=redis_password,
    retry_on_timeout=True, 
    health_check_interval=30
)

I edited the requirements.txt. I changed “redis==3.3.8” for “redis>=3.3.0”

Things to note:

Make sure to use “gcloud beta app deploy” instead of “gcloud app deploy” since it is needed in order for the Serverless VPC Access connector to work.
Make sure that the authorized network you set to the memorystore instance is the same that you select for the Serverless VPC Access connector

This works as expected for me, could you please check if this works for you?

App Engine Standard, Serverless VPCs, Cloud Memorystore giving significant amount of timeouts

2 Answers