Ci/CD using Google Cloud build with Django on App Engine failed

Question

Background: Trying to automate my build process using the new Google Cloud Build with Django on standard app engine. I started with Django polls example provided by the good Google folks here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard_python37/django

Environment: Python3.7 with Django 3.0.1

Made a few changes to the above and now my requirements.txt looks like:

requirements.txt

coverage==5.0.1
Django==3.0.1
entrypoints==0.3
flake8==3.7.9
mccabe==0.6.1
mysqlclient==1.4.6
pycodestyle==2.5.0
pyflakes==2.1.1
pytz==2019.3
sqlparse==0.3.0

app.yaml

runtime: python37

handlers:
# This configures Google App Engine to serve the files in the app's static
# directory.
- url: /static
  static_dir: static/

# This handler routes all requests not caught above to your main app. It is
# required when static routes are defined, but can be omitted (along with
# the entire handlers section) when there are no static files defined.
- url: /.*
  script: auto

env_variables:
# the secret key used for the Django app (from PROJECT-DIRECTORY/settings.py)
  SECRET_KEY: 'DJANGO-SECRET-KEY'
  DEBUG: 'False' # always False for deployment

# everything after /cloudsql/ can be found by entering >> gcloud sql instances describe DATABASE-NAME << in your Terminal
# the DATABASE-NAME is the name you gave your project's PostgreSQL database
# the second line from the describe output called connectionName can be copied and pasted after /cloudsql/
  DB_HOST: '/cloudsql/annular-will-XXXX:asia-east2:XXXX'
  DB_PORT: '5432' # PostgreSQL port
  DB_NAME: 'XXX'
  DB_USER: 'XXX'
  DB_PASSWORD: 'XXXXX'

# [END django_app]

Time to deploy using command line gcloud app deploy app.yaml

Works, yay!!

Time to automate the build and follow the steps here:

https://cloud.google.com/source-repositories/docs/quickstart-triggering-builds-with-source-repositories

I made some minor change of name of cloudbuild.yaml to cloud-build.yaml

Result: The build gets triggered on push to the branch! Things are looking good so far!

Lets look at the cloud_build.yaml: Note, this is part of the steps mentioned in the quick start guide of [cloud build][1]

https://cloud.google.com/source-repositories/docs/quickstart-triggering-builds-with-source-repositories

steps:
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]
timeout: "1600s"

Result: the app gets deployed, but wait, the static assets are not loading :( Let's just add the step in our cloud-build.yaml, so now it looks like:

- name: 'python:3.7'
  entrypoint: python3
  args: ['-m', 'pip', 'install', '-t', '.', '-r', 'requirements.txt']
- name: 'python:3.7'
  entrypoint: python3
  args: ['./manage.py', 'collectstatic', '--noinput']
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]
timeout: "1600s"

We need to install requirements.txt as collectstatic would require Django and other dependencies to be installed. Note the -t parameter passed to pip install. This is passed because every step of google cloud-build.yaml is run in a separate docker image and the only thing common is the /workspace directory. So it make sense to install everything within the workspace. After this step the collectstatic works!

Looking at the history of the Cloud Build all looks green: [![enter image description here][2]][2]

But the app refuses to boot and the instance dies with the following error:

2019-12-27 01:10:49 default[20191227t010033]  [2019-12-27 01:10:49 +0000] [7] [INFO] Starting gunicorn 20.0.4
2019-12-27 01:10:49 default[20191227t010033]  [2019-12-27 01:10:49 +0000] [7] [INFO] Listening at: http://0.0.0.0:8081 (7)
2019-12-27 01:10:49 default[20191227t010033]  [2019-12-27 01:10:49 +0000] [7] [INFO] Using worker: threads
2019-12-27 01:10:49 default[20191227t010033]  [2019-12-27 01:10:49 +0000] [18] [INFO] Booting worker with pid: 18
2019-12-27 01:10:49 default[20191227t010033]  [2019-12-27 01:10:49 +0000] [22] [INFO] Booting worker with pid: 22
2019-12-27 01:10:51 default[20191227t010033]  [2019-12-27 01:10:51 +0000] [22] [ERROR] Exception in worker process
2019-12-27 01:10:51 default[20191227t010033]  Traceback (most recent call last):    File "/srv/django/db/backends/mysql/base.py", line 16, in <module>      import MySQLdb as Database    File "/srv/MySQLdb/__init__.py", line 18, in <module>      from . import _mysql  ImportError: libmariadb.so.3: cannot open shared object file: No such file or directory
2019-12-27 01:10:51 default[20191227t010033]
2019-12-27 01:10:51 default[20191227t010033]  The above exception was the direct cause of the following exception:
2019-12-27 01:10:51 default[20191227t010033]
2019-12-27 01:10:51 default[20191227t010033]  Traceback (most recent call last):    File "/env/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker      worker.init_process()    File "/env/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 92, in init_process      super().init_process()    File "/env/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process      self.load_wsgi()    File "/env/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi      self.wsgi = self.app.wsgi()    File "/env/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi      self.callable = self.load()    File "/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load      return self.load_wsgiapp()    File "/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp      return util.import_app(self.app_uri)    File "/env/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app      mod = importlib.import_module(module)    File "/opt/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module      return _bootstrap._gcd_import(name[level:], package, level)    File "<frozen importlib._bootstrap>", line 1006, in _gcd_import    File "<frozen importlib._bootstrap>", line 983, in _find_and_load    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked    File "<frozen importlib._bootstrap>", line 677, in _load_unlocked    File "<frozen importlib._bootstrap_external>", line 728, in exec_module    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed    File "/srv/main.py", line 1, in <module>      from mysite.wsgi import application    File "/srv/mysite/wsgi.py", line 16, in <module>      application = get_wsgi_application()    File "/srv/django/core/wsgi.py", line 12, in get_wsgi_application      django.setup(set_prefix=False)    File "/srv/django/__init__.py", line 24, in setup      apps.populate(settings.INSTALLED_APPS)    File "/srv/django/apps/registry.py", line 114, in populate      app_config.import_models()    File "/srv/django/apps/config.py", line 211, in import_models      self.models_module = import_module(models_module_name)    File "/opt/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module      return _bootstrap._gcd_import(name[level:], package, level)    File "/srv/polls/models.py", line 7, in <module>      class Question(models.Model):    File "/srv/django/db/models/base.py", line 121, in __new__      new_class.add_to_class('_meta', Options(meta, app_label))    File "/srv/django/db/models/base.py", line 325, in add_to_class      value.contribute_to_class(cls, name)    File "/srv/django/db/models/options.py", line 208, in contribute_to_class      self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())    File "/srv/django/db/__init__.py", line 28, in __getattr__      return getattr(connections[DEFAULT_DB_ALIAS], item)    File "/srv/django/db/utils.py", line 207, in __getitem__      backend = load_backend(db['ENGINE'])    File "/srv/django/db/utils.py", line 111, in load_backend      return import_module('%s.base' % backend_name)    File "/opt/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module      return _bootstrap._gcd_import(name[level:], package, level)    File "/srv/django/db/backends/mysql/base.py", line 21, in <module>      ) from err  django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module.
2019-12-27 01:10:51 default[20191227t010033]  Did you install mysqlclient?
2019-12-27 01:10:51 default[20191227t010033]  [2019-12-27 01:10:51 +0000] [22] [INFO] Worker exiting (pid: 22)
2019-12-27 01:10:51 default[20191227t010033]  [2019-12-27 01:10:51 +0000] [18] [ERROR] Exception in worker process
2019-12-27 01:10:51 default[20191227t010033]  Traceback (most recent call last):    File "/srv/django/db/backends/mysql/base.py", line 16, in <module>      import MySQLdb as Database    File "/srv/MySQLdb/__init__.py", line 18, in <module>      from . import _mysql  ImportError: libmariadb.so.3: cannot open shared object file: No such file or directory
2019-12-27 01:10:51 default[20191227t010033]
2019-12-27 01:10:51 default[20191227t010033]  The above exception was the direct cause of the following exception:
2019-12-27 01:10:51 default[20191227t010033]
2019-12-27 01:10:51 default[20191227t010033]  Traceback (most recent call last):    File "/env/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker      worker.init_process()    File "/env/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 92, in init_process      super().init_process()    File "/env/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process      self.load_wsgi()    File "/env/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi      self.wsgi = self.app.wsgi()    File "/env/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi      self.callable = self.load()    File "/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load      return self.load_wsgiapp()    File "/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp      return util.import_app(self.app_uri)    File "/env/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app      mod = importlib.import_module(module)    File "/opt/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module      return _bootstrap._gcd_import(name[level:], package, level)    File "<frozen importlib._bootstrap>", line 1006, in _gcd_import    File "<frozen importlib._bootstrap>", line 983, in _find_and_load    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked    File "<frozen importlib._bootstrap>", line 677, in _load_unlocked    File "<frozen importlib._bootstrap_external>", line 728, in exec_module    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed    File "/srv/main.py", line 1, in <module>      from mysite.wsgi import application    File "/srv/mysite/wsgi.py", line 16, in <module>      application = get_wsgi_application()    File "/srv/django/core/wsgi.py", line 12, in get_wsgi_application      django.setup(set_prefix=False)    File "/srv/django/__init__.py", line 24, in setup      apps.populate(settings.INSTALLED_APPS)    File "/srv/django/apps/registry.py", line 114, in populate      app_config.import_models()    File "/srv/django/apps/config.py", line 211, in import_models      self.models_module = import_module(models_module_name)    File "/opt/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module      return _bootstrap._gcd_import(name[level:], package, level)    File "/srv/polls/models.py", line 7, in <module>      class Question(models.Model):    File "/srv/django/db/models/base.py", line 121, in __new__      new_class.add_to_class('_meta', Options(meta, app_label))    File "/srv/django/db/models/base.py", line 325, in add_to_class      value.contribute_to_class(cls, name)    File "/srv/django/db/models/options.py", line 208, in contribute_to_class      self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())    File "/srv/django/db/__init__.py", line 28, in __getattr__      return getattr(connections[DEFAULT_DB_ALIAS], item)    File "/srv/django/db/utils.py", line 207, in __getitem__      backend = load_backend(db['ENGINE'])    File "/srv/django/db/utils.py", line 111, in load_backend      return import_module('%s.base' % backend_name)    File "/opt/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module      return _bootstrap._gcd_import(name[level:], package, level)    File "/srv/django/db/backends/mysql/base.py", line 21, in <module>      ) from err  django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module.
2019-12-27 01:10:51 default[20191227t010033]  Did you install mysqlclient?
2019-12-27 01:10:51 default[20191227t010033]  [2019-12-27 01:10:51 +0000] [18] [INFO] Worker exiting (pid: 18)
2019-12-27 01:10:51 default[20191227t010033]  [2019-12-27 01:10:51 +0000] [7] [INFO] Shutting down: Master
2019-12-27 01:10:51 default[20191227t010033]  [2019-12-27 01:10:51 +0000] [7] [INFO] Reason: Worker failed to boot.

If it helps, i have seen another weird error during the course of debugging:

2019-12-27 01:31:35 default[20191227t092757]  Traceback (most recent call last):    File "/env/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker      worker.init_process()    File "/env/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 92, in init_process      super().init_process()    File "/env/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process      self.load_wsgi()    File "/env/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi      self.wsgi = self.app.wsgi()    File "/env/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi      self.callable = self.load()    File "/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load      return self.load_wsgiapp()    File "/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp      return util.import_app(self.app_uri)    File "/env/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app      mod = importlib.import_module(module)    File "/opt/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module      return _bootstrap._gcd_import(name[level:], package, level)    File "<frozen importlib._bootstrap>", line 1006, in _gcd_import    File "<frozen importlib._bootstrap>", line 983, in _find_and_load    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked    File "<frozen importlib._bootstrap>", line 677, in _load_unlocked    File "<frozen importlib._bootstrap_external>", line 728, in exec_module    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed    File "/srv/main.py", line 1, in <module>      from mysite.wsgi import application    File "/srv/mysite/wsgi.py", line 12, in <module>      from django.core.wsgi import get_wsgi_application  ModuleNotFoundError: No module named 'django'```


At this point, i have almost given up on google cloud CD. If you have any working example of how to CI/CD on google app engine with Django, pls let me know.

further debugging:

 - Switched to gcloud app deploy --version 1 and it still works. Note that this happens from the command line from my project and my static directory in settings:

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.1/howto/static-files/

STATIC_ROOT = 'static'
STATIC_URL = '/static/'

Before the app deploy, i run python collectstatic thus creating the static directory. Hence i assume when i do app deploy, the image already has the static directory created. However, during cloud build, we had to explicitly make it as a step. Just thought of mentioning it.

-rw-r--r--    1 amit  staff   868B Dec 26 13:53 README.md
-rw-r--r--    1 amit  staff   1.1K Dec 27 09:21 app.yaml
-rw-r--r--    1 amit  staff    88B Dec 27 09:43 cloud-build.yaml
-rw-r--r--    1 amit  staff   492B Dec 26 14:34 main.py
-rwxr-xr-x    1 amit  staff   538B Dec 26 13:53 manage.py*
drwxr-xr-x    7 amit  staff   224B Dec 26 14:15 mysite/
drwxr-xr-x   12 amit  staff   384B Dec 27 00:27 polls/
-rwxr-xr-x    1 amit  staff   108B Dec 26 13:53 proxy.sh*
-rw-r--r--    1 amit  staff   173B Dec 27 09:43 requirements.txt
drwxr-xr-x    3 amit  staff    96B Dec 26 13:57 static/
drwxr-xr-x    6 amit  staff   192B Dec 26 15:27 venv/


  [1]: https://cloud.google.com/source-repositories/docs/quickstart-triggering-builds-with-source-repositories
  [2]: https://i.stack.imgur.com/aalUR.png

Answer 1

I took @methkal Khalawi comment To avoid all the headache with mysqlclient, I suggest you use pymysql it's more stable included in the requirements file.

However, there is an existing issue with PyMySQL and hence it doesn't work with Django 3.x. Here is the link https://github.com/PyMySQL/PyMySQL/issues/790

In order to circumvent the above do the following in your settings file:

import pymysql
# Hack to make PyMySql work with Django 3.x
# #https://github.com/PyMySQL/PyMySQL/issues/790
pymysql.version_info = (1, 4, 6, 'final', 0)  # change mysqlclient version
pymysql.install_as_MySQLdb()

Now, get rid of mysqlclient from requirements.txt and get PyMySql:

Django==3.0.2
PyMySQL==0.9.3

Now, cloud-build.yaml looks like this:

steps:
- name: 'python:3.7'
  entrypoint: python3
  args: ['-m', 'pip', 'install', '-t', '.', '-r', 'requirements.txt']
- name: 'python:3.7'
  entrypoint: python3
  args: ['./manage.py', 'collectstatic', '--noinput']
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]
timeout: "1600s"

And it Works!