Configure Django and Google Cloud Storage?

Question

I am not using Appengine.

I have a plain vanilla Django application running on a VM. I want to use Google Cloud Storage for serving my staticfiles, and also for uploading/serving my media files.

I have a bucket.

How do I link my Django application to my bucket? I've tried django-storages. That may work, but what do I have to do to prepare my bucket to be used by my django application? And what baseline configuration do I need in my Django settings?

Current settings:

# Google Cloud Storage
# http://django-storages.readthedocs.org/en/latest/backends/apache_libcloud.html
LIBCLOUD_PROVIDERS = {
    'google': {
        'type'  : 'libcloud.storage.types.Provider.GOOGLE_STORAGE',
        'user'  : <I have no idea>,
        'key'   : <ditto above>,
        'bucket': <my bucket name>,
    }
}

DEFAULT_LIBCLOUD_PROVIDER = 'google'
DEFAULT_FILE_STORAGE = 'storages.backends.apache_libcloud.LibCloudStorage'
STATICFILES_STORAGE = 'storages.backends.apache_libcloud.LibCloudStorage'

Answer 1

Django-storages has a backend for Google Cloud Storage, but it is not documented, I realised looking in the repo. Got it working with this setup:

DEFAULT_FILE_STORAGE = 'storages.backends.gs.GSBotoStorage'
GS_ACCESS_KEY_ID = 'YourID'
GS_SECRET_ACCESS_KEY = 'YourKEY'
GS_BUCKET_NAME = 'YourBucket'
STATICFILES_STORAGE = 'storages.backends.gs.GSBotoStorage'

To get YourKEY and YourID you should create Interoperability keys, in the settings tab.

Hope it helps and you don't have to learn it the hard way :)

Ah in case you haven't yet, the dependencies are:

pip install django-storages
pip install boto

Answer 2

Django-storages is, in fact, a viable alternative. You must be careful with it's Google Cloud backend though as the url() method it provides causes unnecessary HTTP calls to Google. (Django calls .url() when rendering static files, for example).

https://github.com/jschneier/django-storages/issues/491

settings.py

    DEFAULT_FILE_STORAGE = 'config.storage_backends.GoogleCloudMediaStorage'
    STATICFILES_STORAGE = 'config.storage_backends.GoogleCloudStaticStorage'
    GS_PROJECT_ID = '<google-cloud-project-id>'
    GS_MEDIA_BUCKET_NAME = '<name-of-static-bucket>'
    GS_STATIC_BUCKET_NAME = '<name-of-static-bucket>'
    STATIC_URL = 'https://storage.googleapis.com/{}/'.format(GS_STATIC_BUCKET_NAME)
    MEDIA_URL = 'https://storage.googleapis.com/{}/'.format(GS_MEDIA_BUCKET_NAME)

storage_backends.py

    """
    GoogleCloudStorage extensions suitable for handing Django's
    Static and Media files.

    Requires following settings:
    MEDIA_URL, GS_MEDIA_BUCKET_NAME
    STATIC_URL, GS_STATIC_BUCKET_NAME

    In addition to
    https://django-storages.readthedocs.io/en/latest/backends/gcloud.html
    """
    from django.conf import settings
    from storages.backends.gcloud import GoogleCloudStorage
    from storages.utils import setting
    from urllib.parse import urljoin


    class GoogleCloudMediaStorage(GoogleCloudStorage):
        """GoogleCloudStorage suitable for Django's Media files."""

        def __init__(self, *args, **kwargs):
            if not settings.MEDIA_URL:
                raise Exception('MEDIA_URL has not been configured')
            kwargs['bucket_name'] = setting('GS_MEDIA_BUCKET_NAME', strict=True)
            super(GoogleCloudMediaStorage, self).__init__(*args, **kwargs)

        def url(self, name):
            """.url that doesn't call Google."""
            return urljoin(settings.MEDIA_URL, name)


    class GoogleCloudStaticStorage(GoogleCloudStorage):
        """GoogleCloudStorage suitable for Django's Static files"""

        def __init__(self, *args, **kwargs):
            if not settings.STATIC_URL:
                raise Exception('STATIC_URL has not been configured')
            kwargs['bucket_name'] = setting('GS_STATIC_BUCKET_NAME', strict=True)
            super(GoogleCloudStaticStorage, self).__init__(*args, **kwargs)

        def url(self, name):
            """.url that doesn't call Google."""
            return urljoin(settings.STATIC_URL, name)

Note: authentication is handled by default via the GOOGLE_APPLICATION_CREDENTIALS environment variable.

https://cloud.google.com/docs/authentication/production#setting_the_environment_variable

Answer 3

So, this basically will work. (With this library and settings).

The trick to making it work, is knowing where to get the 'user' and 'key' parameters for libcloud.

On Google Cloud Console > Storage, click Settings. Then click on the right-hand tab called Interoperability. On that panel, is a lone button, which says something like Enable Interoperability. Click it.

Voila! You now have a username and key.

---

Note: Do not use django-storages from pypi. It has not been updated, and doesn't work with recent releases of Django.

Use this version:

pip install -e 'git+https://github.com/jschneier/django-storages.git#egg=django-storages'

---

Edit: If you want to use a reverse proxy, then you may consider my slightly modified version. https://github.com/jschneier/django-storages/compare/master...halfnibble:master

Description: Under certain circumstances, it may be necessary to load files using a reverse proxy. This could be used to alleviate cross-origin request errors.

This small PR allows the developer to set an optional LIBCLOUD_PROXY_URL in settings.py.

Example Usage

# Apache VirtualHost conf
ProxyPass /foo http://storage.googleapis.com
ProxyPassReverse /foo http://storage.googleapis.com


# settings.py
LIBCLOUD_PROXY_URL = '/foo/'

Answer 4

As the latest version, access key and key ID changed to service account file. And we want to use a bucket with 2 folders static and media like a local server. Below low the update configs:

Create a file like gcloud_storages.py:

"""
Modify django-storages for GCloud to set static, media folder in a bucket
"""
from django.conf import settings
from storages.backends.gcloud import GoogleCloudStorage


class GoogleCloudMediaStorage(GoogleCloudStorage):
    """
    GoogleCloudStorage suitable for Django's Media files.
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'media'
        super(GoogleCloudMediaStorage, self).__init__(*args, **kwargs)


class GoogleCloudStaticStorage(GoogleCloudStorage):
    """
    GoogleCloudStorage suitable for Django's Static files
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'static'
        super(GoogleCloudStaticStorage, self).__init__(*args, **kwargs)

Use location argument to set the location of static, media files in bucket.

In settings.py

from google.oauth2 import service_account
...

GOOGLE_APPLICATION_CREDENTIALS = '/path/service-account.json'
DEFAULT_FILE_STORAGE = 'app.gcloud_storages.GoogleCloudMediaStorage'
STATICFILES_STORAGE = 'app.gcloud_storages.GoogleCloudStaticStorage'
GS_BUCKET_NAME = 'name-of-bucket'
GS_PROJECT_ID = 'project-id'
GS_DEFAULT_ACL = 'publicRead'
GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    GOOGLE_APPLICATION_CREDENTIALS
)

Answer 5

Since I cannot comment on Alan Wagner's answer, here is an addition.

If you are using python3, you may get this error,

...
ImportError: No module named 'google_compute_engine'

If so, you will need to install google-compute-engine. The /etc/boto.cfg file tells python to use the 2.7 version of the library. You will have to run this next line to regenerate /etc/boto.cfg.

python3 -c "from google_compute_engine.boto.boto_config import BotoConfig; BotoConfig()"

Another error you may hit,

...
File "/app/venv/lib/python3.4/site-packages/boto/gs/connection.py", line 95, in create_bucket
    data=get_utf8_value(data))
File "/app/venv/lib/python3.4/site-packages/boto/s3/connection.py", line 656, in make_request
    auth_path = self.calling_format.build_auth_path(bucket, key)
File "/app/venv/lib/python3.4/site-packages/boto/s3/connection.py", line 94, in build_auth_path
    path = '/' + bucket
TypeError: Can't convert 'bytes' object to str implicitly

I made a pull request to fix this. You may use my repo as a pip dependency if you wish until it gets merged.

I will try to keep this repo up to date. I have set the default develop branch as protected. I am the only one who can commit/approve merge requests. I have also only made one commit.

You will have to install google-compute-engine and run that line above before you can instalkl/build my boto repo.

Answer 6

I have detailed my step by step process on another thread

• Serve Static files from Google Cloud Storage Bucket (for Django App hosted on GCE)

Here are my main references:

• https://django-storages.readthedocs.io/en/latest/backends/gcloud.html
• https://medium.com/@umeshsaruk/upload-to-google-cloud-storage-using-django-storages-72ddec2f0d05

I used the following packages:

pip3 install django-storages # https://pypi.org/project/django-storages/
pip3 install google-cloud-storage # https://pypi.org/project/google-cloud-storage/