pip install from a Google Cloud Storage bucket

Question

I would like to install a Python package (my-package-1.0.0.tar.gz) from a Google Cloud Storage bucket.

It works well with publicly available buckets (Allow Public Read access on a GCS bucket?) or with signed url (https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers). However I would like to know if there would be another solution.

I would like to have a private Python package, accessible from a user-friendly (or may I say developer-friendly-url) such as https://storage.googleapis.com/pypi-packages/my-package/my-package-1.0.0.tar.gz.

I would like this package to be accessible from other Google Cloud Platform tools such as Dataflow, Cloud ML, Composer or GKE

Do you think it's something doable?

Answer 1

The way I found was to use a feature from GitLab : deploy token. Basically, this feature allows you to secure a url that can be used with pip install.

For instance :

pip install -e git+https://${GITLAB_PYPI_DEPLOY_TOKEN}@git.repo.net/my-repo.git#egg=my_repo

Please note that this is not really an answer to the initial question but a way that I solve my specific problem.

Answer 2

You can assign a custom domain to your bucket in order to get the friendly url you desire and then make the object you desire accessible with the signed url.

To assign a custom domain you will need to create a bucket whose name is the same as the domain. For example, for a domain like www.anyname.com you will need to call your bucket www.anyname.com

After this you create a CNAME record that points to c.storage.googleapis.com.