Import python module stored in google cloud storage to local script

Question

I have a python script that is stored in google cloud storage . Is there any way to import that python module to my local script ?

Answer 1

if you are working under linux, you can maybe mount your google cloud storage as a remote file system (e.g. via sshfs, but i don't know how it would work for gcs...) and then add the folder of the newly mounted fs to your local $PYTHONPATH. Maybe this could be a way to add the script locally.

Hope it helps.

EDIT Maybe have a look at this post: GCE Use Cloud Storage Bucket as Mounted Drive

Answer 2

It could be done, but locally mounting it in some way like WWhisperer suggested is a better idea.

If you wanted to, though, there's some interesting discussion on importing remote resources into Python elsewhere on Stack overflow and on O'Reilly.

Answer 3

It's possible to override Pythons import process so you could load the source from somewhere other than local disk. Based on the answer at https://stackoverflow.com/a/43573798/1880657 you can build a MetaPathFinder and Loader.

import sys
import types
# One needs to supply some mechanism to get a string with the source code from the
# remote location.
# gcsutils is a utility library in our project. 
# Here we use it to get a blob from a predefined bucket, to return the source code.
from .gcsutils import get_blob
from importlib.abc import Loader, MetaPathFinder
from importlib.machinery import ModuleSpec, SourceFileLoader
import os


class GCSMetaFinder(MetaPathFinder):
    def find_spec(self, fullname, path, target=None):
        # we prefix our code that is not on disk with a "virtual_prefix" that we map into a cloud storage path.
        if fullname.startswith('virtual_prefix'):
            if len(fullname.split(".")) <= 2:
                # We need to create a "package" for our virtual prefix so we can load modules under it.
                return ModuleSpec(fullname, PlaceHolderLoader())
            try:
                _, relative_path = fullname.split('.')
                filename = "code/{}.py".format(relative_path)
                # get_blob is a helper function that uses google.cloud.storage with a preset bucket to get a blob (https://googleapis.dev/python/storage/latest/buckets.html#google.cloud.storage.bucket.Bucket.blob)
                blob = get_blob(filename)
                if blob.exists():
                    return ModuleSpec(fullname, GCSLoader(blob))
            except AssertionError as e:
                return None
        return None

class PlaceHolderLoader(Loader):
    # creates a "package" so that python thinks modules can exist under here.
    def create_module(self, spec):
        dummy_module = types.ModuleType(spec.name)
        dummy_module.__path__ = []
        return dummy_module

    def exec_module(self, module):
        pass


class GCSLoader(Loader):
    def __init__(self, blob):
        self.blob = blob

    def create_module(self, spec):
        return None # use default module creation semantics

    def exec_module(self, module):
        data = self.blob.download_as_string()
        exec(data, vars(module))


def install():
    """Inserts the finder into the import machinery"""
    sys.meta_path.insert(0, GCSMetaFinder())

I've updated this code based on what we have now tested for our project.