ModuleNotFoundError when setting up Google Cloud Function with Cloud Source repository

Question

I am deploying a Google Cloud Function based on code in a BitBucket repository. I have already linked up the BitBucket account to Google Cloud "Source Repositories" and the Google Function can find the repo, etc. The problem is that my main.py function needs to call several functions in other packages/modules within my repository. I have some simple import statements at the top of my main.py file like this:

import base64
import json
from datetime import datetime
from competitor_scrape.headless import headless_browser
...

The first several lines load fine, but the 4th line (the one that call the module/function within my BitBucket repository) causes this error in Google Functions when I try to define my Google Cloud Function with the main.py in my repository:

message:  "Function load error: Code in file main.py can't be loaded.
Did you list all required modules in requirements.txt?
Detailed stack trace: Traceback (most recent call last):
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 211, in check_or_load_user_function
    _function_handler.load_user_function()
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 140, in load_user_function
    spec.loader.exec_module(main)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/user_code/main.py", line 4, in <module>
    from competitor_scrape.headless import headless_browser
ModuleNotFoundError: No module named 'competitor_scrape.headless'

It asks if I have "list[ed] all of the required modules in requirements.txt". Am I allowed to list the modules from my own repository? If so, how should I be doing that? I haven't been able to find any documentation on how to do this.

My current requirements.txt file looks like this:

google-cloud-pubsub
numpy==1.14.5
pandas==0.22.0
psycopg2==2.7.4
selenium==3.4.3
geopy==1.11.0
googlemaps==2.5.1
ratelimiter==1.2.0
sqlalchemy==1.2.0
zeep==2.5.0

EDIT/UPDATE

My repository file structure looks something like this right now:

.
├── competitor_scrape
│   ├── __init__.py
│   └── headless.py
├── main.py
└── requirements.txt

headless_browser is a function defined within headless.py. However, at this point my __init__.py inside competitor_scrape is empty (it was auto-generated by PyCharm). It seems that the __init__.py is probably the root of the problem. How should I be populating that script so that scripts/functions within competitor_scrape are available to the Google Cloud Function?

Can you provide a rough directory layout? Where is your main.py file in relation to the competitor_scrape and headless directories? Where is headless_browser defined? — Dustin Ingram

Dustin Ingram Dustin Ingram · Accepted Answer · 2018-09-19T00:30:03

You shouldn't include the modules in requirements.txt -- that's only for dependencies to be installed from PyPI.

An import statement like:

from competitor_scrape.headless import headless_browser

means that you should have a directory structure something like this:

.
├── competitor_scrape
│   ├── __init__.py
│   └── headless
│       └── __init__.py
├── main.py
└── requirements.txt

And in the competitor_scrape/headless/__init__.py file, you should have a variable called headless_browser.

ModuleNotFoundError when setting up Google Cloud Function with Cloud Source repository

EDIT/UPDATE

3 Answers