Google Cloud Functions sometimes doesn't find @google-cloud/* dependencies although the buildpack builds and runs fine in Docker on local

Question

I am deploying a Google Cloud function and it fails to deploy because it sometimes doesn't find the module @google-cloud/pubsub and when it finds it, it instead fails on a transitive dependency from the pubsub package. I have had this with @google-cloud/storage as well and never managed to solve it.

See the logs screenshot below in this post for an example of the issue. It shows two consecutive deployments.

• In the first deployment, the @google-cloud/pubsub is not found although it is listed in package.json
• In the second deployment, the pubsub package is found(!), but then instead it fails on the transitive dependency google-gax.

Things I tried

• Deploy using Google Cloud Build, hooked up to my github repo (deplyoment fails)
• Deploy from local command line. See script target deployAnalyzer in package.json below. (deployment fails with same errors a Google Cloud Builds)
• Deploy the function locally to the functions framework. See script target devFunctionAnalyzer below. (Deplyoment works and the function can be invoked)
• Build locally using pack to repeat the same build procedure as Google Cloud Build. Result: Image builds and runs fine!

In short: It work fine locally, both in the Functions Framework and running the docker image built using the google nodejs buildpack.

The Full Code

Work in progress: https://github.com/aweijnitz/ornitho-de-monitor

Minimalistic repo with one dependency, that fails with the same error: https://github.com/aweijnitz/cloud-functions-dependency-test-01/tree/main

EDIT Added link to stripped down repo that illustrates the problem without all the application code. Just a single dependency now.

package.json

{
  "name": "ornitho-de-monitor",
  "version": "0.0.1",
  "description": "Collect latest interesting observations from ornitho.de",
  "main": "src/index.js",
  "engines": {
    "node": ">=12.13"
  },
  "scripts": {
    "devFunctionAnalyzer": "npx functions-framework --target=analyzeObservations --signature-type=event",
    "devFunctionNotifier": "npx functions-framework --target=notifyAll --signature-type=event",
    "buildCloudRunContainer": "gcloud builds submit --tag gcr.io/ornitho-de-monitor/ornitho-de-scraper",
    "deployCloudRunContainer": "gcloud run deploy ornitho-de-scraper --region=europe-west6 --image gcr.io/ornitho-de-monitor/ornitho-de-scraper --platform managed",
    "deployAnalyzer": "gcloud functions deploy analyzeObservations --source=./src --entry-point=analyzeObservations --runtime nodejs12 --memory=256MB --max-instances=3 --trigger-topic=ornitho-bus",
    "deployNotifier": "gcloud functions deploy notifyAll --source=./src --entry-point=notifyAll --runtime nodejs12 --memory=256MB --max-instances=3 --trigger-topic=ornitho-bus",
    "invoke": "gcloud functions call ornitho-de-monitor --data '{\"name\":\"Keyboard Cat\"}'",
    "invokeEncoded": "DATA=$(printf 'Hello!'|base64) && gcloud functions call helloPubSub --data '{\"data\":\"'$DATA'\"}'",
    "viewFunctionLogs": "gcloud functions logs read ornitho-de-monitor",
    "serveTestFile": "cd testdata && python -m SimpleHTTPServer 8000",
    "test": "echo \"No test specified\" && exit 0"
  },
  "author": "Anders Weijnitz",
  "license": "ISC",
  "dependencies": {
    "@google-cloud/pubsub": "^2.7.0",
    "@google-cloud/storage": "^5.7.0",
    "express": "^4.17.1",
    "express-async-handler": "^1.1.4",
    "express-rate-limit": "^5.2.3",
    "knex": "^0.21.15",
    "pg": "^8.5.1"
  },
  "devDependencies": {
    "@google-cloud/functions-framework": "^1.7.1"
  }
}

These are the relevant steps from cloudbuild.yaml

steps:
  # Retrieve credentials from Fort Knox
  - name: 'gcr.io/cloud-builders/gcloud'
    entrypoint: 'bash'
    args: [ '-c', "gcloud secrets versions access latest --secret=ornitho-pubsub --format='get(payload.data)' | tr '_-' '/+' | base64 -d > pubsubkey.json" ]
  # Install dependencies
  - name: 'gcr.io/cloud-builders/npm'
    args: [ 'install' ]
    dir: '.'
  # Run tests
  - name: 'gcr.io/cloud-builders/npm'
    args: [ 'test' ]
    dir: '.'
  # Deploy the notifier function
  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - 'functions'
      - 'deploy'
      - 'notifyAll'
      - '--source=./src'
      - '--entry-point=notifyAll'
      - '--runtime=nodejs12'
      - '--memory=128MB'
      - '--max-instances=3'
      - '--trigger-topic=ornitho-bus'
  # Deploy the analyzer function
  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - 'functions'
      - 'deploy'
      - 'analyzeObservations'
      - '--source=./src'
      - '--entry-point=analyzeObservations'
      - '--runtime=nodejs12'
      - '--memory=128MB'
      - '--max-instances=3'
      - '--trigger-topic=ornitho-bus'

Answer 1

SOLVED!

In turns out that the --source parameter should not point to the location of the application source, but to the location of the project root, where package.json resides. Bad: --source=./src Good: --source=. .

For more details, see github.com/aweijnitz/cloud-functions-dependency-test-01 –