Connecting Mongoose on Google App Engine

7
votes

I've deployed a custom Node.js app on Google App engine engine successfully. However I'm having issues starting the app because mongoose times out when attempting to connect. Frustratingly, the mongoose connects just fine on my local machine with the exact same parameters.

My MongoDb URI is of the form:

 mongodb://<dbuser>:<dbpassword>@xxxx.mlab.com:<portNumber>/d‌​b-name

As the URI implies, the db is hosted by mlab. It is a sandbox instance on Google Cloud Platform. I however did not deploy the db with Google Compute Engine. I merely opted to host in on GCP when going through the setup process on mlab.

I've come across similar questions on SO but most of them do not have an accepted answer. Just varying discourse in comments with no accepted solution.

My question therefore is, what differs between my App Engine instance and my local computer when trying to connect to the above URI? Does the fact that I chose for mlab to host it on GCP matter? Would it be any different if I chose for it to be hosted on Amazon AWS? What exactly is the root cause of the issue?

For reference, here are similar questions I've found:

Also, if it helps, mongodb is used as the db for business models, the datastore for the app's Express Session, and for storing transient data from socket.io.

Error stacktrace from App Engine after deployment:

2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent ENOENT: no such file or directory, open '/app/package.json'
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent ENOENT: no such file or directory, open '/app/package.json'
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent This is most likely not a problem with npm itself
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent and is related to npm not being able to find a file.
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent
2017-10-18 02:13:46 default[20171017t215757]
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! Please include the following file with any support request:
2017-10-18 02:13:46 default[20171017t215757]  npm ERR!     /app/npm-debug.log
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! Linux 3.16.0-4-amd64
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! argv "/nodejs/bin/node" "/nodejs/bin/npm" "start"
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! node v6.11.3
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! npm  v3.10.10
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! path /app/package.json
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! code ENOENT
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! errno -2
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! syscall open

Error stacktrace from running app locally in Google Cloud Shell:

/home/myuser/src/project-id/teammate-express-server/node_modules/mongodb/lib/mongo_client.js:421
          throw err
          ^
MongoError: failed to connect to server [ds147454.mlab.com:47454] on first connect [MongoError: connection 1 to ds147454.mlab.com:47454 timed out]
    at Pool.<anonymous> (/home/myuser/src/project-id/teammate-express-server/node_modules/mongodb-core/lib/topologies/server.js:336:35)
    at emitOne (events.js:96:13)
    at Pool.emit (events.js:188:7)
    at Connection.<anonymous> (/home/myuser/src/project-id/teammate-express-server/node_modules/mongodb-core/lib/connection/pool.js:280:12)
    at Connection.g (events.js:292:16)
    at emitTwo (events.js:106:13)
    at Connection.emit (events.js:191:7)
    at Socket.<anonymous> (/home/myuser/src/project-id/teammate-express-server/node_modules/mongodb-core/lib/connection/connection.js:197:10)
    at Socket.g (events.js:292:16)
    at emitNone (events.js:86:13)
    at Socket.emit (events.js:185:7)
    at Socket._onTimeout (net.js:338:8)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5)
3
node.jsmongodbgoogle-app-enginemongoosegoogle-compute-engine
Is it actually hosted on GCP? Meaning you can log in to the Google Cloud Platform console, look under GCE instances and find the VM server provisioned for it? If so, what firewall rules are associated with it? - BrettJ
@BrettJ I assume it is, mlab says so. The instance is however deployed and managed by mlab so it's not under my GCP console. I ran my project on my laptop and on a DigitalOcean Droplet, and they both connect to the db fine. It's just App Engine that refuses to do so, so I can only assume the issue is stemming from some App Engine configuration. - Tunji_D
Did you follow along with the docs at cloud.google.com/nodejs/getting-started/deploy-mongodb ? - BrettJ
@BrettJ that just might be it! I do have a config json file, but mine is called serverConfig.json, and the key mappings are different. When following the tutorial I thought it was a best practice thing so I changed the keys and file name to better describe my setup. I didn't know App Engine was relying on the file for internal setup as well. I'll edit my config file, to match the tutorial exactly and try again. - Tunji_D
@BrettJ Quick question, the config json in the example specifies the MONGO_COLLECTION. My app uses several. Do I pass in a json array of all collections? Can I ignore this field? - Tunji_D

3 Answers

6
votes

As it's been pointed out, you can follow the instruction on https://cloud.google.com/nodejs/getting-started/deploy-mongodb to configure Google App Engine using config.json file. (update: link no longer works, try: https://cloud.google.com/community/tutorials/nodejs-mongodb-on-appengine instead)

Since you're using mongoose the models will dictate the collection usage. i.e. save(), update(). If you're connecting using node.js directly, you can also change database/collection easily in the code.

What differs between my App Engine instance and my local computer when trying to connect to the above URI?

Google App Engine is a fully managed platform. It is in a sense serverless, you are utilising a pool of workers to up/down scale your application. When a request comes in to your application, an instance from the pool will handle the task/request.

Unlike your local machine, where request will be handled by your machine as well.

You may also find Google App Engine FAQ useful.

Does the fact that I chose for mlab to host it on GCP matter?

Not a lot. As you mentioned, you're currently on the SandBox (free) plan with GCP. At the moment this would deploy your MongoDB instance on us-central1 region, and there's no other choice of region. There are also slight variation of RAM/Storage/Price between different cloud provider.

You may be able to reduce network latency by choosing GAE on us-central1 region as well.

What exactly is the root cause of the issue?

It's difficult to know for certain what is the issue without proper stack-trace or error log describing the timeout issue.

Commonly node.js project contain a JavaScript file called config.js which has a line below:

  nconf.argv()
       .env()
       .file({ file: path.join(__dirname, 'config.json') })
       .defaults()

Which basically trying to read configuration in the order of :

  • Command line argument
  • Environment variables
  • Config file (specified path)
  • Defaults values

See also Google Cloud Platform: nodejs-getting-started

Also GoogleCloudPlatform/nodejs-docs-samples/appengine/mongodb project for an example.

3
votes

Well this took me entirely too long, but the issue was with my Dockerfile.

I was extending the node.js runtime but my Dockerfile was missing lines to copy the app directory and install npm.

To create the appropriate Dockerfile I used the following command:

gcloud beta app gen-config --custom

More information about the command and extending the node.js runtime can be found here.

My final Dockerfile after I added the dependencies I needed looks like this:

NOTE: I can only connect to my MongoDb instance after deployment. Running locally in Cloud Shell still times out.

# Dockerfile extending the generic Node image with application files for a
# single application.
FROM gcr.io/google_appengine/nodejs
COPY . /app/

RUN apt-get -y update && apt-get install -y libav-tools && apt-get install -y graphicsmagick

# You have to specify "--unsafe-perm" with npm install
# when running as root.  Failing to do this can cause
# install to appear to succeed even if a preinstall
# script fails, and may have other adverse consequences
# as well.
# This command will also cat the npm-debug.log file after the
# build, if it exists.
RUN npm install --unsafe-perm || \
  ((if [ -f npm-debug.log ]; then \
      cat npm-debug.log; \
    fi) && false)
CMD npm start
0
votes

I have had the same problem for a while but eventually this worked for me.Hope it helps.Ehen I read the the db connection details from a json file using nconf package as described by google https://cloud.google.com/community/tutorials/nodejs-mongodb-on-appengine with no mongoose I was able to solve it.Initially i was reading from env variables.

`         nconf = require('nconf');
          const mongoose = require('mongoose');
          //key.json contains mongodb connection information
          nconf.argv().env().file('keys.json');
          const user = nconf.get('mongoUser');
          const pass = nconf.get('mongoPass');
          const host = nconf.get('mongoHost');
          const port = nconf.get('mongoPort')
          const mongoDatabase=nconf.get('mongoDatabase');
          let mongo_url=`mongodb://${user}:${pass}@${host}:${port}/${mongoDatabase}`;
          mongoose.connect(mongo_url,{useNewUrlParser: true});`

You dont have to use config.json as suggested by other answers.