Is it possible to run npm package installation locally from a downloaded bundle?

Question

I am quite new to NodeJS, so please excuse me if my question is naive. I find it fascinating to run Javascript on the server.

Yet there is one thing that severely dampens my experience. It is the lengthy error prone installation process for every new package resulting in what seems to be never ending series of HTTP requests. I can even see multiple requests of the same resources again and again, which could have been cached instead I suppose.

What makes it even worse, I get errors often after many other requests completed and time wasted, after which the installation fails for no clear reason.

My question is:

Is it possible to download a stable bundle of most used packages first to avoid them being requested over HTTP?
And then run the installation locally, using most packages from the bundle, and only requesting over HTTP what is really new and important?

The advantages would be:

Minimising the number of requests, making the whole process much faster. I feel it is the number of requests that make it taking so long, so the performance would increase dramatically. Especially if you are on slow connection.
Using only stable and tested packages instead of very last ones that can be responsible for errors and leading to installation failures. I am guessing these are the very last unstable packages that cause errors leading to installation failures.

Being new to Node, my question can be naive, and I may be missing something fundamental. In which case I would like to learn it and understand the reasons behind.

Just to give some "meat", here is my recent (failed) attempt to install Yeoman:

$ sudo npm install -g yo
Password:
npm http GET https://registry.npmjs.org/yo
npm http 304 https://registry.npmjs.org/yo
npm http GET https://registry.npmjs.org/yeoman-generator
npm http GET https://registry.npmjs.org/nopt
npm http GET https://registry.npmjs.org/update-notifier
npm http GET https://registry.npmjs.org/insight
npm http GET https://registry.npmjs.org/sudo-block
npm http GET https://registry.npmjs.org/open/0.0.4
npm http GET https://registry.npmjs.org/chalk
npm http GET https://registry.npmjs.org/findup
npm http GET https://registry.npmjs.org/lodash
npm http GET https://registry.npmjs.org/async
npm http 200 https://registry.npmjs.org/update-notifier
npm http GET https://registry.npmjs.org/update-notifier/-/update-notifier-0.1.7.tgz
npm http 200 https://registry.npmjs.org/open/0.0.4
npm http GET https://registry.npmjs.org/open/-/open-0.0.4.tgz
npm http 200 https://registry.npmjs.org/open/-/open-0.0.4.tgz
npm http 200 https://registry.npmjs.org/update-notifier/-/update-notifier-0.1.7.tgz
...

Note repeated request of the last resource! Then after pages and pages the errors come:

npm http 200 https://registry.npmjs.org/rimraf/-/rimraf-2.2.5.tgz
npm ERR! Error: shasum check failed for /Users/dmitrizaitsev/tmp/npm-2533/1387980648215-0.5165209104306996/tmp.tgz
npm ERR! Expected: a4663b53686b895ff074e2ba504dfb76a8e2b770
npm ERR! Actual:   8d2a8a2b726937c7cc6a86207b56e3a38d853f1e
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/sha/index.js:32:8
npm ERR!     at ReadStream. (/usr/local/lib/node_modules/npm/node_modules/sha/index.js:61:7)
npm ERR!     at ReadStream.EventEmitter.emit (events.js:117:20)
npm ERR!     at _stream_readable.js:883:14
npm ERR!     at process._tickCallback (node.js:415:13)
npm ERR! If you need help, you may report this log at:
npm ERR!     
npm ERR! or email it to:
npm ERR!     

npm ERR! System Darwin 12.5.0
npm ERR! command "node" "/usr/local/bin/npm" "install" "-g" "yo"
npm ERR! cwd /Users/dmitrizaitsev/Dropbox/Priv/APP
npm ERR! node -v v0.10.3
npm ERR! npm -v 1.2.21
npm http 200 https://registry.npmjs.org/ansi-styles
npm http GET https://registry.npmjs.org/ansi-styles/-/ansi-styles-0.2.0.tgz

...

npm http 200 https://registry.npmjs.org/cheerio
npm ERR! registry error parsing json
npm http 200 https://registry.npmjs.org/form-data/-/form-data-0.1.2.tgz

...

And here the failure (again after pages and pages):

npm http 200 https://registry.npmjs.org/hoek/-/hoek-0.9.1.tgz
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /Users/dmitrizaitsev/Dropbox/Priv/APP/npm-debug.log
npm ERR! not ok code 0

Benjamin Kaiser Benjamin Kaiser · Accepted Answer · 2013-12-26T04:11:22

First of all, to address your two questions,

There doesn't seem to be a bundle of the "most used packages", this is probably because they see it as easier to just download the ones you need.
Yes that seems like more of an addition to the first question. Wouldn't you be pulling the 'bundle' of packages over http(s) anyway?

npm should cache packages by default, see the config example here (my ~/.npm has cached packages in it). I am assuming you are using a package.json file for your packages?

If you really are keen to have some version of a bunch of packages on your own machine, perhaps take a look at npm-mirror?

By the looks of it the problem with your rimraf download seems to be that it is failing/being corrupted (the sha1 sums don't match). I can download rimraf successfully on my machine with no problems.

Also if you use git to push to your server and are worried about your server reinstalling packages every time you push it, what I have in one of my .git hooks (post-receive) is to cache the npm packages, then checkout my repo (which does not keep the npm packages), then place them back in again before running npm install, so that they don't have to re-download.

Is it possible to run npm package installation locally from a downloaded bundle?

1 Answers