How to automatically zip files with Node.js and npm

Question

Is there a way to automatically zip certain files at the build time with Node.js and npm?

For example, I have a project, that file structure looks like this:

Project/
--lib/
--node_modules/
--test/
--index.js
--package.json

I want to be able to zip lib folder, certain modules from node_modules and index.js into some zip archive to upload it on the AWS Lambda, for example. I do not need test folder or test Node.js modules (mocha and chai) to be zipped. I have even created a bash script for generating zip file, but is there a way to automatically execute this script, when 'npm install' is called?

This should be a standard problem and it should have a standard solution, but I was unable to discover such.

UPDATE

thanks to michael, decided to use gulp. This is my script, in case some one else will need it for AWS Lambda:

var gulp = require('gulp');
var clean = require('gulp-clean');
var zip = require('gulp-zip');
var merge = require('merge-stream');

gulp.task('clean', function () {
    var build = gulp.src('build', {read: false})
        .pipe(clean());
    var dist = gulp.src('dist', {read: false})
        .pipe(clean());

    return merge(build, dist);
});

gulp.task('build', function() {
    var index = gulp.src('index.js')
        .pipe(gulp.dest('build'));
    var lib = gulp.src('lib/**')
        .pipe(gulp.dest('build/lib'));
    var async = gulp.src('node_modules/async/**')
        .pipe(gulp.dest('build/node_modules/async'));
    var collections = gulp.src('node_modules/collections/**')
        .pipe(gulp.dest('build/node_modules/collections'));
    var underscore = gulp.src('node_modules/underscore/**')
        .pipe(gulp.dest('build/node_modules/underscore'));
    var util = gulp.src('node_modules/util/**')
        .pipe(gulp.dest('build/node_modules/util'));
    var xml2js = gulp.src('node_modules/xml2js/**')
        .pipe(gulp.dest('build/node_modules/xml2js'));

    return merge(index, lib, async, collections, underscore, util, xml2js);
});

gulp.task('zip', ['build'], function() {
    return gulp.src('build/*')
        .pipe(zip('archive.zip'))
        .pipe(gulp.dest('dist'));
});

gulp.task('default', ['zip']);

Answer 1

I realize this answer comes years too late for the original poster. But I had virtually the same question about packaging up a Lambda function, so for posterity, here's a solution that doesn't require any additional devDependencies (like gulp or grunt) and just uses npm pack along with the following package.json (but does assume you have sed and zip available to you):

{
  "name": "my-lambda",
  "version": "1.0.0",
  "scripts": {
    "postpack": "tarball=$(npm list --depth 0 | sed 's/@/-/g; s/ .*/.tgz/g; 1q;'); tar -tf $tarball | sed 's/^package\\///' | zip -@r package; rm $tarball"
  },
  "files": [
    "/index.js", 
    "/lib"
  ],
  "dependencies": {
    "async": "*",
    "collections": "*",
    "underscore": "*",
    "util": "*",
    "xml2js": "*"
  },
  "bundledDependencies": [
    "async",
    "collections",
    "underscore",
    "util",
    "xml2js"
  ],
  "devDependencies": {
    "chai": "*",
    "mocha": "*"
  }
}

Given the above package.json, calling npm pack will produce a package.zip file that contains:

index.js
lib/
node_modules/
├── async/
├── collections/
├── underscore/
├── util/
└── xml2js/

The files array is a whitelist of what to include. Here, it's just index.js and the lib directory.

However, npm will also automatically include package.json, README (and variants like README.md, CHANGELOG (and its variants), and LICENSE (and the alternative spelling LICENCE) unless you explicitly exclude them (e.g. with .npmignore).

The bundledDependencies array specifies what packages to bundle. In this case, it's all the dependencies but none of the devDependencies.

Finally, the postpack script is run after npm pack because npm pack generates a tarball, but we need to generate a zip for AWS Lambda.

A more detailed explanation of what the postpack script is doing is available at https://hackernoon.com/package-lambda-functions-the-easy-way-with-npm-e38fc14613ba (and is also the source of the general approach).

Answer 2

I would go with gulp using gulp-sftp, gulp-tar and gulp-gzip and an alias as command. Create a file called .bash_aliases in your users home folder containing

alias installAndUpload='npm install && gulp runUploader'

After a reboot you can call both actions at once with this alias.

A gulp file could look something like this

var gulp = require('gulp');
var watch = require('gulp-watch');
var sftp = require('gulp-sftp');
var gzip = require('gulp-gzip');

gulp.task('runUploader', function () {
    gulp.src('.path/to/folder/to/compress/**')
        .pipe(tar('archive.tar'))
        .pipe(gzip())
        .pipe(gulp.dest('path/to/folder/to/store')) // if you want a local copy
        .pipe(sftp({
            host: 'website.com',
            user: 'johndoe',
            pass: '1234'
        }))
});

Of course, you can also add gulp-watch to automatically create the tar/zip and upload it whenever there is a change in the directory.

Answer 3

You should take a look to npm scripts.

You'll still need a bash script laying around in your repository, but it will be automatically triggered by some npm tasks when they are executed.

Answer 4

npm-pack-zip worked for me.

npm install --save-dev npm-pack-zip

To publish the whole lambda using aws I used this node script in package.json:

    "publish": "npm-pack-zip && aws lambda update-function-code --function-name %npm_package_name% --zip-file fileb://%npm_package_name%.zip && rm %npm_package_name%.zip"

Answer 5

If you need automate tasks take a look to Grunt or Gulp.

In the case of Grunt needed plugins:

https://www.npmjs.com/package/grunt-zip

https://www.npmjs.com/package/grunt-aws-lambda

Answer 6

You can use Zip-Build, this little package will use the data in your package.json file and create a compressed file named project-name_version.zip.

Disclaimer: I am a developer of this library.

How to use zip-build

Just install in your project as dev dependency with:

$ npm install --save-dev zib-build

Then modify the build script in your package.json, adding && zip-build at the end, like this:

"scripts": {
  "build": your-build-script && zip-build
}

If your build directory is named different than build and your desired directory for compressed files is named different than dist, you can provide the directory names as arguments for zip-build:

"scripts": {
  "build": your-build-script && zip-build build-dirname zip-dirname 
}

Answer 7

Check out my gist at https://gist.github.com/ctulek/6f16352ebdfc166ce905

This uses gulp for all the tasks you mentioned except creating the lambda function initially (it only updates the code)

It assumes every lambda function is implemented in its own folder, and you need to define your AWS credential profile.

Answer 8

If you're UNIX-based you could also just use the zip command in one of your scripts:

"scripts": {
  "zip": "zip -r build.zip build/"
  "build": "build",
  "build-n-zip": "build && zip
}

The above creates a build.zip at the root, which is a zipped up version of the /build folder.

If you wanted to zip multiple folders/files, just add them to the end:

"scripts": {
  "zip": "zip -r build.zip build/ some-file.js some-other-folder/"
}