node exec() not firing all data events

Question

It seems for big enough stdout output node does not fire data events:

Example bash script:

#!/usr/bin/env sh

for i in {1..100000}; do
    echo $i
done

Running the above script from bash should prints all numbers from 1 to 100k one number per line

However when proxying it via node's exec:

#!/usr/bin/env node --harmony --harmony_destructuring --harmony_default_parameters

const {exec} = require('child_process');

let stdout = '';

const child = exec('./tmp.sh');

child.stdout.on('data', data => {
    stdout += data;
});

child.stdout.on('close', code => {
    if (code) process.exit(code);
    console.log(stdout.split('\n').slice(0, 3).join('\n'));
    console.log('...');
    console.log(stdout.split('\n').slice(-3).join('\n'));
});

For the above script, I'd expect to get:

However when run It returns:

About a third of the output is truncated. Logging inside the data handler shows that only 1 data event is fired with 1 chunk

I've tried listening to the child.on('exit') event instead with the same result (also stdout.on('exit'))

It works the same when replacing exec with spawn

node --version # v5.4.1
osx 10.10

jfriend00 jfriend00 · Accepted Answer · 2016-05-02T20:59:31

child_process.exec() buffers the standard output into a fixed size buffer. You are likely filling the buffer and thus you don't get the rest of the output.

There is a maxBuffer option you can pass to .exec() to make the buffer larger or you can use .spawn() and receive an unlimited stream of output.

See Stdout buffer issue using node child_process for related info.

Also worth reading: Difference between spawn and exec of Node.js child_process and Stdout of Node.js child_process exec is cut short.

node exec() not firing all data events

1 Answers