jsforce bulk query api 10k limit

Question

I am trying to understand how I can use jsforce & bulk query to export 50k records from salesforce. Only the first 10k is returned, I understand that this is due to a batch size limit of 10k however I don't understand how to create the next batch to get records 10001 though 20000 and so on.

Currently I have the follow, any help would be much appreciated.

  conn.bulk.query('SELECT Id FROM Account')
    .on('record', function record(rec) {
      log.debug('dumpAllObject', 'rec', rec);
    })
    .on('error', function handle(err) { log.error('dumpAllObject', 'error', err); })
    .on('end', function resolve() {
          log.info('dumpAllObject', 'Completed');
    });

d512 d512 · Accepted Answer · 2018-06-05T21:06:10

Try using the authFetch and maxFetch options:

conn.query('SELECT Id FROM Account')
    .on('record', function record(rec) {
      log.debug('dumpAllObject', 'rec', rec);
    })
    .on('error', function handle(err) { log.error('dumpAllObject', 'error', err); })
    .on('end', function resolve() {
          log.info('dumpAllObject', 'Completed');
    })
    .run({
        authFetch: true,
        maxFetch: 20000
    });

This works for me on queries only (not inserts, updates, or deletes).

Also not sure what the difference between conn.query and conn.bulk.query is.

jsforce bulk query api 10k limit

1 Answers