Sending tasks to Azure Batch Jobs with Node SDK

Question

I'm trying to send a large number of tasks to Azure Batch. There are three jobs and a total of 67,000 tasks spread across those jobs.

When I send only a few tasks everything works perfectly, as soon as I open it up to send the full list it seems to fall over, but without any useful error message. Is there some kind of limit on the total? Or the rate at which I can send them? I'm using Azure Functions, Node SDK and promises to add tasks asynchronously.

                async.each(tasks, function(task, callback) {
                    var taskID = task.id + '_process';

                    var config = {
                        id: taskID,
                        displayName: 'Render portion ' + task.id + ' in job ' + task.job.id,
                        commandLine: "python3 main.py '" + JSON.stringify(task) + "'"
                    };

                    batchClient.task.add(task.job.id, config).then(function(result) {
                        context.log('Task for portion : ' + task.id + ' submitted successfully');
                        callback();
                    }).catch(function(error){
                        context.log('Task failed to be added');
                        callbacK(error);
                    });

                }, function(error) {
                    if( error ) {
                        context.log('Error adding task.');
                        context.res = { body: error };
                        context.done();
                    } else {
                        context.log('All tasks have been queued successfully');
                        context.res = { body: 'All tasks have been queued successfully' };
                        context.done();
                    }
                });

At this point, I'd be thrilled to even get an error, if only to give me something to go on. I've tried sending 10 tasks and it works fine, send 65,000 and it fails silently.

Hiya Mark, I reckon couple of things will be key to understand this further: What is the work load looks like? (i.e. how many pools in play, what is the strategy the design has?) Also can you please flick the exact user error you are getting please? :) thanks! — Tats_innit
The workload is the 65,000 tasks spread across three jobs on a single pool. The pool has 128 NC6 machines ~780 cores in total. I get no error messages at all out of the script, only a timeout after 30 minutes of run time. — Mark

Hydraxy Hydraxy · Accepted Answer · 2017-11-10T01:56:27

I suggest that you use the Add task collection API instead of the Add task API, as the collection API can take up to 100 tasks in a single API call. That should allow you to reduce the total number of API calls being made by a factor of 100.

If you use that API and are still having trouble, it's possible that there's some issue in the SDK with really high levels of parallelism... it might be worth trying to remove the call to Batch and confirm that there's no issue with the code you have as is (just replace it with a simple http get on a static URL maybe?)

Sending tasks to Azure Batch Jobs with Node SDK

1 Answers