Getting values from a loop of asynchronous requests

Question

I am trying to write a nodejs program that queries github for a list of repos (via a Node wrapper for the github API: https://www.npmjs.com/package/github) and retrieves the git clone url in an array, which I then wish to sort alphabetically.

Due to the asynchronous nature of the calls, I am not sure how to wait until all of the async requests are returned?

Here is the loop in question. repoArrayis an array of repos in [username/reponame] format

var urls = [];

for (var i=0; i < repoArray.length; i++) {

    var components = repoArray[i].split('/');

    github.repos.get({
        user: components[0],
        repo: components[1]
    }, function(err, res) {
        urls.push(res.ssh_url);
    });
}

// do a case-insensitive sort
urls.sort(function(a,b) {
    return a.localeCompare(b, 'en', {'sensitivity': 'base'});
});

console.log("urls: " + urls);

Basically, since github.repos.get() calls in the loop are all asynchronous/callback-based, when the code reaches urls.sort() and then the console.log(), none or some of the github.repos.get() calls are done yet.

I am not that familiar with promises or deferreds, but is that the way to go? I'm not sure how I could refactor that loop so that urls.sort() is called only after all the requests from the loop are complete?

Drew Harwell Drew Harwell · Accepted Answer · 2015-07-31T23:42:54

The Async library is meant for exactly these scenarios, and is usually what people tend to use for these problems. It can help you execute asynchronous tasks in parallel and execute a callback when they all finish, using async.each.

var async = require('async');
var urls = [];

//make each HTTP request
function process(repo,callback){    
  var components = repo.split('/');

  github.repos.get({
    user: components[0],
    repo: components[1]
  }, function(err, res) {
    if(err){
      // call callback(err) if there is an error
      return callback(err);
    }
    else{
      urls.push(res.ssh_url);
      // call callback(null) if it was a success,
      return callback(null);
    }
  });

}

// this will iterate over repoArray and pass each repo to the 'process' function.  
// if any of the calls to 'process' result in an error, 
// the final callback will be immediately called with an error object
async.each(repoArray,process,function(error){
  if(error){
    console.error('uh-oh: '+error)
    return;
  }
  else{
    // do a case-insensitive sort
    urls.sort(function(a,b) {
      return a.localeCompare(b, 'en', {'sensitivity': 'base'});
    });
    console.log("urls: " + urls);
  }
});

edit: since you are sorting them at the end, the urls will be in order.

Getting values from a loop of asynchronous requests

1 Answers