Speed up r loop string match (vector vs. data.frame)

Question

I'm trying to optimize a loop in r that counts the number of string matches of each element in a vector regarding each row in a data frame. In small datasets it works pretty good (~15 min; 11 columns, 914 rows). However, it takes days for running in huge datasets (914 columns, 18.000 rows). Here's my extremely basic loop:

for (j in 1: dim(pddbnh)[1]){
  for (i in 1:dim(pidf)[1]){
    richa[i,j] <- length(pidf[i,][pidf[i,] == row.names(pddbnh)[j] ])
   }
}

I'm wondering if anyone knows how to optimize this loop using other approach (e.g. vectorization). Any solution would be much appreciated!

UPDATE Here's a small dataset. That's the fastest one

 df<-data.frame(replicate(10,sample(c("sp1", "sp2"),10,rep=TRUE)))
 vec<-c("sp1", "sp2")
 richa <- data.frame()

  for (j in 1:length(vec)){
    for (i in 1:dim(df)[1]){
     richa[i,j] <- length(df[i,][df[i,] == vec[j] ])
     }
    }

Yes, I probably know how to do this. Please provide a minimal reproducible example for testing. — Roland
Thank you! Here are the two files: dropbox.com/s/vvksp7c1kerqjbq/pddbnh.csv?dl=0 dropbox.com/s/zjt9shku0gjf03t/pidf.csv?dl=0 — CRP
(Obviously you have not yet done a thorough search of SO, since there are many Q&A's about improving performance of for-loops.) Asking for [ to process [ pidf[i,] == row.names(pddbnh)[j] ] would seem doomed to failure. The result of that argument will be either 0 or 1 after coercion (so errors regarding replacement having length zero should appear). Maybe you should explain what it is that you are actually attempting. — IRTFM
Actually, it works better than other functions (e.g. grep). I'm just trying to count the number of times a string (row.names(pddbnh)[j]) is found in each line from a dataframe (pidf[i,]). — CRP
Post a small example (... dropbox is not a friendly way to share data anymore). Maybe the example will demonstrate why a logical index wouldn't generate errors if it returned FALSE's. — IRTFM

rosscova rosscova · Accepted Answer · 2016-11-29T09:12:38

Here's a method using lapply (see below for even faster):

richa <- lapply( X = vec, FUN = function(x) rowSums( df == x ) )
richa <- do.call( cbind, richa )

A quick microbenchmark on the small dataset you've provided shows this at about 10x faster than your for loop method.

Just to add, this could easily be multi-threaded as well for really big datasets, using either parallel::mclapply or plyr::laply (with parallel = TRUE). It takes a little extra work, but might be worthwhile for those 18000 x 914 datasets you've got.

EDIT TO ADD: since you've got a few for loops going there (and since I'm learning Rcpp, and keen to practice) here's an even faster solution using Rcpp. Here's the function definition (which needs to be compiled once):

Rcpp::cppFunction(' IntegerMatrix charCrossCheck( CharacterMatrix df,
                          CharacterVector vec ) {

              IntegerMatrix output( df.nrow(), vec.size() );

              for (int j=0; j < vec.size(); ++j ){
                  for (int i=0; i < df.nrow(); ++i ){
                      int count = 0;
                      for( int k=0; k < df.ncol(); k++ ){
                          if( df(i,k) == vec[j] ) {
                              count++;
                          }
                      }
                      output(i,j) = count;
                  }
              }
              return output;

              } ')

Then you can call that function with:

richa <- charCrossCheck( as.matrix(df), vec )

Rcpp is very fast here. Microbenchmark on your very small sample shows it more than 3x faster than my lapply solution above, and about 38x faster than your for loops in R.

Interestingly, expanding the input data out to a df size 4000x4000, and a vec of length 10, both the Rcpp and lapply methods complete the job in very similar times (3.4 and 3.9 seconds respectively). On a dataset the size you mention (18000 rows x 914 columns, with vec length 2), both solutions are well under 1 second. Not bad either way!

Speed up r loop string match (vector vs. data.frame)

1 Answers