Programming using the tidyverse: speed issues

4
votes

We released the package quickpsy a few years ago (paper in the R journal paper). The package used R base functions, but also made an extensive use of functions of what was called at that time the Hadleyverse. We are now developing a new version of the package that mostly uses functions from the tidyverse and that incorporates the new non-standard evaluation approach and found that the package is much much slower (more than four times slower). We found for example that purrr::map is much slower than dplyr::do (which is deprecated):

library(tidyverse)

system.time(
  mtcars %>% 
    group_by(cyl) %>% 
    do(head(., 2))
  )

system.time(
  mtcars %>% 
    group_by(cyl) %>% 
    nest() %>% 
    mutate(temp = map(data, ~head(., 2))) %>% 
    unnest(temp)
)

We also found that functions like pull are very slow.

We are not sure whether the tidyverse is not meant to be used for this type of programming or we are not using it properly.

2
rtidyversepurrrnse
if you care about speed data.table is going to be your friend. - sindri_baldur
Not to mention concision: as.data.table(mtcars)[, .SD[1:2], by = cyl] - sindri_baldur
There may be some people here who can help (though as it stands, this is fairly broad). One place you might ask about speed differences is in the Rstudio community tidyverse site. - lmo
According to Hadley: We optimise dplyr for expressiveness on medium data; feel free to use data.table for raw speed on bigger data. With data.table you can: data.table(mtcars)[, .SD[1:2], cyl] - pogibas
It is likely not map() that is slow, but nest and unnest. Perhaps because of selection? Tidyselect is written in pure R. Your mtcars benchmark does not demonstrate that the functions are slow though, just that they are slower. If the slowness increases with data frame size, then there might be a problem. - Lionel Henry

2 Answers

3
votes

slice() is the proper tool to use if you want the first two rows of each group. Both do() and nest() %>% mutate(map()) %>% unnest() are too heavy and use more memory: 

library(dplyr, warn.conflicts = FALSE)
library(tidyr)
library(purrr)

library(tidyverse)

system.time(
  mtcars %>% 
    group_by(cyl) %>% 
    do(head(., 2))
)
#>    user  system elapsed 
#>   0.065   0.003   0.075

system.time(
  mtcars %>% 
    group_by(cyl) %>% 
    nest() %>% 
    mutate(temp = map(data, ~head(., 2))) %>% 
    unnest(temp)
)
#>    user  system elapsed 
#>   0.024   0.000   0.024

system.time(
  mtcars %>% 
    group_by(cyl) %>% 
    slice(1:2)
)
#>    user  system elapsed 
#>   0.002   0.000   0.002

Created on 2018-10-23 by the reprex package (v0.2.1.9000)

See also benchmark results in this tidyr issue

0
votes

For this particular example, the slowness caused by the nest and unnest computations can be solved using group_modify 

system.time(
   mtcars %>% 
   group_by(cyl) %>% 
   group_modify(~head(., 2))
)