Fastest way to import SPSS data into R as dataframe

Question

I need to import an SPSS .sav file into R every day as a data frame without value labels. The file is 120,000+ obs and growing. This process is getting incredibly slow, so I want to make sure I'm using the fastest possible method. I've been playing around with the functions in foreign, haven, and memisc. I'm working with RDS if that makes a difference.

Edit: My file is 126343 x 33067 and 12.1 GB.I'm just simply running the following code:

library(haven)
data <- read_sav(file)

I can't share this file, but to attempt to replicate, I did:

library(haven)
n <- 126343
exd <- data.frame(c(replicate(2000, sample(letters, n, replace = TRUE),
                              simplify = FALSE),
                    replicate(1306, runif(n),
                              simplify = FALSE)))
dim(exd)
## [1] 126343    3306
tmp <- tempfile(fileext = ".sav")
write_sav(exd, tmp)
system.time(exd2 <- read_sav(tmp))
##   user  system elapsed 
##  173.34   13.94   187.66

Thanks!

I ended up saving my files as csvs and using the data.table fread function: — Jklein

Ista Ista · Accepted Answer · 2018-05-30T18:10:21

120000 isn't very big. Unless you have a very low resource system I wouldn't expect this to be much of a bottleneck at all. On my mid-range laptop it takes just a few seconds to read a 122000 X 150 .sav file:

library(haven)
n <- 122000
exd <- data.frame(c(replicate(50, sample(letters, n, replace = TRUE),
                              simplify = FALSE),
                    replicate(100, runif(n),
                              simplify = FALSE)))
dim(exd)
## [1] 122000    150
tmp <- tempfile(fileext = ".sav")
write_sav(exd, tmp)
system.time(exd2 <- read_sav(tmp))
##   user  system elapsed 
##  1.913   0.096   2.015

Since I can't reproduce the problem as you've described it you should provide more details to make it clearer what the issue is. If you show the code and (a subset or simulation of) the data you're working with you might get some help identifying the likely bottleneck.

Fastest way to import SPSS data into R as dataframe

2 Answers