increasing memory in R

Question

I'm working with a large data set (41,000 observations and 22 predictor variables) and trying to fit a Random Forest model using this code:

model <- randomForest(as.factor(data$usvsa) ~ ., ntree=1000, importance=TRUE, + proximity=TRUE, data=data).

I am running into the following error:

Error: cannot allocate vector of size 12.7 Gb
In addition: Warning messages:
1: In matrix(0, n, n) :
  Reached total allocation of 6019Mb: see help(memory.size)
2: In matrix(0, n, n) :
  Reached total allocation of 6019Mb: see help(memory.size)
3: In matrix(0, n, n) :
  Reached total allocation of 6019Mb: see help(memory.size)
4: In matrix(0, n, n) :
  Reached total allocation of 6019Mb: see help(memory.size)

I have done some reading in the R help on memory limits and on this site and am thinking that I need to buy 12+ GB of RAM since my memoryLimit is already set to about 6GB of RAM (my computer only has 6 GB of RAM). But first I wanted to double check that this is the only solution. I am running a windows 7 with a 64 bit processor and 6GB of RAM. Here is the R sessionInfo:

sessionInfo()
R version 2.15.3 (2013-03-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] randomForest_4.6-7

loaded via a namespace (and not attached):
[1] tools_2.15.3

Any tips?

since n is pretty large, if you cross-validate intelligently you maybe able to just train and test on many manageable chunks of the data without the memory over-run. — Stephen Henderson
The issue in this case will almost certainly be the 41000 x 41000 proximity matrix that you asked for, not the forest itself. You should be able to run randomForest on data that size with relatively little trouble, I would think. But the utility of the proximity matrix might require it being built on the whole data. So, do you really need the proximities...? — joran
Thanks! I think you're right, when I take out proximity it works! — user3088823

Dirk Eddelbuettel Dirk Eddelbuettel · Accepted Answer · 2013-12-10T23:05:12

Quoting from the wonderful paper "Big Data: New Tricks for Econometrics" by Hal Varian:

If the extracted data is still inconveniently large, it is often possible to select a subsample for statistical analysis. At Google, for example, I have found that random samples on the order of 0.1 percent work for analysis of economic data.

So how about if you don't use all 41k rows and 22 predictors?

increasing memory in R

3 Answers