Spatial data and memory

Question

I am trying to add up geotiffs but am running into memory issues. R is using all 32GB according the the following R error...

In writeValues(y, x, start = 1) :
  Reached total allocation of 32710Mb: see help(memory.size)

I also checked the properties of R and it is 64 bit and the target is...

"C:\Program Files\R\R-3.3.0\bin\x64\Rgui.exe"

The version is

R.Version() 
$platform
[1] "x86_64-w64-mingw32"

$arch
[1] "x86_64"

$os
[1] "mingw32"

$system
[1] "x86_64, mingw32"

$status
[1] ""

$major
[1] "3"

$minor
[1] "3.0"

$year
[1] "2016"

$month
[1] "05"

$day
[1] "03"

$`svn rev`
[1] "70573"

$language
[1] "R"

$version.string
[1] "R version 3.3.0 (2016-05-03)"

$nickname
[1] "Supposedly Educational"

So it looks like my max memory is being used by R. I tried the to use bigmemory package in R. So in the code below I tried changing the matrix to big.matrix but that failed and the error occurs when trying to write the output file. Any suggestions for trying to alter the code so less memory is used or try to work in the package ff or bigmemory?

############  LOOP THROUGH AGE MAPS TO COMPILE THE NUMBER OF TIMES A CELL BURNS DURING A GIVEN SPAN OF TIME  ####################
## Empirical Fires
print("1 of 3: 2010-2015")

burn.mat<- matrix(0,nrow,ncol) #create matrix of all zero's, the dimension of your landscape  (row, col)

# Read in Historical Fire maps
for (j in 2010:2015){ #Year Loop

  age.tmp<- as.matrix(raster(paste('fr',j,'.tif',sep=''))) #read in Age Map

  burn.mat<- burn.mat+(age.tmp==1) #when something has burned in ALFRESCO empirical fire history files, AGE=1. (age.tmp==0) is a 'logic' cmd, returning a 0,1 map for True/False

  #Write the data to a geotiff
  out <- raster(burn.mat,xmn=-1692148,xmx= 1321752, ymn = 490809.9, ymx = 2245610, crs = '+proj=aea +lat_1=55 +lat_2=65 +lat_0=50 +lon_0=-154 +x_0=0 +y_0=0 +ellps=GRS80 +datum=NAD83 +units=m +no_defs')
  writeRaster(out,filename=paste(outdir,'/burn.mat.hist.1950-2007.tif',sep=''),format = 'GTiff',options='COMPRESS=LZW',datatype='FLT4S',overwrite=T)
}

Why is the code writing to the same filename each time through the loop? — mdsumner

Robert Hijmans Robert Hijmans · Accepted Answer · 2016-08-24T06:05:23

The problem will probably go away if you use Raster* objects rather than matrices. Something like

library(raster)
r <- raster('fr2010.tif')

burn.mat <- setValues(r, 0)

for (j in 2010:2015) {
  age.tmp <- raster(paste0('fr', j, '.tif'))
  burn.mat <- burn.mat + (age.tmp==1) 
 # if age.tmp only has values of 0 and 1 use this instead:  
 # burn.mat <- burn.mat + age.tmp 
}

# write the results outside of the loop
writeRaster(burn.mat, filename=file.path(outdir, 'burn.mat.hist.1950-2007.tif'), options='COMPRESS=LZW',datatype='FLT4S',overwrite=TRUE)

A more direct approach without a loop

 files <- paste0('fr', 2010:2015, '.tif'))
 s <- stack(files)
 burn <- sum(s)

Or

 burn <- sum(s == 1)

Or to write to a file in one step

 b <- calc(s, sum, filename=file.path(outdir, 'burn.mat.hist.1950-2007.tif'), options='COMPRESS=LZW', datatype='FLT4S', overwrite=TRUE)

Spatial data and memory

1 Answers