forrtl: severe (179): Cannot allocate array - overflow on array size calculation

2
votes

I am trying to run a simulation program on my laptop (Linux 3.8.0-25-generic x86_64, with Ubuntu 13.04).

It compiles okay, but when getting to allocation of some array sizes I get:

forrtl: severe (179): Cannot allocate array - overflow on array size calculation.\

Some googling about this message led me to conclude that this is generated because my program is out of memory.

To check this assumption I tried to allocate the same array a smaller dimension, but still got the same problem.

What I tried next was to increase my stack memory, but still got the same problem.

This is the code:

program memoria

implicit none

integer :: n,num
complex, dimension(:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:), allocatable :: ddptrj

n=4

num=21


allocate(ddptrj(n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,num))

deallocate(ddptrj)


endprogram memoria

How can I solve this problem?

2
memory-managementfortranintel-fortran
My guess is you are doing something physically/mathematically wrong if you are trying to use over 1 petabyte of memory for your work. - Kyle Kanos
I would like to point out that it's really a bad idea to use 20+ dimensions for a Fortran array, even if your compiler allows it. The code is going to be unreadable. You really need to use lower dimensional arrays and implement your own indexing scheme. - Sean Patrick Santos
What makes you think you need 22 dimensions and 1344TB for your problem? It must be an extraordinary problem, or it is extraordinarily badly designed... You should first explain what you are trying to do. - user1220978

2 Answers

2
votes

The Fortran 2008 Standard defines a maximum rank of 15. ifort allows 31. gfortran still does not support more than 7 dimensions as of May 2013 . So - if you really need all 22 dimensions, use ifort - but be aware that this feature is an extension of the standard, other compilers might not compile your code.

EDIT: To sum up the discussion in the comments:

There are two problems - one related to the maximum dimensions and one related to the array size.

For compilers that do not allow for 22 dimensions, one could use one large array of rank 1 that holds all elements and calculate the index inside the element oneself using strides.

Internally all arrays are stored contiguous in memory - think of it as a large array of rank one and size N*M*... Fortran actually stores the resulting array in column-major format. So a matrix Z(NxM) is stored as a vector A(N*M), and access to the elements is calculated by strides: Z(3,4) is at index A( M*(4-1)+3 ). Note that the memory is the same for both arrays!

The next problem is the array size:

For n=2 the code works because than it fits into memory: 2**21*21=44040192 at complex (16 bytes) this amounts to 672 MB. At n=4 this becomes 4**21*21 which is 1344TB.

0
votes

If you're using a method that requires 22 dimensions then you almost certainly need to change your algorithm. Not only will you use enormous amounts of memory, as people have noted elsewhere, but the runtime itself will become absolutely enormous if you want to do something with all the elements in that array.

Because you're using Fortran, my guess is that you are looking at some kind of parameter space with 22 dimensions in, and recording a goodness of fit or likelihood for each combination of parameters. In that case you should consider other algorithms like MCMC for exploring the space.

But this is just a guess - if you tell us what underlying problem you're trying to solve people may have more suggestions.