Large arrays in Matlab

2
votes

I am trying to build five single precision arrays of the size 744×744×744×3×3 in the latest MATLAB version (R2016b).

However, when I build the first array, I get the error:

Requested 744x744x744x2x3 (9.2GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.

I set the workspace preferences in MATLAB to max array size 1e4, which is all that it allows. And I set the maximum virtual memory in Windows 10 to 400GB.

I also read the relevant posts in this forum, but they don't answer my question. Is it impossible to build arrays that size or am I missing something?

1
arraysmatlab
For what reason you want to save such a big matrix? It's very difficult and slow to work with a matrix like that, it's better to divide it into some files or variables... - Adiel
If it's possible, then I would prefer doing it that way because I already wrote a programme that works very well for smaller arrays. It's a stochastic dynamic programming optimization with backward induction. If it's not possible to have that size of arrays, I think have to write the model anew and go stepwise, i.e. build part of the matrix and save the last layer and then build next part. But I'd prefer to just leave it the way it works now. - LenaH
That should be five 13.8GB arrays (69GB total). Needless to say, that's going to be problematic if MATLAB doesn't have enough contiguous RAM available. What is the output of memory (in MATLAB)? Would sparse be of any help? - Rody Oldenhuis
By the way, is it 744×744×744×3×3 (in your question) or 744×744×744×2×3 (in MATLAB's error message)? - Rody Oldenhuis
I've worked with servers and clusters that had well over 512GB of RAM available. On those sorts of machines, your arrays will not pose much of a problem. But it sounds like you're on a regular PC, with (probably) more like 16GB or RAM. If that is the case, this operation is going to be painfully slow -- split the problem up into smaller pieces, like @Adiel suggested - Rody Oldenhuis

1 Answers

1
votes

You are exceeding your RAM, I can suggest to use matfile.

To save the large matrices (for example My_var, having size Nvar1 x Nvar2), without slowing the other processes... 

myObject = matfile('myFilename.mat','Writable',true);
myObject.myVariablenameinObject(1:Nvar1,1:Nvar2)=My_var(1:Nvar1,1:Nvar2)

By setting 'Writable' as true, you can access, modify or write data. If you don't want to write. Just use:

myObject = matfile('myFilename.mat')

For more details, refer to this link.