MATLAB - Problems with large arrays

Question

The code below works great for small vectors.

[X1, X2]=meshgrid(Data1(:,1), Data2(:,1));
[Y1, Y2]=meshgrid(Data1(:,2), Data2(:,2));
[Z1, Z2]=meshgrid(Data1(:,3), Data2(:,3));

Rxy = sqrt( (X1-X2).^2 + (Y1-Y2).^2 );
Rz = abs( Z1-Z2 );

[I1, I2] =find( Rxy<=100 & Rz<=0.2);

However, as I work with a large amount of data, matlab does not support and does not work correctly. Matlab generates the following message:

Error using repmat Requested 75027x68517 (38.3GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to Become unresponsive. See array size limit or preference panel for more information.

Seeking alternatives to "out of memory error," but I'm not getting an efficient manner. I made a loop to function is, but it was extremely slow.

Yikes... that matrix size is HUGE. Let's take a step back. Can you explain to us what it is you're trying to do? If we know the purpose of that code, then we may even be able to provide alternatives. — rayryeng
I'm an oceanographer and work with bathymetric data analysis (.xyz). Where X is position, position and Y is Z is a water depth I'm making a program to join the line segments that will make up a terrain profile. First, I select and identify the segments, posteriomente, I find the set of intersection between the two segments. Within this set I find the best connection point. This point should be less than 10m distance and difference Z (water depth) of less than 0.2m. So I find the possible points for joining the profiles. — Vinicius Nogueira
If you have enough memory, MATLAB should not stop you from creating that array (Assuming you have >40 GB of RAM). See here. It may be the limitation of repmat. Try to not use repmat to replicate arrays. See here. Also, if you feel using loop is costing you time, why don't you write a MEX-file, which will save you time. — Autonomous

EelkeSpaak EelkeSpaak · Accepted Answer · 2015-06-04T11:08:37

Presumably meshgrid is using repmat internally to create some massive matrices. Have a look at bsxfun, allows these types of operations without actually replicating the data. From the help:

C = bsxfun(fun,A,B) applies the element-by-element binary operation specified by the function handle fun to arrays A and B, with singleton expansion enabled.

So line 5 of your example would become (untested, don't have Matlab available right now:

Rxy = sqrt( bsxfun(@minus, Data1(:,1), Data2(:,1)).^2 + bsxfun(@minus, Data1(:,2), Data2(:,2)).^2 );

and change the line below analogously.

MATLAB - Problems with large arrays

1 Answers