Python Memory Error - Numpy Matrix (1568x1568)

Question

I have two very similar programs whose functions differ only slightly. Within both programs, a 1568 x 1568 numpy matrix is produced. In one of my programs, this matrix creation works perfectly, however, in the other, I receive a MemoryError. Error message below:

    a = empty(shape, dtype, order)
MemoryError

Also, when I copy the section of code that creates the matrix, from the program that does work, into its own python file, this too produces the MemoryError.

The code below reproduces the error:

import matplotlib.pyplot as plt
import numpy as np

Z = np.zeros((1568, 1568))
fig = plt.figure(figsize=(6,6))
X = np.arange(0, 1568, 0.1)
Y = np.arange(0, 1568, 0.1)
X, Y = np.meshgrid(X, Y)

plt.contourf(X, Y, Z)
plt.show()

I don't think this is caused by the fact that I'm using 32 bit python rather than 64 bit (as is the case with others who have posted questions with similar problems), purely because of the fact that this does work in my other program and the matrix should only be using 9.83 MB of RAM anyway.

Any help would be greatly appreciated.

Is that the full error message? Usually the actual line of code that causes the error is included. I assume that line is more likely to be X, Y = np.meshgrid(X, Y), as that is going to be two huge 15680x15680 matrices — Daniel F
After meshgrid, X and Y are both roughly 1.8 GB in size, and contourf will also need to use some RAM. With 32 bit you are limited to 4 GB total RAM size so yes, it is related to you using 32 bit Python. — Nils Werner
@DanielF yes, sorry, you are correct that is the line causing the error. Is there any reson you can think of why this would work in another program? The code posted is exactly the same as the code used in the functioning program. — C. Pugh
are the linspace assignments the same? i.e. is the step size 0.1? I'm not sure that code would work as written as Z is 1/10th the size of X and Y — Daniel F
As written, even without memory errors, plt.contourf(X, Y, Z) should fail because Z.shape is (1568, 1568) and X.shape == Y.shape == (15680, 15680) and in the docs it states "X and Y must both be 2-D with the same shape as Z," — Daniel F

C. Pugh C. Pugh · Accepted Answer · 2017-07-04T09:52:03

Thanks to the help in the comments, I realised that the code in the program that I said was working, wasn't actually exactly the same as the other program and had a matrix of size 156.8 x 156.8 rather than 1568 x 1568. In the full code, I have variables rather than numbers in the matrix set up, so this wasn't immediately obvious. I changed the program giving the error to match this and the problem was solved.

Python Memory Error - Numpy Matrix (1568x1568)

1 Answers