0
votes

I have an array X with dimension mxn, for every row m I want to get a correlation with a vector y with dimension n.

In Matlab this would be possible with the corr function corr(X,y). For Python however this does not seem possible with the np.corrcoef function:

import numpy as np
X = np.random.random([1000, 10])
y = np.random.random(10)
np.corrcoef(X,y).shape

Which results in shape (1001, 1001). But this will fail when the dimension of X is large. In my case, there is an error:

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 5.93 TiB for an array with shape (902630, 902630) and data type float64

Since the X.shape[0] dimension is 902630.

My question is, how can I only get the row wise correlations with the vector resulting in shape (1000,) of all correlations?

Of course this could be done via a list comprehension:

np.array([np.corrcoef(X[i, :], y)[0,1] for i in range(X.shape[0])])

Currently I am therefore using numba with a for loop running through the >900000 elemens. But I think there could be a much more efficient matrix operation function for this problem.