Dimension lost after adding np.array to list

Question

I am adding 3000 np.arrays to a list and converting the list back to np.array. The dimension of the np.array is (270000,). After the entire run of the loop and post np.array conversion process, the dimension of the resultant array is : (3000,) while it's expected to be (3000,270000). The weird thing is that on stopping the process midway, say at 1900 arrays, the observed shape is (1900,270000) which is how I want the output to be at the end of the loop.

from scipy import misc
from os import walk
labels = []
X=[]
Y=[]
for (_, dirnames, _) in walk(mypath):
    labels.extend(dirnames)
for category, label in enumerate(labels):
#     print 'Category:',category,'\tLabel:',label
    for root, dirs, files in os.walk(os.path.abspath(join(mypath,label))):
        for file in files:
            imagePath=os.path.join(root, file)
            face = misc.imread(imagePath)
            flattenImgMatrix=misc.imresize(face,[300, 300], interp='nearest').flatten()
            X.append(flattenImgMatrix)
            Y.append([category])
arrX=np.array(X)
arrY=np.array(Y)

Then check the type of your array, e.g. if the inner elements are of type object. This happens when numpy can't infer enough about those shapes inside to create a numerical array on the outside. Often people feed this with unequal-shaped images, but it seems you care about that. Also keep in mind, that your code is incomplete (e.g. misc: skimage, scipy, imageio, matplotlib?) and we can only guess! — sascha
One or more of the arrays that you add after 1900 does not have the shape (270000,). For example if an image is color it will be (300,300,3) after imresize, and (810000,) after flattening. — hpaulj
Ah, color-channels. Very good guess! I was too shallow there. — sascha
Thank you for your advices. It actually went out of dimension, i.e., expected (300*300*3) but some were (300*300*1). Thus the (3000,) in the resulting array. — achyuta26

jpp jpp · Accepted Answer · 2018-11-05T21:17:19

Without your data, this is impossible to resolve. However, you can add an explicit check in your for loop to debug this for yourself. This will not only confirm an array is the wrong shape, but will provide some useful information to isolate where it occurs:

assert flattenImgMatrix.shape == (270000,), f'Error: {file}: {flattenImgMatrix.shape}'

Dimension lost after adding np.array to list

1 Answers