Insert array columns iteratively in MATLAB

Question

I am trying to construct an array by attaching single columns on that array within a for loop. I am doing that by first initializing a column of zeros and placing each new column I read at the right hand side of the updated vector. But unfortunately, this works partially, and only after making a restriction, i.e. keeping only the first - for example - 500 rows of each new vector.

Regarding the following solution on the problem, first of all I don't understand why this zero column vector do no appear in the last array! The other problem is that even if I do know the total number of columns of the final array from the beginning, in some cases I do not know the actual number of rows of the separate columns. Actually, I want to consider, as the default number of rows in my final array, the bigger number of rows among the different single columns to be attached, so I assume that, the best practice should be to run into the whole number of columns deciding about the bigger number of rows, then setup up the appropriate dimension of the initial array to be filled by the single columns data, and finally to fill the first based on the latter ones.

Any suggestion will be highly appreciated!

data = zeros(500,1);
for symbol = 1:length(symbol_list),
    [hist_data] = retrieve_data(symbol_list(symbol));
    data = [data hist_data(1:500)];
end

EDIT:

Let me give you an input/output example:

I want to place the following three columns (I retrieve one of them in each iteration)

0.402513860949959   0.401546899405730   0.405949059210334
0.401396441661866   0.400288848738267   0.403936753893693
0.400862023741474   0.402233108860710   0.404473368644797
0.411161714570851   0.413612748989126   0.409437055092511
0                   0.414870799656589   0.415071509979105
0                   0.416586323294039   0

one next to the other, ending up with the following table (keep only the ones with the bigger yet the same number of rows - thus, here I keep only the second column):

0.401546899405730
0.400288848738267
0.402233108860710
0.413612748989126
0.414870799656589
0.416586323294039

In other words, I have a very big number of columns that I want to attach one next to the other. The majority of them have a very big length of rows which is the same. However, some other have arbitrary smaller number (length) of rows. What I what is to find this "big" number of rows, keep only the columns correspond to this "big" length, and finally place one next to the other.

Answer 1

I think you might be better off using a cell array for storing the data you get from retrieve_data, since you're handling columns with a different number of rows. Having gathered everything in a cell array, you can still do the conversion to a standard matrix afterwards by padding the shorter rows, or by trimming the longer ones.

c={[1 2 3 4]',[2 43 5]', [1 2]' , [2 5 6 7 3]'}
sizesC = cellfun(@length,c)
% pad to maximal size with Nan's
for el=1:length(c)
    c{el}(sizesC(el)+1:max(sizesC))=NaN;
end
% Trim to minimal size
c={[1 2 3 4]',[2 43 5]', [1 2]' , [2 5 6 7 3]'}
for el=1:length(c)
    c{el}(min(sizesC)+1:end)=[];
end
% Convert rectangular cellarray to matrix
m=cell2mat(c)

Answer 2

I'm not sure I followed most of your question, so this may not be the answer to the question you asked ...

If the data structure you are trying to build is not really an array, for example if it is really a list of vectors of unequal lengths, then a plain old array may not be the best data structure to use in Matlab. It might be the practical choice, but a cell array might be a better choice.

If the array is the right or practical choice, building it by adding a new column at each go round a loop is generally thought, by most experienced Matlab users, to be a VERY BAD IDEA. Building arrays this way can be very expensive in time and may cause memory issues too. That's the general thinking anyway, no one is going to force you to follow the thought.

A much better approach is to allocate your array in large chunks. In an ideal (?) situation you would allocate the entire array in one go, with a statement such as

data = zeros(500,462);

and then write data into each column inside your loop. If you don't know the number of columns you'll eventually want then create an array which is likely to be big enough; if you run out of room, add N columns in one go, carry on looping. Repeat if necessary. A good choice of N is highly dependent on your problem and your data.

The reason that building (large) arrays column-by-column (or row-by-row) is that whenever you add a new column Matlab:

1. grabs enough free memory for the new, larger, array;
2. copies the values from the old memory locations to the new ones;
3. returns the block of memory where the old version of the array was stored to the free memory pool (or whatever Matlab actually calls it)

Doing this repeatedly can take a lot of time. It may also give rise to excessive memory fragmentation, none of the blocks of memory returned to the pool will be large enough to be used for a later version of the same array; get your pattern of memory usage wrong and you'll end up with memory occupied by freed blocks too small for any new arrays, but way over what you need for the odd scalar here and there. This is more of a probability than a certainty, but is worth bearing in mind when cramming data into every byte in the box.