Dynamically adding elements to a NumPy array of unknown final length

Question

I want to create a very large vector of numbers, but the length of that vector is not known. However, I could give a maximum length (which would be around 100k), although most likely the list will actually be around 10k. Essentially, I have a for loop, within which I keep adding numbers to the vector, until a certain criterion is met.

My first attempt at doing this was with Python lists, where I used the following:

x = []
for i in range(K):
    y = get_list_of_numbers()
    x += y

However, ultimately I want to convert this list to a NumPy array, for further processing. If I do this using a = np.array(x), it takes a very long time to create that array.

So, my second solution was to create an empty NumPy array from the beginning, and add elements to it as I go along:

x = np.empty([])
for i in range(K):
    y = get_list_of_numbers()
    np.append(x, y)

However, here the np.append(x, y) takes a very long time to process.

So, both my solutions are very slow. Is there a quicker solution out there?

The only remaining solution I can think of is to create a huge NumPy array at the maximum length, and then insert each element into the appropriate slot in that array. However, this would be very memory inefficient, as I do not really have a good estimate of the maximum vector length...

Thanks!

Your use of np.append is wrong - stay away from that function. The list append approach is usually fastest - even if the final np.array call takes time. stackoverflow.com/questions/43237035/… — hpaulj
I agree with Divakar. Make an initial array with np.zeros(initial_size_guess) and keep track of how many numbers you've added to the vector (n), then take slice your vector to only take the first n elements. — Alex

hpaulj hpaulj · Accepted Answer · 2017-04-07T17:54:38

If I define:

def get_list_of_numbers():
    n = np.random.randint(0,10)
    return list(range(n))

def foo(K):
    x=[]
    for i in range(K):
        y = get_list_of_numbers()
        x.extend(y)
    return x

Simply calling get_list_of_numbers takes the most time. Turning the result into an array doesn't take much time:

In [69]: timeit foo(1000)
100 loops, best of 3: 5.9 ms per loop
In [70]: timeit np.array(foo(1000))
100 loops, best of 3: 6.38 ms per loop
In [73]: timeit -n1000 get_list_of_numbers()
1000 loops, best of 3: 6.04 µs per loop

Lets try the preallocation approach:

def foo1(K):
    x = np.zeros(K*10,int)
    cnt = 0
    for i in range(K):
        y = get_list_of_numbers()
        n = len(y)
        x[cnt:cnt+n] = y
        cnt += n
    x = x[:cnt]
    return x

In [80]: timeit foo1(1000)
100 loops, best of 3: 10.1 ms per loop

The array concatenation approach

In [48]: def foo1(K):
    ...:     x = np.zeros(0,int)
    ...:     for i in range(K):
    ...:         y = get_list_of_numbers()
    ...:         x = np.concatenate((x, y), axis=0)
    ...:     return x
In [51]: timeit foo1(1000).shape
100 loops, best of 3: 15.9 ms per loop

Dynamically adding elements to a NumPy array of unknown final length

1 Answers