Time Series With Unevenly Sized Elements

Question

I have a question about taking an element within a numpy array and resizing it in a loop.

The basic issue is that I have a time series in the format (x,y) with some elements that are missing y values so they are read into the program as being of length one.

ie: [x,y] [x,y] [x] [x,y]

So i need to take these occasional points and resize them to the standard (1,2) (and then generate a point based of a distribution but that isn't the trouble). I am aware of the numpy.resize function, but when I try:

for element in list:
   if len(element)==1:
      element=n.resize(element,(1,2))

it works within the scope of the loop but if I print the list all of the elements are the same as if the loop never occurred. The resize function returns an array so I'm confused as why setting the element of list equal to that array does not work.

Edit: I found a simple solution using regular python lists:

for element in list:
    if len(element)==1:
       element.append(0)

But I am still curious as to why the above doesn't work, because certainly setting an element equal to something is possible. Or can you only set element[i] equal to something in a [1,x] array?

for dealing with time series, I would suggest pandas. The read_csv function should provide everything you need to parse your data. — bmu

Pierre GM Pierre GM · Accepted Answer · 2012-09-28T08:13:58

With element=n.resize(element, (1,2)), you're actually creating a new object element that is the result of np.resize on the old element, but as you're not doing anything with this new object, your modifications get lost.

When you do element.append(0), you're modifying element in place, so you're still referring to the initial object and your modifications are saved.

That said, there are better ways to deal with missing elements in a ndarray. As you've noticed, you cannot create a (N,2) array out of a list of N elements if not all these elements are sequences of two items: you have to fill the gaps somehow. If your initial data came from a text file, you could use np.genfromtxt with the usemask=True parameter to create a MaskedArray:

>>> data = StringIO.StringIO("1,2\n3,4\n,6\n7,\n")
>>> x = np.genfromtxt(data, delimiter=",", usemask=True)
>>> x
masked_array(data =
 [[1.0 2.0]
 [3.0 4.0]
 [-- 6.0]
 [7.0 --]],
             mask =
 [[False False]
 [False False]
 [ True False]
 [False  True]],
       fill_value = 1e+20)

The initial gaps have been filled with the special np.ma.masked value for you.

Time Series With Unevenly Sized Elements

2 Answers