find the set of column indices for non-zero values in each row in pandas' data frame

Question

Is there a good way to find the set of column indices for non-zero values in each row in pandas' data frame? Do I have to traverse the data frame row-by-row?

For example, the data frame is

c1  c2  c3  c4 c5 c6 c7 c8  c9
 1   1   0   0  0  0  0  0   0
 1   0   0   0  0  0  0  0   0
 0   1   0   0  0  0  0  0   0
 1   0   0   0  0  0  0  0   0
 0   1   0   0  0  0  0  0   0
 0   0   0   0  0  0  0  0   0
 0   2   1   1  1  1  1  0   2
 1   5   5   0  0  1  0  4   6
 4   3   0   1  1  1  1  5  10
 3   5   2   4  1  2  2  1   3
 6   4   0   1  0  0  0  0   0
 3   9   1   0  1  0  2  1   0

The output is expected to be

['c1','c2']
['c1']
['c2']
...

Answer 1

It seems you have to traverse the DataFrame by row.

cols = df.columns
bt = df.apply(lambda x: x > 0)
bt.apply(lambda x: list(cols[x.values]), axis=1)

and you will get:

0                                 [c1, c2]
1                                     [c1]
2                                     [c2]
3                                     [c1]
4                                     [c2]
5                                       []
6             [c2, c3, c4, c5, c6, c7, c9]
7                 [c1, c2, c3, c6, c8, c9]
8         [c1, c2, c4, c5, c6, c7, c8, c9]
9     [c1, c2, c3, c4, c5, c6, c7, c8, c9]
10                            [c1, c2, c4]
11                [c1, c2, c3, c5, c7, c8]
dtype: object

If performance is matter, try to pass raw=True to boolean DataFrame creation like below:

%timeit df.apply(lambda x: x > 0, raw=True).apply(lambda x: list(cols[x.values]), axis=1)
1000 loops, best of 3: 812 µs per loop

It brings you a better performance gain. Following is raw=False (which is default) result:

%timeit df.apply(lambda x: x > 0).apply(lambda x: list(cols[x.values]), axis=1)
100 loops, best of 3: 2.59 ms per loop

Answer 2

Potentially a better data structure (rather than a Series of lists) is to stack:

In [11]: res = df[df!=0].stack()

In [12]: res
Out[12]:
0   c1     1
    c2     1
1   c1     1
2   c2     1
3   c1     1
...

And you can iterate over the original rows:

In [13]: res.loc[0]
Out[13]:
c1    1
c2    1
dtype: float64

In [14]: res.loc[0].index
Out[14]: Index(['c1', 'c2'], dtype='object')

---

Note: I thought you used to be able to return a list in an apply (to create a DataFrame which has list elements) this no longer appears to be the case.

Answer 3

How about this approach?

#create a True / False data frame
df_boolean = df>0

#a little helper method that uses boolean slicing internally 
def bar(x,columns):
    return ','.join(list(columns[x]))

#use an apply along the column axis
df_boolean['result'] = df_boolean.apply(lambda x: bar(x,df_boolean.columns),axis=1)

# filter out the empty "rows" adn grab the result column
df_result =  df_boolean[df_boolean['result'] != '']['result']

#append an axis, just so each line will will output a list 
lst_result = df_result.values[:,np.newaxis]

print '\n'.join([ str(myelement) for myelement in lst_result])

and this produces:

['c1,c2']
['c1']
['c2']
['c1']
['c2']
['c2,c3,c4,c5,c6,c7,c9']
['c1,c2,c3,c6,c8,c9']
['c1,c2,c4,c5,c6,c7,c8,c9']
['c1,c2,c3,c4,c5,c6,c7,c8,c9']
['c1,c2,c4']
['c1,c2,c3,c5,c7,c8']