python - Pandas: conditional rolling count

53

votes

I have a Series that looks the following:

   col
0  B
1  B
2  A
3  A
4  A
5  B

It's a time series, therefore the index is ordered by time.

For each row, I'd like to count how many times the value has appeared consecutively, i.e.:

Output:

   col count
0  B   1
1  B   2
2  A   1 # Value does not match previous row => reset counter to 1
3  A   2
4  A   3
5  B   1 # Value does not match previous row => reset counter to 1

I found 2 related questions, but I can't figure out how to "write" that information as a new column in the DataFrame, for each row (as above). Using rolling_apply does not work well.

Finding consecutive segments in a pandas data frame

pythonpandas

56

votes

I think there is a nice way to combine the solution of @chrisb and @CodeShaman (As it was pointed out CodeShamans solution counts total and not consecutive values).

  df['count'] = df.groupby((df['col'] != df['col'].shift(1)).cumsum()).cumcount()+1

  col  count
0   B      1
1   B      2
2   A      1
3   A      2
4   A      3
5   B      1

24

votes

One-liner:

df['count'] = df.groupby('col').cumcount()

or

df['count'] = df.groupby('col').cumcount() + 1

if you want the counts to begin at 1.

19

votes

Based on the second answer you linked, assuming s is your series.

df = pd.DataFrame(s)
df['block'] = (df['col'] != df['col'].shift(1)).astype(int).cumsum()
df['count'] = df.groupby('block').transform(lambda x: range(1, len(x) + 1))


In [88]: df
Out[88]: 
  col  block  count
0   B      1      1
1   B      1      2
2   A      2      1
3   A      2      2
4   A      2      3
5   B      3      1

12

votes

I like the answer by @chrisb but wanted to share my own solution, since some people might find it more readable and easier to use with similar problems....

1) Create a function that uses static variables

def rolling_count(val):
    if val == rolling_count.previous:
        rolling_count.count +=1
    else:
        rolling_count.previous = val
        rolling_count.count = 1
    return rolling_count.count
rolling_count.count = 0 #static variable
rolling_count.previous = None #static variable

2) apply it to your Series after converting to dataframe

df  = pd.DataFrame(s)
df['count'] = df['col'].apply(rolling_count) #new column in dataframe

output of df

  col  count
0   B      1
1   B      2
2   A      1
3   A      2
4   A      3
5   B      1

1

votes

If you wish to do the same thing but filter on two columns, you can use this.

def count_consecutive_items_n_cols(df, col_name_list, output_col):
    cum_sum_list = [
        (df[col_name] != df[col_name].shift(1)).cumsum().tolist() for col_name in col_name_list
    ]
    df[output_col] = df.groupby(
        ["_".join(map(str, x)) for x in zip(*cum_sum_list)]
    ).cumcount() + 1
    return df

col_a col_b count
0   1     B     1
1   1     B     2
2   1     A     1
3   2     A     1
4   2     A     2
5   2     B     1

python - Pandas: conditional rolling count

5 Answers