How to drop duplicates in Vaex?

Question

I have some entries from users and how many interactions this user had on my website...

I have 340k rows and 70+ columns, and I want to use Vaex, but I'm having problems to do simple things like to drop duplicates.

Could someone help me on how to do it?

import pandas as pd

df = pd.DataFrame({'user': ['Bob', 'Bob', 'Alice', 'Alice', 'Alice', "Ralph", "Ralph"], 
                   'date': ['2013-12-05', '2014-02-05', '2013-11-07', '2014-04-22', '2014-04-30',  '2014-04-20', '2014-05-29'],
                   'interaction_num': ['1', '2', '1', '2', '3', '1','2']})

I want to have the same result of the pandas.drop_duplicates(keep="last") function

df.drop_duplicates('user', keep='last', inplace=True)

the expected result using Vaex should be:

    user    date    interaction_num
1   Bob     2014-02-05  2
4   Alice   2014-04-30  3
6   Ralph   2014-05-29  2

radupm radupm · Accepted Answer · 2021-02-27T19:00:07

Duplicate question

It seems there is none yet, but we should expect this functionality at some point.

In the meantime, there is an attempt from the creator of vaex

How to drop duplicates in Vaex?

1 Answers