I am working on a panel data database which is organized as following: first column is the date under the format yyyymm and the second column is the group identifier under the format of numbers such as 100232. The database source is VerticaPy under Jupyter notebook. Due to the massive amount of data the data preferably transferred as Pandas DataFrame after limiting the amount of lines.
The problem is the following: I want to conserve only the groups that appear for all the dates (starting first date, up till last date). For example, identifier: 12345 is present from june 2012 till june 2022 so I keep it, while identifier 33333 starts from may 2015 till june 2022 so I want to drop it. This excel screenshot summarizes the problem.
Is there a way to keep a constant population sampling over time either through pandas, VerticaPy, or even SQL (VerticaPy accepts SQL queries) ?
In the photo, we keep A & C groups because they are there in all the dates while B & D are not kept.enter image description here
