The documentation for Dask talks about repartioning to reduce overhead here.
They however seem to indicate you need some knowledge of what your dataframe will look like beforehand (ie that there will 1/100th of the data expected).
Is there a good way to repartition sensibly without making assumptions? At the moment I just repartition with npartitions = ncores * magic_number, and set force to True to expand partitions if need be. This one size fits all approach works but is definitely suboptimal as my dataset varies in size.
The data is time series data, but unfortunately not at regular intervals, I've used repartition by time frequency in the past but this would be suboptimal because of how irregular the data is (sometimes nothing for minutes then thousands in seconds)
len()ondf[df.name == 'Alice']be too expensive? I feel like it shouldn't be - each worker sums up their dataframe length and the scheduler reduces that to a sum. Then, once you have that number, you can create a ratio of the original height to the current, and update the partition count via repartition accordingly. - kuanbcount()on you subset query get you the length without needing load it into memory as a Pandas DataFrame and then resend it back to Dask? Seems like that operation itself is intense? - kuanbdf.memory_usage().sum().compute()to help determine a good number of partitions. - MRocklin