Hive don't get write lock by on new partition when there is read lock on older partitions

Question

We have data that is partitioned by date. Our pipeline everyday add new partition to table. We also have use cases, where people are running long read queries by filtering on partitions.

We have seen the behavior, where, If there is some read query running on old partitions (Having SHARED lock), then we can't acquire lock for new partition (EXCLUSIVE).

Is there a way that HIVE recognize that, new partition will not be read by earlier query and acquire EXLUSIVE lock for that partition to increase parallelism.

Answer 1

According to the documentation having a shared lock on a partition, does not prevent you from getting an exclusive lock on that same partition.

In theory your question would then appear invalid, however in practice I could think of the following two things:

1. It is too hard for hive to see which partitions you are using (complicated/indirect where statement?)
2. The documentation is not accurate, or misinterpreted by me

I would suggest you to check for point 1, and otherwise add a minimal reproducible example.