Impala Resource Management

Question

Admission control is embedded within each impalad daemon and communicates through the statestore service. The impalad daemon determines if a query runs immediately or if the query is queued. However If a sudden flow of requests causes more queries to run concurrently than expected, the overall Impala memory limit and the Linux cgroups mechanism at the cluster level serve as hard limits to prevent over allocation of memory. When queries hit these limits, Impala cancels the queries.

Does this mean Impala Resource Limits Enforced at individual Impala daemon level or at the cluster level?.

Answer 1

The answer is both. Each impalad daemon has its own MEM_LIMIT. Exceeding it will cause the query to be canceled. The admission control pool works at the cluster level, even though the gatekeep (decide whether the query should be run or queued) is at each impalad level, even though these impalad makes the admission decision based on the pool resource at the cluster level. That's why when there are a flood of queries sent to different impalad instances, the impalad daemons might admit more queries than they should because they cannot get the most current cluster resource usage information at the time. CGroup limit does not cause query to be canceled. It determines the percentage of CPU that the impalad should get when there is CPU contention.