Admission Control and Query Queuing
Admission control is an Impala feature that imposes limits on concurrent SQL queries, to avoid resource usage spikes and out-of-memory conditions on busy clusters. It is a form of "throttling". New queries are accepted and executed until certain conditions are met, such as too many queries or too much total memory used across the cluster. When one of these thresholds is reached, incoming queries wait to begin execution. These queries are queued and are admitted (that is, begin executing) when the resources become available.
In addition to the threshold values for currently executing queries, you can place limits on the maximum number of queries that are queued (waiting) and a limit on the amount of time they might wait before returning with an error. These queue settings let you ensure that queries do not wait indefinitely, so that you can detect and correct "starvation" scenarios.
Enable this feature if your cluster is underutilized at some times and overutilized at others. Overutilization is indicated by performance bottlenecks and queries being cancelled due to out-of-memory conditions, when those same queries are successful and perform well during times with less concurrent load. Admission control works as a safeguard to avoid out-of-memory conditions during heavy concurrent usage.
The use of the Llama component for integrated resource management within YARN is no longer supported with Impala 2.3 and higher. The Llama support code is removed entirely in Impala 2.8 and higher.
For clusters running Impala alongside other data management components, you define static service pools to define the resources available to Impala and other components. Then within the area allocated for Impala, you can create dynamic service pools, each with its own settings for the Impala admission control feature.
Overview of Impala Admission Control
On a busy cluster, you might find there is an optimal number of Impala queries that run concurrently. For example, when the I/O capacity is fully utilized by I/O-intensive queries, you might not find any throughput benefit in running more concurrent queries. By allowing some queries to run at full speed while others wait, rather than having all queries contend for resources and run slowly, admission control can result in higher overall throughput.
For another example, consider a memory-bound workload such as many large joins or aggregation queries. Each such query could briefly use many gigabytes of memory to process intermediate results. Because Impala by default cancels queries that exceed the specified memory limit, running multiple large-scale queries at once might require re-running some queries that are cancelled. In this case, admission control improves the reliability and stability of the overall workload by only allowing as many concurrent queries as the overall memory of the cluster can accomodate.
The admission control feature lets you set an upper limit on the number of concurrent Impala queries and on the memory used by those queries. Any additional queries are queued until the earlier ones finish, rather than being cancelled or running slowly and causing contention. As other queries finish, the queued queries are allowed to proceed.
In Impala 2.5 and higher, you can specify these limits and thresholds for each pool rather than globally. That way, you can balance the resource usage and throughput between steady well-defined workloads, rare resource-intensive queries, and ad hoc exploratory queries.
For details on the internal workings of admission control, see How Impala Schedules and Enforces Limits on Concurrent Queries.
Concurrent Queries and Admission Control
One way to limit resource usage through admission control is to set an upper limit on the number of concurrent queries. This is the initial technique you might use when you do not have extensive information about memory usage for your workload. This setting can be specified separately for each dynamic resource pool.
You can combine this setting with the memory-based approach described in Memory Limits and Admission Control. If either the maximum number of or the expected memory usage of the concurrent queries is exceeded, subsequent queries are queued until the concurrent workload falls below the threshold again.
Memory Limits and Admission Control
Each dynamic resource pool can have an upper limit on the cluster-wide memory used by queries executing in that pool. This is the technique to use once you have a stable workload with well-understood memory requirements.
Always specify the Default Query Memory Limit for the expected maximum amount of RAM
that a query might require on each host, which is equivalent to setting the
query option for every query run in that pool. That value affects the execution of each query, preventing it
from overallocating memory on each host, and potentially activating the spill-to-disk mechanism or cancelling
the query when necessary.
Optionally, specify the Max Memory setting, a cluster-wide limit that determines how many queries can be safely run concurrently, based on the upper memory limit per host multiplied by the number of Impala nodes in the cluster.
- The cluster is running impalad daemons on five DataNodes.
- A dynamic resource pool has Max Memory set to 100 GB.
- The Default Query Memory Limit for the pool is 10 GB. Therefore, any query running in this pool could use up to 50 GB of memory (default query memory limit * number of Impala nodes).
- The maximum number of queries that Impala executes concurrently within this dynamic resource pool is two, which is the most that could be accomodated within the 100 GB Max Memory cluster-wide limit.
- There is no memory penalty if queries use less memory than the Default Query Memory Limit per-host setting or the Max Memory cluster-wide limit. These values are only used to estimate how many queries can be run concurrently within the resource constraints for the pool.
You can combine the memory-based settings with the upper limit on concurrent queries described in Concurrent Queries and Admission Control. If either the maximum number of or the expected memory usage of the concurrent queries is exceeded, subsequent queries are queued until the concurrent workload falls below the threshold again.
How Impala Admission Control Relates to Other Resource Management Tools
The admission control feature is similar in some ways to the YARN resource management framework. These features can be used separately or together. This section describes some similarities and differences, to help you decide which combination of resource management features to use for Impala.
Admission control is a lightweight, decentralized system that is suitable for workloads consisting primarily of Impala queries and other SQL statements. It sets "soft" limits that smooth out Impala memory usage during times of heavy load, rather than taking an all-or-nothing approach that cancels jobs that are too resource-intensive.
Because the admission control system does not interact with other Hadoop workloads such as MapReduce jobs, you might use YARN with static service pools on clusters where resources are shared between Impala and other Hadoop components. This configuration is recommended when using Impala in a multitenant cluster. Devote a percentage of cluster resources to Impala, and allocate another percentage for MapReduce and other batch-style workloads. Let admission control handle the concurrency and memory usage for the Impala work within the cluster, and let YARN manage the work for other components within the cluster. In this scenario, Impala's resources are not managed by YARN.
The Impala admission control feature uses the same configuration mechanism as the YARN resource manager to map users to pools and authenticate them.
Although the Impala admission control feature uses a
fair-scheduler.xml configuration file
behind the scenes, this file does not depend on which scheduler is used for YARN. You still use this file
even when YARN is using the capacity scheduler.
How Impala Schedules and Enforces Limits on Concurrent Queries
The admission control system is decentralized, embedded in each Impala daemon and communicating through the statestore mechanism. Although the limits you set for memory usage and number of concurrent queries apply cluster-wide, each Impala daemon makes its own decisions about whether to allow each query to run immediately or to queue it for a less-busy time. These decisions are fast, meaning the admission control mechanism is low-overhead, but might be imprecise during times of heavy load across many coordinators. There could be times when the more queries were queued (in aggregate across the cluster) than the specified limit, or when number of admitted queries exceeds the expected number. Thus, you typically err on the high side for the size of the queue, because there is not a big penalty for having a large number of queued queries; and you typically err on the low side for configuring memory resources, to leave some headroom in case more queries are admitted than expected, without running out of memory and being cancelled as a result.
To avoid a large backlog of queued requests, you can set an upper limit on the size of the queue for queries that are queued. When the number of queued queries exceeds this limit, further queries are cancelled rather than being queued. You can also configure a timeout period per pool, after which queued queries are cancelled, to avoid indefinite waits. If a cluster reaches this state where queries are cancelled due to too many concurrent requests or long waits for query execution to begin, that is a signal for an administrator to take action, either by provisioning more resources, scheduling work on the cluster to smooth out the load, or by doing Impala performance tuning to enable higher throughput.
How Admission Control works with Impala Clients (JDBC, ODBC, HiveServer2)
Most aspects of admission control work transparently with client interfaces such as JDBC and ODBC:
- If a SQL statement is put into a queue rather than running immediately, the API call blocks until the statement is dequeued and begins execution. At that point, the client program can request to fetch results, which might also block until results become available.
- If a SQL statement is cancelled because it has been queued for too long or because it exceeded the memory limit during execution, the error is returned to the client program with a descriptive error message.
In Impala 2.0 and higher, you can submit
SET statement from the client application
to change the
REQUEST_POOL query option.
This option lets you submit queries to different resource pools,
as described in REQUEST_POOL Query Option.
At any time, the set of queued queries could include queries submitted through multiple different Impala
daemon hosts. All the queries submitted through a particular host will be executed in order, so a
CREATE TABLE followed by an
INSERT on the same table would succeed.
Queries submitted through different hosts are not guaranteed to be executed in the order they were
received. Therefore, if you are using load-balancing or other round-robin scheduling where different
statements are submitted through different hosts, set up all table structures ahead of time so that the
statements controlled by the queuing system are primarily queries, where order is not significant. Or, if a
sequence of statements needs to happen in strict order (such as an
INSERT followed by a
SELECT), submit all those statements through a single session, while connected to the same
Impala daemon host.
Admission control has the following limitations or special behavior when used with JDBC or ODBC applications:
The other resource-related query options,
V_CPU_CORES, are no longer used. Those query options only applied to using Impala with Llama, which is no longer supported.
SQL and Schema Considerations for Admission Control
When queries complete quickly and are tuned for optimal memory usage, there is less chance of performance or capacity problems during times of heavy load. Before setting up admission control, tune your Impala queries to ensure that the query plans are efficient and the memory estimates are accurate. Understanding the nature of your workload, and which queries are the most resource-intensive, helps you to plan how to divide the queries into different pools and decide what limits to define for each pool.
For large tables, especially those involved in join queries, keep their statistics up to date
after loading substantial amounts of new data or adding new partitions.
COMPUTE STATS statement for unpartitioned tables, and
COMPUTE INCREMENTAL STATS for partitioned tables.
When you use dynamic resource pools with a Max Memory setting enabled,
you typically override the memory estimates that Impala makes based on the statistics from the
COMPUTE STATS statement.
You either set the
MEM_LIMIT query option within a particular session to
set an upper memory limit for queries within that session, or a default
setting for all queries processed by the impalad instance, or
MEM_LIMIT setting for all queries assigned to a particular
dynamic resource pool. By designating a consistent memory limit for a set of similar queries
that use the same resource pool, you avoid unnecessary query queuing or out-of-memory conditions
that can arise during high-concurrency workloads when memory estimates for some queries are inaccurate.
Follow other steps from Tuning Impala for Performance to tune your queries.
Configuring Admission Control
The configuration options for admission control range from the simple (a single resource pool with a single set of options) to the complex (multiple resource pools with different options, each pool handling queries for a different set of users and groups).
Impala Service Flags for Admission Control (Advanced)
The following Impala configuration options let you adjust the settings of the admission control feature. When supplying the
options on the impalad command line, prepend the option name with
Purpose: Maximum amount of time (in milliseconds) that a
request waits to be admitted before timing out.
Purpose: Maximum number of concurrent outstanding requests
allowed to run before incoming requests are queued. Because this
limit applies cluster-wide, but each Impala node makes independent
decisions to run queries immediately or queue them, it is a soft
limit; the overall number of concurrent queries might be slightly
higher during times of heavy load. A negative value indicates no
limit. Ignored if
Default: -1, meaning unlimited (prior to Impala 2.5 the default was 200)
Purpose: Maximum number of requests allowed to be queued
before rejecting requests. Because this limit applies
cluster-wide, but each Impala node makes independent decisions to
run queries immediately or queue them, it is a soft limit; the
overall number of queued queries might be slightly higher during
times of heavy load. A negative value or 0 indicates requests are
always rejected once the maximum concurrent requests are
executing. Ignored if
Purpose: Maximum amount of memory (across the entire
cluster) that all outstanding requests in this pool can use before
new requests to this pool are queued. Specified in bytes,
megabytes, or gigabytes by a number followed by the suffix
g, either uppercase or lowercase. You can specify floating-point values for megabytes and gigabytes, to represent fractional numbers such as
1.5. You can also specify it as a percentage of the physical memory by specifying the suffix
%. 0 or no setting indicates no limit. Defaults to bytes if no unit is given. Because this limit applies cluster-wide, but each Impala node makes independent decisions to run queries immediately or queue them, it is a soft limit; the overall memory used by concurrent queries might be slightly higher during times of heavy load. Ignored if
llama_site_pathare set.Note: Impala relies on the statistics produced by the
COMPUTE STATSstatement to estimate memory usage for each query. See COMPUTE STATS Statement for guidelines about how and when to use this statement.
""(empty string, meaning unlimited)
Purpose: Turns off the admission control feature entirely,
regardless of other configuration option settings.
Purpose: Disables all per-pool limits on the maximum number
of running requests.
Purpose: Disables all per-pool mem limits.
Purpose: Path to the fair scheduler allocation file
Usage notes: Admission control only uses a small subset of the settings that can go in this file, as described below. For details about all the Fair Scheduler configuration settings, see the Apache wiki.
Purpose: Path to the configuration file used by admission control
llama-site.xml). If set,
fair_scheduler_allocation_pathmust also be set.
Usage notes: Admission control only uses a few of the settings that can go in this file, as described below.
Configuring Admission Control Using the Command Line
To configure admission control, use a combination of startup options for the Impala daemon and edit or create the configuration files fair-scheduler.xml and llama-site.xml.
For a straightforward configuration using a single resource pool named
default, you can
specify configuration options on the command line and skip the fair-scheduler.xml
and llama-site.xml configuration files.
For an advanced configuration with multiple resource pools using different settings, set up the
fair-scheduler.xml and llama-site.xml configuration files
manually. Provide the paths to each one using the impalad command-line options,
The Impala admission control feature only uses the Fair Scheduler configuration settings to determine how to map users and groups to different resource pools. For example, you might set up different resource pools with separate memory limits, and maximum number of concurrent and queued queries, for different categories of users within your organization. For details about all the Fair Scheduler configuration settings, see the Apache wiki.
The Impala admission control feature only uses a small subset of possible settings from the llama-site.xml configuration file:
llama.am.throttling.maximum.placed.reservations.queue_name llama.am.throttling.maximum.queued.reservations.queue_name impala.admission-control.pool-default-query-options.queue_name impala.admission-control.pool-queue-timeout-ms.queue_name
setting specifies the timeout value for this pool, in milliseconds.
settings designates the default query options for all queries that run
in this pool. Its argument value is a comma-delimited string of
'key=value' pairs, for example,
For example, this is where you might set a default memory limit
for all queries in the pool, using an argument such as
impala.admission-control.* configuration settings are available in
Impala 2.5 and higher.
Example of Admission Control Configuration
Here are sample fair-scheduler.xml and
llama-site.xml files that define resource pools
root.production. These sample files are stripped down: in a real
deployment they might contain other settings for use with various aspects of the YARN
component. The settings shown here are the significant ones for the Impala admission
Although Impala does not use the
vcores value, you must still specify it to satisfy
YARN requirements for the file contents.
<aclSubmitApps> tag (other than the one for
a comma-separated list of users, then a space, then a comma-separated list of groups; these are the
users and groups allowed to submit Impala statements to the corresponding resource pool.
If you leave the
<aclSubmitApps> element empty for a pool, nobody can submit
directly to that pool; child pools can specify their own
to authorize users and groups to submit to those pools.
<allocations> <queue name="root"> <aclSubmitApps> </aclSubmitApps> <queue name="default"> <maxResources>50000 mb, 0 vcores</maxResources> <aclSubmitApps>*</aclSubmitApps> </queue> <queue name="development"> <maxResources>200000 mb, 0 vcores</maxResources> <aclSubmitApps>user1,user2 dev,ops,admin</aclSubmitApps> </queue> <queue name="production"> <maxResources>1000000 mb, 0 vcores</maxResources> <aclSubmitApps> ops,admin</aclSubmitApps> </queue> </queue> <queuePlacementPolicy> <rule name="specified" create="false"/> <rule name="default" /> </queuePlacementPolicy> </allocations>
<?xml version="1.0" encoding="UTF-8"?> <configuration> <property> <name>llama.am.throttling.maximum.placed.reservations.root.default</name> <value>10</value> </property> <property> <name>llama.am.throttling.maximum.queued.reservations.root.default</name> <value>50</value> </property> <property> <name>impala.admission-control.pool-default-query-options.root.default</name> <value>mem_limit=128m,query_timeout_s=20,max_io_buffers=10</value> </property> <property> <name>impala.admission-control.pool-queue-timeout-ms.root.default</name> <value>30000</value> </property> <property> <name>llama.am.throttling.maximum.placed.reservations.root.development</name> <value>50</value> </property> <property> <name>llama.am.throttling.maximum.queued.reservations.root.development</name> <value>100</value> </property> <property> <name>impala.admission-control.pool-default-query-options.root.development</name> <value>mem_limit=256m,query_timeout_s=30,max_io_buffers=10</value> </property> <property> <name>impala.admission-control.pool-queue-timeout-ms.root.development</name> <value>15000</value> </property> <property> <name>llama.am.throttling.maximum.placed.reservations.root.production</name> <value>100</value> </property> <property> <name>llama.am.throttling.maximum.queued.reservations.root.production</name> <value>200</value> </property> <!-- Default query options for the 'root.production' pool. THIS IS A NEW PARAMETER in Impala 2.5. Note that the MEM_LIMIT query option still shows up in here even though it is a separate box in the UI. We do that because it is the most important query option that people will need (everything else is somewhat advanced). MEM_LIMIT takes a per-node memory limit which is specified using one of the following: - '<int>[bB]?' -> bytes (default if no unit given) - '<float>[mM(bB)]' -> megabytes - '<float>[gG(bB)]' -> in gigabytes E.g. 'MEM_LIMIT=12345' (no unit) means 12345 bytes, and you can append m or g to specify megabytes or gigabytes, though that is not required. --> <property> <name>impala.admission-control.pool-default-query-options.root.production</name> <value>mem_limit=386m,query_timeout_s=30,max_io_buffers=10</value> </property> <!-- Default queue timeout (ms) for the pool 'root.production'. If this isn’t set, the process-wide flag is used. THIS IS A NEW PARAMETER in Impala 2.5. --> <property> <name>impala.admission-control.pool-queue-timeout-ms.root.production</name> <value>30000</value> </property> </configuration>
Guidelines for Using Admission Control
To see how admission control works for particular queries, examine the profile output for the query. This
information is available through the
PROFILE statement in impala-shell
immediately after running a query in the shell, on the queries page of the Impala
debug web UI, or in the Impala log file (basic information at log level 1, more detailed information at log
level 2). The profile output contains details about the admission decision, such as whether the query was
queued or not and which resource pool it was assigned to. It also includes the estimated and actual memory
usage for the query, so you can fine-tune the configuration for the memory limits of the resource pools.
Remember that the limits imposed by admission control are "soft" limits.
The decentralized nature of this mechanism means that each Impala node makes its own decisions about whether
to allow queries to run immediately or to queue them. These decisions rely on information passed back and forth
between nodes by the statestore service. If a sudden surge in requests causes more queries than anticipated to run
concurrently, then throughput could decrease due to queries spilling to disk or contending for resources;
or queries could be cancelled if they exceed the
MEM_LIMIT setting while running.
In impala-shell, you can also specify which resource pool to direct queries to by
REQUEST_POOL query option.
The statements affected by the admission control feature are primarily queries, but also include statements
that write data such as
CREATE TABLE AS SELECT. Most write
operations in Impala are not resource-intensive, but inserting into a Parquet table can require substantial
memory due to buffering intermediate data before writing out each Parquet data block. See
Loading Data into Parquet Tables for instructions about inserting data efficiently into
Although admission control does not scrutinize memory usage for other kinds of DDL statements, if a query is queued due to a limit on concurrent queries or memory usage, subsequent statements in the same session are also queued so that they are processed in the correct order:
-- This query could be queued to avoid out-of-memory at times of heavy load. select * from huge_table join enormous_table using (id); -- If so, this subsequent statement in the same session is also queued -- until the previous statement completes. drop table huge_table;
If you set up different resource pools for different users and groups, consider reusing any classifications you developed for use with Sentry security. See Enabling Sentry Authorization for Impala for details.
For details about all the Fair Scheduler configuration settings, see
Fair Scheduler Configuration, in particular the tags such as
<aclSubmitApps> to map users and groups to particular resource pools (queues).