SCHEDULE_RANDOM_REPLICA Query Option (Impala 2.5 or higher only)
SCHEDULE_RANDOM_REPLICA query option fine-tunes the algorithm for deciding which host
processes each HDFS data block. It only applies to tables and partitions that are not enabled
for the HDFS caching feature.
Type: Boolean; recognized values are 1 and 0, or
any other value interpreted as
Added in: Impala 2.5.0
In the presence of HDFS cached replicas, Impala randomizes
which host processes each cached data block.
To ensure that HDFS data blocks are cached on more
than one host, use the
WITH REPLICATION clause along with
CACHED IN clause in a
CREATE TABLE or
ALTER TABLE statement.
Specify a replication value greater than or equal to the HDFS block replication factor.
SCHEDULE_RANDOM_REPLICA query option applies to tables and partitions
that do not use HDFS caching.
By default, Impala estimates how much work each host has done for
the query, and selects the host that has the lowest workload.
This algorithm is intended to reduce CPU hotspots arising when the
same host is selected to process multiple data blocks, but hotspots
might still arise for some combinations of queries and data layout.
SCHEDULE_RANDOM_REPLICA option is enabled,
Impala further randomizes the scheduling algorithm for non-HDFS cached blocks,
which can further reduce the chance of CPU hotspots.
This query option works in conjunction with the work scheduling improvements
in Impala 2.5 and higher. The scheduling improvements
distribute the processing for cached HDFS data blocks to minimize hotspots:
if a data block is cached on more than one host, Impala chooses which host
to process each block based on which host has read the fewest bytes during
the current query. Enable
SCHEDULE_RANDOM_REPLICA setting if CPU hotspots
still persist because of cases where hosts are "tied" in terms of
the amount of work done; by default, Impala picks the first eligible host
in this case.