The SCAN_BYTES_LIMIT query option sets a limit on the bytes scanned by
HDFS and HBase SCAN operations. If a query is still executing when the query’s
coordinator detects that it has exceeded the limit, the query is terminated with an error.
The option is intended to prevent runaway queries that scan more data than is intended.
For example, an Impala administrator could set a default value of
SCAN_BYTES_LIMIT=100GB for a resource pool to automatically kill queries
that scan more than 100 GB of data (see
Impala
Admission Control and Query Queuing for information about default query options).
If a user accidentally omits a partition filter in a WHERE clause and
runs a large query that scans a lot of data, the query will be automatically terminated
after it scans more data than the SCAN_BYTES_LIMIT.
SCAN_BYTES_LIMIT value to
apply to a specific query or session.
Only data actually read from the underlying storage layer is counted towards the
limit. E.g. Impala’s Parquet scanner employs several techniques to skip over
data in a file that is not relevant to a specific query, so often only a fraction
of the file size is counted towards SCAN_BYTES_LIMIT.
As of Impala 3.1, bytes scanned by Kudu tablet servers are not counted towards the limit.
Because the checks are done periodically, the query may scan over the limit at times.
Syntax: SET SCAN_BYTES_LIMIT=bytes;
Type: numeric
m or mb for megabytes.
g or gb for gigabytes.
Default: 0 (no limit)
Added in: Impala 3.1