MAX_NUM_RUNTIME_FILTERS Query Option (Impala 2.5 or higher only)
MAX_NUM_RUNTIME_FILTERS query option
sets an upper limit on the number of runtime filters that can be produced for each query.
Added in: Impala 2.5.0
Each runtime filter imposes some memory overhead on the query.
Depending on the setting of the
query option, each filter might consume between 1 and 16 megabytes
per plan fragment. There are typically 5 or fewer filters per plan fragment.
Impala evaluates the effectiveness of each filter, and keeps the ones that eliminate the largest number of partitions or rows. Therefore, this setting can protect against potential problems due to excessive memory overhead for filter production, while still allowing a high level of optimization for suitable queries.
Because the runtime filtering feature applies mainly to resource-intensive and long-running queries, only adjust this query option when tuning long-running queries involving some combination of large partitioned tables and joins involving large tables.