PARQUET_DICTIONARY_FILTERING Query Option (Impala 2.9 or higher only)
PARQUET_DICTIONARY_FILTERING query option controls whether Impala
uses dictionary filtering for Parquet files.
To efficiently process a highly selective scan query, when this option is enabled, Impala checks the values in the Parquet dictionary page and determines if the whole row group can be thrown out.
encoding_statsis in the Parquet file, dictionary filtering uses it to determine if there are only dictionary encoded pages (i.e. there are no data pages with an encoding other than
If the encoding stats are not present, dictionary filtering looks at the encodings.
The column is purely dictionary encoded if both of the conditions satisfy:
BIT_PACKEDencodings are listed.
- Dictionary filtering works for the Parquet dictionaries with less than 40000 values if the file was written by or lower.
In the query runtime profile output for each Impalad instance, the
NumDictFilteredRowGroups field in the SCAN node section shows the number
of row groups that were skipped based on dictionary filtering.
Note that row groups can be filtered out by Parquet statistics, and in such cases, dictionary filtering will not be considered.
1): Use dictionary filtering.
0): Do not use dictionary filtering
Any other values are treated as
Added in: Impala 2.9.0