PARQUET_READ_STATISTICS Query Option (Impala 2.9 or higher only)
PARQUET_READ_STATISTICS query option controls whether to read
statistics from Parquet files and use them during query processing.
Parquet stores min/max stats which can be used to skip reading row groups if they don't
qualify a certain predicate. When this query option is set to
Impala reads the Parquet statistics and skips reading row groups that do not match the
conditions in the
Impala supports filtering based on Parquet statistics:
- Of the numerical types for the old version of the statistics: Boolean, Integer, Float
- Of the types for the new version of the statistics (starting in IMPALA 2.8): Boolean, Integer, Float, Decimal, String, Timestamp
For simple predicates of the forms:
<slot> <op> <constant>or
<constant> <op> <slot>, where
<op>is LT, LE, GE, GT, and EQ
PARQUET_READ_STATISTICS option provides a workaround when dealing
with files that have corrupt Parquet statistics and unknown errors.
In the query runtime profile output for each Impalad instance, the
NumStatsFilteredRowGroups field in the SCAN node section shows the number
of row groups that were skipped based on Parquet statistics.
1): Read statistics from Parquet files and use them in query processing.
0): Do not use Parquet read statistics.
Any other values are treated as
Added in: Impala 2.9.0