PARQUET_READ_STATISTICS Query Option (Impala 2.9 or higher only)
The PARQUET_READ_STATISTICS
query option controls whether to read
statistics from Parquet files and use them during query processing.
Parquet stores min/max stats which can be used to skip reading row groups if they don't
qualify a certain predicate. When this query option is set to true
,
Impala reads the Parquet statistics and skips reading row groups that do not match the
conditions in the WHERE
clause.
Impala supports filtering based on Parquet statistics:
- Of the numerical types for the old version of the statistics: Boolean, Integer, Float
- Of the types for the new version of the statistics (starting in IMPALA 2.8): Boolean, Integer, Float, Decimal, String, Timestamp
-
For simple predicates of the forms:
<slot> <op> <constant>
or<constant> <op> <slot>
, where<op>
is LT, LE, GE, GT, and EQ
The PARQUET_READ_STATISTICS
option provides a workaround when dealing
with files that have corrupt Parquet statistics and unknown errors.
In the query runtime profile output for each Impalad instance, the
NumStatsFilteredRowGroups
field in the SCAN node section shows the number
of row groups that were skipped based on Parquet statistics.
-
true
(1
): Read statistics from Parquet files and use them in query processing. -
false
(0
): Do not use Parquet read statistics. -
Any other values are treated as
false
.
Type: Boolean
Default: true
Added in: Impala 2.9.0