Upgrading Impala

Upgrading Impala involves building or acquiring new Impala-related binaries, and then restarting Impala services.

Upgrading Impala

Note:

If the services did not start successfully (even though the sudo service command might display [OK]), check for errors in the Impala log file, typically in /var/log/impala.

Impala Upgrade Considerations

Grant REFRESH Privilege to Impala Roles with SELECT or INSERT Privilege when Upgrading to Impala 3.0

To use the fine grained privileges feature in Impala 3.0, if a role has the SELECT or INSERT privilege on an object in Impala before upgrading to Impala 3.0, grant that role the REFRESH privilege after the upgrade.

List of Reserved Words Updated in Impala 3.0

The list of reserved words in Impala was updated in Impala 3.0. If you need to use a reserved word as an identifier, e.g. a table name, enclose the word in back-ticks.

If you need to use the reserved words from previous versions of Impala, set the impalad and catalogd startup flag. Note that this startup option will be deprecated in a future release.
--reserved_words_version=2.11.0

Decimal V2 Used by Default in Impala 3.0

In Impala, two different implementations of DECIMAL types are supported. Starting in Impala 3.0, DECIMAL V2 is used by default. See DECIMAL Type for detail information.

If you need to continue using the first version of the DECIMAL type for the backward compatibility of your queries, set the DECIMAL_V2 query option to FALSE:
SET DECIMAL_V2=FALSE;

Behavior of Column Aliases Changed in Impala 3.0

To conform to the SQL standard, Impala no longer performs alias substitution in the subexpressions of GROUP BY, HAVING, and ORDER BY. See Overview of Impala Aliases for examples of supported and unsupported aliases syntax.

Default PARQUET_ARRAY_RESOLUTION Changed in Impala 3.0

The default value for the PARQUET_ARRAY_RESOLUTION was changed to THREE_LEVEL in Impala 3.0, to match the Parquet standard 3-level encoding.

See PARQUET_ARRAY_RESOLUTION Query Option (Impala 2.9 or higher only) for the information about the query option.

Enable Clustering Hint for Inserts

In Impala 3.0, the clustered hint is enabled by default. The hint adds a local sort by the partitioning columns to a query plan.

The clustered hint is only effective for HDFS and Kudu tables.

As in previous versions, the noclustered hint prevents clustering. If a table has ordering columns defined, the noclustered hint is ignored with a warning.

Deprecated Query Options Removed in Impala 3.0

The following query options have been deprecated for several releases and removed:
  • DEFAULT_ORDER_BY_LIMIT
  • ABORT_ON_DEFAULT_LIMIT_EXCEEDED
  • V_CPU_CORES
  • RESERVATION_REQUEST_TIMEOUT
  • RM_INITIAL_MEM
  • SCAN_NODE_CODEGEN_THRESHOLD
  • MAX_IO_BUFFERS
  • RM_INITIAL_MEM
  • DISABLE_CACHED_READS

Fine-grained Privileges Added in Impala 3.0

Starting in Impala 3.0, finer grained privileges are enforced, such as the REFRESH, CREATE, DROP, and ALTER privileges. In particular, running REFRESH or INVALIDATE METADATA now requires the new REFRESH privilege. Users who did not previously have the ALL privilege will no longer be able to run REFRESH or INVALIDATE METADATA after an upgrade. Those users need to have the REFRESH or ALL privilege granted to run REFRESH or INVALIDATE METADATA.

See GRANT Statement (Impala 2.0 or higher only) for the new privileges, the scope, and other information about the new privileges.

refresh_after_connect Impala Shell Option Removed in Impala 3.0

The deprecated refresh_after_connect option was removed from Impala Shell in Impala 3.0

Return Type Changed for EXTRACT and DATE_PART Functions in Impala 3.0

The following changes were made to the EXTRACT and DATE_PART functions:
  • The output type of the EXTRACT and DATE_PART functions was changed to BIGINT.
  • Extracting the millisecond part from a TIMESTAMP returns the seconds component and the milliseconds component. For example, EXTRACT (CAST('2006-05-12 18:27:28.123456789' AS TIMESTAMP), 'MILLISECOND') will return 28123.

Default Setting Changes

Release Changed Setting Default Value
Impala 2.12 compact_catalog_topicimpalad flag true
Impala 2.12 max_cached_file_handlesimpalad flag 20000
Impala 3.0 PARQUET_ARRAY_RESOLUTION query option THREE_LEVEL