Auditing Impala Operations

To monitor how Impala data is being used within your organization, ensure that your Impala authorization and authentication policies are effective. To detect attempts at intrusion or unauthorized access to Impala data, you can use the auditing feature in Impala 1.2.1 and higher:

  • Enable auditing by including the option ‑‑audit_event_log_dir=directory_path in your impalad startup options. The log directory must be a local directory on the server, not an HDFS directory.
  • Decide how many queries will be represented in each audit event log file. By default, Impala starts a new audit event log file every 5000 queries. To specify a different number, include the option ‑‑max_audit_event_log_file_size=number_of_queries in the impalad startup options.
  • In Impala 2.9 and higher, you can control how many audit event log files are kept on each host. Specify the option ‑‑max_audit_event_log_files=number_of_log_files in the impalad startup options. Once the limit is reached, older files are rotated out using the same mechanism as for other Impala log files. The default value for this setting is 0, representing an unlimited number of audit event log files.
  • Use a cluster manager with governance capabilities to filter, visualize, and produce reports based on the audit logs collected from all the hosts in the cluster.

Durability and Performance Considerations for Impala Auditing

The auditing feature only imposes performance overhead while auditing is enabled.

Because any Impala host can process a query, enable auditing on all hosts where the impalad daemon runs. Each host stores its own log files, in a directory in the local filesystem. The log data is periodically flushed to disk (through an fsync() system call) to avoid loss of audit data in case of a crash.

The runtime overhead of auditing applies to whichever host serves as the coordinator for the query, that is, the host you connect to when you issue the query. This might be the same host for all queries, or different applications or users might connect to and issue queries through different hosts.

To avoid excessive I/O overhead on busy coordinator hosts, Impala syncs the audit log data (using the fsync() system call) periodically rather than after every query. Currently, the fsync() calls are issued at a fixed interval, every 5 seconds.

By default, Impala avoids losing any audit log data in the case of an error during a logging operation (such as a disk full error), by immediately shutting down impalad on the host where the auditing problem occurred. You can override this setting by specifying the option ‑‑abort_on_failed_audit_event=false in the impalad startup options.

Format of the Audit Log Files

The audit log files represent the query information in JSON format, one query per line. Typically, rather than looking at the log files themselves, you should use cluster-management software to consolidate the log data from all Impala hosts and filter and visualize the results in useful ways. (If you do examine the raw log data, you might run the files through a JSON pretty-printer first.)

All the information about schema objects accessed by the query is encoded in a single nested record on the same line. For example, the audit log for an INSERT ... SELECT statement records that a select operation occurs on the source table and an insert operation occurs on the destination table. The audit log for a query against a view records the base table accessed by the view, or multiple base tables in the case of a view that includes a join query. Every Impala operation that corresponds to a SQL statement is recorded in the audit logs, whether the operation succeeds or fails. Impala records more information for a successful operation than for a failed one, because an unauthorized query is stopped immediately, before all the query planning is completed.

The information logged for each query includes:

  • Client session state:
    • Session ID
    • User name
    • Network address of the client connection
  • SQL statement details:
    • Query ID
    • Statement Type - DML, DDL, and so on
    • SQL statement text
    • Execution start time, in local time
    • Execution Status - Details on any errors that were encountered
    • Target Catalog Objects:
      • Object Type - Table, View, or Database
      • Fully qualified object name
      • Privilege - How the object is being used (SELECT, INSERT, CREATE, and so on)

Which Operations Are Audited

The following types of SQL operations are recorded in the audit log:

  • Queries that are prevented due to lack of authorization.
  • Queries that Impala can analyze and parse to determine that they are authorized. The audit data is recorded immediately after Impala finishes its analysis, before the query is actually executed.
  • Queries whose results are available to be fetched by clients.
  • Finished DDL operations.

The audit log does not contain entries for queries that could not be parsed and analyzed. For example, a query that fails due to a syntax error is not recorded in the audit log.

The audit log does not contain queries that fail due to a reference to a table that does not exist.

Certain statements in the impala-shell interpreter, such as CONNECT, SUMMARY, PROFILE, SET, and QUIT, do not correspond to actual SQL queries, and these statements are not recorded in the audit log.