Auditing Impala Operations
To monitor how Impala data is being used within your organization, ensure that your Impala authorization and authentication policies are effective. To detect attempts at intrusion or unauthorized access to Impala data, you can use the auditing feature in Impala 1.2.1 and higher:
Enable auditing by including the option
-audit_event_log_dir=directory_pathin your impalad startup options. The log directory must be a local directory on the server, not an HDFS directory.
Decide how many queries will be represented in each audit event log file. By default,
Impala starts a new audit event log file every 5000 queries. To specify a different number,
--max_audit_event_log_file_size=number_of_queriesin the impalad startup options.
In Impala 2.9 and higher, you can control how many
audit event log files are kept on each host. Specify the option
--max_audit_event_log_files=number_of_log_filesin the impalad startup options. Once the limit is reached, older files are rotated out using the same mechanism as for other Impala log files. The default value for this setting is 0, representing an unlimited number of audit event log files.
- Use a cluster manager with governance capabilities to filter, visualize, and produce reports based on the audit logs collected from all the hosts in the cluster.
Durability and Performance Considerations for Impala Auditing
The auditing feature only imposes performance overhead while auditing is enabled.
Because any Impala host can process a query, enable auditing on all hosts where the
runs. Each host stores its own log
files, in a directory in the local filesystem. The log data is periodically flushed to disk (through an
fsync() system call) to avoid loss of audit data in case of a crash.
The runtime overhead of auditing applies to whichever host serves as the coordinator for the query, that is, the host you connect to when you issue the query. This might be the same host for all queries, or different applications or users might connect to and issue queries through different hosts.
To avoid excessive I/O overhead on busy coordinator hosts, Impala syncs the audit log
data (using the
fsync() system call) periodically rather than after
every query. Currently, the
fsync() calls are issued at a fixed
interval, every 5 seconds.
By default, Impala avoids losing any audit log data in the case of an error during a logging operation
(such as a disk full error), by immediately shutting down
impalad on the host where the auditing problem occurred.
You can override this setting by specifying the option
-abort_on_failed_audit_event=false in the impalad startup options.
Format of the Audit Log Files
The audit log files represent the query information in JSON format, one query per line. Typically, rather than looking at the log files themselves, you should use cluster-management software to consolidate the log data from all Impala hosts and filter and visualize the results in useful ways. (If you do examine the raw log data, you might run the files through a JSON pretty-printer first.)
All the information about schema objects accessed by the query is encoded in a single nested record on the
same line. For example, the audit log for an
INSERT ... SELECT statement records that a
select operation occurs on the source table and an insert operation occurs on the destination table. The
audit log for a query against a view records the base table accessed by the view, or multiple base tables
in the case of a view that includes a join query. Every Impala operation that corresponds to a SQL
statement is recorded in the audit logs, whether the operation succeeds or fails. Impala records more
information for a successful operation than for a failed one, because an unauthorized query is stopped
immediately, before all the query planning is completed.
The information logged for each query includes:
Client session state:
- Session ID
- User name
- Network address of the client connection
SQL statement details:
- Query ID
- Statement Type - DML, DDL, and so on
- SQL statement text
- Execution start time, in local time
- Execution Status - Details on any errors that were encountered
Target Catalog Objects:
- Object Type - Table, View, or Database
- Fully qualified object name
Privilege - How the object is being used (
CREATE, and so on)
Which Operations Are Audited
The kinds of SQL queries represented in the audit log are:
- Queries that are prevented due to lack of authorization.
- Queries that Impala can analyze and parse to determine that they are authorized. The audit data is recorded immediately after Impala finishes its analysis, before the query is actually executed.
The audit log does not contain entries for queries that could not be parsed and analyzed. For example, a query that fails due to a syntax error is not recorded in the audit log. The audit log also does not contain queries that fail due to a reference to a table that does not exist, if you would be authorized to access the table if it did exist.
Certain statements in the impala-shell interpreter, such as
QUIT, do not correspond to actual SQL queries, and these statements are not reflected in
the audit log.