Using Impala Logging

• By default, the log files are under the directory /var/log/impala. To change log file locations, modify the defaults file described in Starting Impala.
• The significant files for the impalad process are impalad.INFO, impalad.WARNING, and impalad.ERROR. You might also see a file impalad.FATAL, although this is only present in rare conditions.
• The significant files for the statestored process are statestored.INFO, statestored.WARNING, and statestored.ERROR. You might also see a file statestored.FATAL, although this is only present in rare conditions.
• The significant files for the catalogd process are catalogd.INFO, catalogd.WARNING, and catalogd.ERROR. You might also see a file catalogd.FATAL, although this is only present in rare conditions.
• Examine the .INFO files to see configuration settings for the processes.
• Examine the .WARNING files to see all kinds of problem information, including such things as suboptimal settings and also serious runtime errors.
• Examine the .ERROR and/or .FATAL files to see only the most serious errors, if the processes crash, or queries fail to complete. These messages are also in the .WARNING file.
• A new set of log files is produced each time the associated daemon is restarted. These log files have long names including a timestamp. The .INFO, .WARNING, and .ERROR files are physically represented as symbolic links to the latest applicable log files.

Impala stores information using the glog_v logging system. You will see some messages referring to C++ file names. Logging is affected by:

• The GLOG_v environment variable specifies which types of messages are logged. See Setting Logging Levels for details.
• The ‑‑logbuflevel startup flag for the impalad daemon specifies how often the log information is written to disk. The default is 0, meaning that the log is immediately flushed to disk when Impala outputs an important messages such as a warning or an error, but less important messages such as informational ones are buffered in memory rather than being flushed to disk immediately.

Impala periodically switches the physical files representing the current log files, after which it is safe to remove the old files if they are no longer needed.

Impala can automatically remove older unneeded log files, a feature known as log rotation.

Impala checks to see if any old logs need to be removed based on the interval specified in the ‑‑logbufsecs setting, every 5 seconds by default.

For some log levels, Impala logs are first temporarily buffered in memory and only written to disk periodically. The ‑‑logbufsecs setting controls the maximum time that log messages are buffered for. For example, with the default value of 5 seconds, there may be up to a 5 second delay before a logged message shows up in the log file.

It is not recommended that you set ‑‑logbufsecs to 0 as the setting makes the Impala daemon to spin in the thread that tries to delete old log files.

For debugging purposes you may be adjusting the logging configuration for Catalog and impalad servers. This required restarting the services. Impala supports adjusting the log levels dynamically without the need to restart the server. There is a /log_level tab in the debug page of all Impala servers. You can query the log4j log level of root or org.apache.impala by using the Get Java Log Level button. Also you can change the vlog/log4j levels to any supported levels of logging. You can select the log level using the LOG LEVEL drop down box. You also have an option to restore the log levels to their original configuration by using the RESET button.

Here is the format of a Glog:

${level}${month}${day} HH:MM:SS.${us} ${thread-id} ${source-file}:${line}] ${query-id}] ${message}

where

• ${level} — Log Levels; displays the levels as I for INFO, W for WARNING, E for ERROR, F for FATAL.
• ${month}${day} — Month and Date.
• HH:MM:SS — Hours, Minutes, Seconds.
• ${us} — Microseconds.
• ${thread-id} — TID of the thread.
• ${source-file}:${line}] — File name and line number.
• ${query-id}] — An unique id for each and every query that is run in Impala.
• ${message} — Actual log message.

By default, the Impala log is stored at /var/log/impalad/. The most comprehensive log, showing informational, warning, and error messages, is in the file name impalad.INFO. View log file contents by using the web interface or by examining the contents of the log file. (When you examine the logs through the file system, you can troubleshoot problems by reading the impalad.WARNING and/or impalad.ERROR files, which contain the subsets of messages indicating potential problems.)

You can view the contents of the impalad.INFO log file in the file system. With the default configuration settings, the start of the log file appears as follows:

[user@example impalad]$ pwd
/var/log/impalad
[user@example impalad]$ more impalad.INFO
Log file created at: 2013/01/07 08:42:12
Running on machine: impala.example.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0107 08:42:12.292155 14876 daemon.cc:34] impalad version 0.4 RELEASE (build 9d7fadca0461ab40b9e9df8cdb47107ec6b27cff)
Built on Fri, 21 Dec 2012 12:55:19 PST
I0107 08:42:12.292484 14876 daemon.cc:35] Using hostname: impala.example.com
I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug webserver):
--dump_ir=false
--module_output=
--be_port=22000
--classpath=
--hostname=impala.example.com

The logs store information about Impala startup options. This information appears once for each time Impala is started and may include:

• Machine name.
• Impala version number.
• Flags used to start Impala.
• CPU information.
• The number of available disks.

Impala uses the GLOG system, which supports three logging levels. You can adjust logging levels by exporting variable settings. To change logging settings manually, use a command similar to the following on each node before starting impalad:

export GLOG_v=1

For more information on how to configure GLOG, including how to set variable logging levels for different system components, see documentation for the glog project on github.

Log redaction is a security feature that prevents sensitive information from being displayed in locations used by administrators for monitoring and troubleshooting, such as log files and the Impala debug web user interface. You configure regular expressions that match sensitive types of information processed by your system, such as credit card numbers or tax IDs, and literals matching these patterns are obfuscated wherever they would normally be recorded in log files or displayed in administration or debugging user interfaces.

In a security context, the log redaction feature is complementary to the Ranger authorization framework. Ranger prevents unauthorized users from being able to directly access table data. Redaction prevents administrators or support personnel from seeing the smaller amounts of sensitive or personally identifying information (PII) that might appear in queries issued by those authorized users.

See the documentation for your Apache Hadoop distribution for details about how to enable this feature and set up the regular expressions to detect and redact sensitive information within SQL statement text.