Breakpad Minidumps for Impala (Impala 2.6 or higher only)
project is an open-source framework for crash reporting.
In Impala 2.6 and higher, Impala can use
breakpad to record stack information and
register values when any of the Impala-related daemons crash due to an error such as
or unhandled exceptions.
The dump files are much smaller than traditional core dump files. The dump mechanism itself uses very little
memory, which improves reliability if the crash occurs while the system is low on memory.
Enabling or Disabling Minidump Generation
By default, a minidump file is generated when an Impala-related daemon crashes. To turn off generation of the minidump files, change the minidump_path configuration setting of one or more Impala-related daemons to the empty string, and restart the corresponding services or daemons.
In Impala 2.7 and higher,
you can send a
SIGUSR1 signal to any Impala-related daemon to write a
Breakpad minidump. For advanced troubleshooting, you can now produce a minidump
without triggering a crash.
Specifying the Location for Minidump Files
Clusters not managed by cluster management software: impala_log_dir/daemon_name/minidumps/daemon_name
To specify a different location, set the minidump_path configuration setting of one or more Impala-related daemons, and restart the corresponding services or daemons.
If you specify a relative path for this setting, the value is interpreted relative to the default minidump_path directory.
Controlling the Number of Minidump Files
Like any files used for logging or troubleshooting, consider limiting the number of minidump files, or removing unneeded ones, depending on the amount of free storage space on the hosts in the cluster.
Because the minidump files are only used for problem resolution, you can remove any such files that are not needed to debug current issues.
To control how many minidump files Impala keeps around at any one time, set the max_minidumps configuration setting for of one or more Impala-related daemon, and restart the corresponding services or daemons. The default for this setting is 9. A zero or negative value is interpreted as "unlimited".
Detecting Crash Events
You can see in the Impala log files when crash events occur that generate minidump files. Because each restart begins a new log file, the "crashed" message is always at or near the bottom of the log file. There might be another later message if core dumps are also enabled.
Demonstration of Breakpad Feature
The following example uses the command kill -11 to
SIGSEGV crash for an impalad
process on a single DataNode, then examines the relevant log files and minidump file.
First, as root on a worker node, kill the impalad process with a
SIGSEGV error. The original process ID was 23114.
# ps ax | grep impalad 23114 ? Sl 0:18 /opt/local/parcels/<parcel_version>/lib/impala/sbin/impalad --flagfile=/var/run/impala/process/114-impala-IMPALAD/impala-conf/impalad_flags 31259 pts/0 S+ 0:00 grep impalad # # kill -11 23114 # # ps ax | grep impalad 31374 ? Rl 0:04 /opt/local/parcels/<parcel_version>/lib/impala/sbin/impalad --flagfile=/var/run/impala/process/114-impala-IMPALAD/impala-conf/impalad_flags 31475 pts/0 S+ 0:00 grep impalad
We locate the log directory underneath /var/log.
There is a
log file for the 23114 process ID. The minidump message is written to the
.INFO file and the
.ERROR file, but not the
.WARNING file. In this case, a large core file was also produced.
# cd /var/log/impalad # ls -la | grep 23114 -rw------- 1 impala impala 3539079168 Jun 23 15:20 core.23114 -rw-r--r-- 1 impala impala 99057 Jun 23 15:20 hs_err_pid23114.log -rw-r--r-- 1 impala impala 351 Jun 23 15:20 impalad.worker_node_123.impala.log.ERROR.20160623-140343.23114 -rw-r--r-- 1 impala impala 29101 Jun 23 15:20 impalad.worker_node_123.impala.log.INFO.20160623-140343.23114 -rw-r--r-- 1 impala impala 228 Jun 23 14:03 impalad.worker_node_123.impala.log.WARNING.20160623-140343.23114
.INFO log includes the location of the minidump file, followed by
a report of a core dump. With the breakpad minidump feature enabled, now we might
disable core dumps or keep fewer of them around.
# cat impalad.worker_node_123.impala.log.INFO.20160623-140343.23114 ... Wrote minidump to /var/log/impala-minidumps/impalad/0980da2d-a905-01e1-25ff883a-04ee027a.dmp # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00000030c0e0b68a, pid=23114, tid=139869541455968 # # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libpthread.so.0+0xb68a] pthread_cond_wait+0xca # # Core dump written. Default location: /var/log/impalad/core or core.23114 # # An error report file with more information is saved as: # /var/log/impalad/hs_err_pid23114.log # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. ... # cat impalad.worker_node_123.impala.log.ERROR.20160623-140343.23114 Log file created at: 2016/06/23 14:03:43 Running on machine:.worker_node_123 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg E0623 14:03:43.911002 23114 logging.cc:118] stderr will be logged to this file. Wrote minidump to /var/log/impala-minidumps/impalad/0980da2d-a905-01e1-25ff883a-04ee027a.dmp
The resulting minidump file is much smaller than the corresponding core file, making it much easier to supply diagnostic information to the appropriate support channel.
# pwd /var/log/impalad # cd ../impala-minidumps/impalad # ls 0980da2d-a905-01e1-25ff883a-04ee027a.dmp # du -kh * 2.4M 0980da2d-a905-01e1-25ff883a-04ee027a.dmp