Impala Web User Interface for Debugging

Each of the Impala daemons (impalad, statestored, and catalogd) includes a built-in web server that displays diagnostic and status information.

impalad Web UI

The impalad Web UI includes information about configuration settings, running and completed queries, and associated performance and resource usage for queries. In particular, the Details link for each query displays alternative views of the query including a graphical representation of the plan, and the output of the EXPLAIN, SUMMARY, and PROFILE statements from impala-shell. Each host that runs the impalad daemon has its own instance of the Web UI, with details about those queries for which that host served as the coordinator. The impalad Web UI is primarily used for diagnosing query problems that can be traced to a particular node.

statestored Web UI

The statestored Web UI includes information about memory usage, configuration settings, and ongoing health checks performed by statestored. Because there is only a single instance of the statestored within any Impala cluster, you access the Web UI only on the particular host that serves as the Impala StateStore.

catalogd Web UI

The catalogd Web UI includes information about the databases, tables, and other objects managed by Impala, in addition to the resource usage and configuration settings of the catalogd. Because there is only a single instance of the catalogd within any Impala cluster, you access the Web UI only on the particular host that serves as the Impala Catalog Server.

Configuring Debug Web UIs

Turning off the Web UI

To disable Web UI for the impalad, statestored, or catalogd:

  1. Stop the Impala role you want to turn off the Web UI for, impalad, statestored, or catalogd.
  2. Restart the role with the ‑‑enable_webserver=false startup flag.

Using Kerberos HTTP SPNEGO Authentication for Web UI

To provide security through Kerberos, Impala Web UIs support SPNEGO, a protocol for securing HTTP requests with Kerberos by passing negotiation tokens through HTTP headers.

To enable authorization using SPNEGO, set the --webserver_require_spnego=true startup flag and restart the Impala roles, impalad, statestored, or catalogd.

Debug Web UI for impalad

To debug and troubleshoot an impalad using a web-based interface, open the URL http://impala‑server‑hostname:25000/ in a browser. (For secure clusters, use the prefix https:// instead of http://.)

Because each Impala node produces its own set of debug information, you should choose a specific node that you want to investigate an issue on.

Main Page

The main impalad Web UI page at / lists the following information about the impalad:

  • The version of the impalad daemon

    The Version section also contains other information, such as when Impala was built and what build flags were used.

  • Process start time
  • Hardware information
  • OS information
  • Process information
  • CGroup information

Admission Controller Page

The Admission Controller impalad debug Web UI is at /admission page under the main impalad Web UI.

Use the /admission page to troubleshoot queued queries and the admission control.

The admission page provides the following information about each resource pool to which queries have been submitted at least once:

  • Time since the statestored received the last update
  • A warning if this impalad is considered disconnected from the statestored and thus the information on this page could be stale.
  • Pool configuration
  • Queued queries submitted to this coordinator, in the order of submission
  • Running queries submitted to this coordinator
  • Pool stats
    • Average of time in queue: An exponential moving average which represents the average time in queue over the last 10 to 12 queries. If a query is admitted immediately, the wait time of 0 is used in calculating this average wait time.
  • Histogram of the distribution of peak memory used by queries admitted to the pool

    Use the histogram to figure out settings for the minimum and maximum query MEM_LIMIT ranges for this pool.

    The histogram displays data for all queries admitted to the pool, including the queries that finished, got canceled, or hit an error.

Click on the pool name to only display information relevant to that pool. You can then refresh the debug page to see only the information for that specific pool.

Click Reset informational stats for all pools to reset the stats that keep track of historical data, such as Totals stats, Time in queue (exponential moving average), and the histogram.

The above information is also available as a JSON object from the following HTTP endpoint: 
http://impala-server-hostname:port/admission?json

See Admission Control and Query Queuing for the description of the properties in admission control.

Known Backends Page

The Known backends page of the impalad debug Web UI is at /backends under the main impalad Web UI.

This page lists the following info for each of the impalad nodes in the cluster. Because each impalad daemon knows about every other impalad daemon through the StateStore, this information should be the same regardless of which node you select.

  • Address of the node: Host name and port
  • KRPC address: The KRPC address of the impalad. Use this address when you issue the SHUTDOWN command for this impalad.
  • Whether acting as a coordinator
  • Whether acting as an executor
  • Quiescing status: Specify whether the graceful shutdown process has been initiated on this impalad.
  • Memory limit for admission: The amount of memory that can be admitted to this backend by the admission controller.
  • Memory reserved: The amount of memory reserved by queries that are active, either currently executing or finished but not yet closed, on this backend.

    The memory reserved for a query that is currently executing is its memory limit, if set. Otherwise, if the query has no limit or if the query finished executing, the current consumption is used.

  • Memory of the queries admitted to this coordinator: The memory submitted to this particular host by the queries admitted by this coordinator.

Catalog Page

The Catalog page of the impalad debug Web UI is at /catalog under the main impalad Web UI.

This page displays a list of databases and associated tables recognized by this instance of impalad. You can use this page to locate which database a table is in, check the exact spelling of a database or table name, look for identical table names in multiple databases. The primary debugging use case would be to check if an impalad instance has knowledge of a particular table that someone expects to be in a particular database.

Hadoop Configuration

The Hadoop Configuration page of the impalad debug Web UI is at /hadoop-varz under the main impalad Web UI.

This page displays the Hadoop common configurations that Impala is running with.

JMX

The JMX page of the impalad debug Web UI is at /jmx under the main impalad Web UI.

This page displays monitoring information about various JVM subsystems, such as memory pools, thread management, runtime. etc.

Java Log Level

The Change log level page of the impalad debug Web UI is at /log_level under the main impalad Web UI.

This page displays the current Java and backend log levels, and it allows you to change the log levels dynamically without having to restart the impalad.

Logs Page

The INFO logs page of the impalad debug Web UI is at /logs under the main impalad Web UI.

This page shows the last portion of the impalad.INFO log file, including the info, warning, and error logs for the impalad. You can see the details of the most recent operations, whether the operations succeeded or encountered errors. This page provides one central place for the log files and saves you from looking around the filesystem for the log files, which could be in different locations on clusters that use cluster management software.

Memz Page

The Memory Usage page of the impalad debug Web UI is at /memz under the main impalad Web UI.

This page displays the summary and detailed information about memory usage by the impalad.

Metrics Page

The Metrics page of the impalad debug Web UI is at /metrics under the main impalad Web UI.

This page displays the current set of metrics, counters and flags representing various aspects of impalad internal operations.

Queries Page

The Queries page of the impalad debug Web UI is at /queries under the main impalad Web UI.

This page lists:
  • Currently running queries
  • Queries that have completed their execution, but have not been closed yet
  • Completed queries whose details still reside in memory
The queries are listed in reverse chronological order, with the most recent at the top. You can control the amount of memory devoted to completed queries by specifying the -‑-‑query_log_size and -‑-‑query_log_size_in_bytes startup options for impalad.
This page provides:
  • How many SQL statements are failing (State value of EXCEPTION)
  • How large the result sets are (# rows fetched)
  • How long each statement took (Start Time and End Time)

Click the Details link for a query to display the detailed performance characteristics of that query, such as the profile output.

On the query detail page, in the Profile tab, you have options to export the query profile output to the Thrift, text, or Json format.

The Queries page also includes the Query Locations section that lists the number of running queries with fragments on this host.

RPC Services Page

The RPC durations page of the impalad debug Web UI is at /rpcz under the main impalad Web UI.

This page displays information, such as the duration, about the RPC communications of this impalad with other Impala daemons.

Sessions Page

The Sessions page of the impalad debug Web UI is at /session under the main impalad Web UI.

This page displays information about the sessions currently connected to this impalad instance. For example, sessions could include connections from the impala-shell command, JDBC or ODBC applications, or the Impala Query UI in the Hue web interface.

Threadz Page

The Threads page of the impalad debug Web UI is at /threadz under the main impalad Web UI.

This page displays information about the threads used by this instance of impalad, and it shows which categories they are grouped into. Making use of this information requires substantial knowledge about Impala internals.

Varz Page

The Varz page of the impalad debug Web UI is at /varz under the main impalad Web UI.

This page shows the configuration settings in effect when this instance of impalad communicates with other Hadoop components such as HDFS and YARN. These settings are collected from a set of configuration files.

The bottom of this page also lists all the command-line settings in effect for this instance of impalad. See Modifying Impala Startup Options for information about modifying these values.

Prometheus Metrics Page

At /metrics_prometheus under the main impalad Web UI, the metrics are generated in Prometheus exposition format that Prometheus can consume for event monitoring and alerting. The /metrics_prometheus is not shown in the Web UI list of pages.

Debug Web UI for statestored

To debug and troubleshoot the statestored daemon using a web-based interface, open the URL http://impala‑server‑hostname:25010/ in a browser. (For secure clusters, use the prefix https:// instead of http://.)

Main Page

The main statestored Web UI page at / lists the following information about the statestored:

  • The version of the statestored daemon
  • Process start time
  • Hardware information
  • OS information
  • Process information
  • CGroup information

Logs Page

The INFO logs page of the debug Web UI is at /logs under the main statestored Web UI.

This page shows the last portion of the statestored.INFO log file, including the info, warning, and error logs for the statestored. You can refer here to see the details of the most recent operations, whether the operations succeeded or encountered errors. This page provides one central place for the log files and saves you from looking around the filesystem for the log files, which could be in different locations on clusters that use cluster management software.

Memz Page

The Memory Usage page of the debug Web UI is at /memz under the main statestored Web UI.

This page displays summary and detailed information about memory usage by the statestored. You can see the memory limit in effect for the node, and how much of that memory Impala is currently using.

Metrics Page

The Metrics page of the debug Web UI is at /metrics under the main statestored Web UI.

This page displays the current set of metrics: counters and flags representing various aspects of statestored internal operation.

RPC Services Page

The RPC durations page of the statestored debug Web UI is at /rpcz under the main statestored Web UI.

This page displays information, such as the durations, about the RPC communications of this statestored with other Impala daemons.

Subscribers Page

The Subscribers page of the debug Web UI is at /subscribers under the main statestored Web UI.

This page displays information about the other Impala daemons that have registered with the statestored to receive and send updates.

Threadz Page

The Threads page of the debug Web UI is at /threadz under the main statestored Web UI.

This page displays information about the threads used by this instance of statestored, and shows which categories they are grouped into. Making use of this information requires substantial knowledge about Impala internals.

Topics Page

The Topics page of the debug Web UI is at /topics under the main statestored Web UI.

This page displays information about the topics to which the other Impala daemons have registered to receive updates.

Varz Page

The Varz page of the debug Web UI is at /varz under the main statestored Web UI.

This page shows the configuration settings in effect when this instance of statestored communicates with other Hadoop components such as HDFS and YARN. These settings are collected from a set of configuration files.

The bottom of this page also lists all the command-line settings in effect for this instance of statestored. See Modifying Impala Startup Options for information about modifying these values.

Prometheus Metrics Page

At /metrics_prometheus under the main statestored Web UI, the metrics are generated in Prometheus exposition format that Prometheus can consume for event monitoring and alerting. The /metrics_prometheus is not shown in the Web UI list of pages.

Debug Web UI for catalogd

The main page of the debug Web UI is at http://impala‑server‑hostname:25020/ (non-secure cluster) or https://impala‑server‑hostname:25020/ (secure cluster).

Main Page

The main catalogd Web UI page at / lists the following information about the catalogd:

  • The version of the catalogd daemon
  • Process start time
  • Hardware information
  • OS information
  • Process information
  • CGroup information

Catalog Page

The Catalog page of the debug Web UI is at /catalog under the main catalogd Web UI.

This page displays a list of databases and associated tables recognized by this instance of catalogd. You can use this page to locate which database a table is in, check the exact spelling of a database or table name, look for identical table names in multiple databases. The catalog information is represented as the underlying Thrift data structures.

JMX

The JMX page of the catalogd debug Web UI is at /jmx under the main catalogd Web UI.

This page displays monitoring information about various JVM subsystems, such as memory pools, thread management, runtime. etc.

Java Log Level

The Change log level page of the catalogd debug Web UI is at /log_level under the main catalogd Web UI.

The page displays the current Java and backend log levels and allows you to change the log levels dynamically without having to restart the catalogd

Logs Page

The INFO logs page of the debug Web UI is at /logs under the main catalogd Web UI.

This page shows the last portion of the catalogd.INFO log file, including the info, warning, and error logs for the catalogd daemon. You can refer here to see the details of the most recent operations, whether the operations succeeded or encountered errors. This page provides one central place for the log files and saves you from looking around the filesystem for the log files, which could be in different locations on clusters that use cluster management software.

Memz Page

The Memory Usage page of the debug Web UI is at /memz under the main catalogd Web UI.

This page displays summary and detailed information about memory usage by the catalogd. You can see the memory limit in effect for the node, and how much of that memory Impala is currently using.

Metrics Page

The Metrics page of the debug Web UI is at /metrics under the main catalogd Web UI.

This page displays the current set of metrics: counters and flags representing various aspects of catalogd internal operation.

RPC Services Page

The RPC durations page of the catalogd debug Web UI is at /rpcz under the main catalogd Web UI.

This page displays information, such as the durations, about the RPC communications of this catalogd with other Impala daemons.

Threadz Page

The Threads page of the debug Web UI is at /threadz under the main catalogd Web UI.

This page displays information about the threads used by this instance of catalogd, and shows which categories they are grouped into. Making use of this information requires substantial knowledge about Impala internals.

Varz Page

The Varz page of the debug Web UI is at /varz under the main catalogd Web UI.

This page shows the configuration settings in effect when this instance of catalogd communicates with other Hadoop components such as HDFS and YARN. These settings are collected from a set of configuration files.

The bottom of this page also lists all the command-line settings in effect for this instance of catalogd. See Modifying Impala Startup Options for information about modifying these values.

Prometheus Metrics Page

At /metrics_prometheus under the main catalogd Web UI, the metrics are generated in Prometheus exposition format that Prometheus can consume for event monitoring and alerting. The /metrics_prometheus is not shown in the Web UI list of pages.