Setting Timeout Periods for Daemons, Queries, and Sessions

Depending on how busy your cluster is, you might increase or decrease various timeout values. Increase timeouts if Impala is cancelling operations prematurely, when the system is responding slower than usual but the operations are still successful if given extra time. Decrease timeouts if operations are idle or hanging for long periods, and the idle or hung operations are consuming resources and reducing concurrency.

Increasing the Statestore Timeout

If you have an extensive Impala schema, for example with hundreds of databases, tens of thousands of tables, and so on, you might encounter timeout errors during startup as the Impala catalog service broadcasts metadata to all the Impala nodes using the statestore service. To avoid such timeout errors on startup, increase the statestore timeout value from its default of 10 seconds. Specify the timeout value using the -statestore_subscriber_timeout_seconds option for the statestore service, using the configuration instructions in Modifying Impala Startup Options. The symptom of this problem is messages in the impalad log such as: 

Connection with state-store lost
Trying to re-register with state-store

See Scalability Considerations for the Impala Statestore for more details about statestore operation and settings on clusters with a large number of Impala-related objects such as tables and partitions.

Setting the Idle Query and Idle Session Timeouts for impalad

To keep long-running queries or idle sessions from tying up cluster resources, you can set timeout intervals for both individual queries, and entire sessions.

Note:

The timeout clock for queries and sessions only starts ticking when the query or session is idle.

For queries, this means the query has results ready but is waiting for a client to fetch the data. A query can run for an arbitrary time without triggering a timeout, because the query is computing results rather than sitting idle waiting for the results to be fetched. The timeout period is intended to prevent unclosed queries from consuming resources and taking up slots in the admission count of running queries, potentially preventing other queries from starting.

For sessions, this means that no query has been submitted for some period of time.

Use the following startup options for the impalad daemon to specify timeout values:

  • --idle_query_timeout

    Specifies the time in seconds after which an idle query is cancelled. This could be a query whose results were all fetched but was never closed, or one whose results were partially fetched and then the client program stopped requesting further results. This condition is most likely to occur in a client program using the JDBC or ODBC interfaces, rather than in the interactive impala-shell interpreter. Once a query is cancelled, the client program cannot retrieve any further results from the query.

    You can reduce the idle query timeout by using the QUERY_TIMEOUT_S query option. Any non-zero value specified for the --idle_query_timeout startup option serves as an upper limit for the QUERY_TIMEOUT_S query option. See QUERY_TIMEOUT_S Query Option (Impala 2.0 or higher only) about the query option.

    A zero value for --idle_query_timeout disables query timeouts.

    Cancelled queries are closed, but the client can still fetch their exception details. If the query was in a good state when cancelled, it will present an error like "Query 6f49e509bfa5b347:207d8ef900000000 expired due to client inactivity", otherwise it will show the relevant error.

  • --idle_session_timeout

    Specifies the time in seconds after which an idle session expires. A session is idle when no activity is occurring for any of the queries in that session, and the session has not started any new queries. Once a session is expired, you cannot issue any new query requests to it. The session remains open, but the only operation you can perform is to close it.

    The default value of 0 specifies sessions never expire.

    You can override the --idle_session_timeout value with the IDLE_SESSION_TIMEOUT Query Option (Impala 2.12 or higher only) at the session level.

For instructions on changing impalad startup options, see Modifying Impala Startup Options.

Note:

Impala checks periodically for idle sessions and queries to cancel. The actual idle time before cancellation might be up to 50% greater than the specified configuration setting. For example, if the timeout setting was 60, the session or query might be cancelled after being idle between 60 and 90 seconds.

Setting Timeout and Retries for Thrift Connections to the Backend Client

Impala connections to the backend client are subject to failure in cases when the network is momentarily overloaded. To avoid failed queries due to transient network problems, you can configure the number of Thrift connection retries using the following option:

  • The --backend_client_connection_num_retries option specifies the number of times Impala will try connecting to the backend client after the first connection attempt fails. By default, impalad will attempt three re-connections before it returns a failure.

You can configure timeouts for sending and receiving data from the backend client. Therefore, if for some reason a query hangs, instead of waiting indefinitely for a response, Impala will terminate the connection after a configurable timeout.

  • The --backend_client_rpc_timeout_ms option can be used to specify the number of milliseconds Impala should wait for a response from the backend client before it terminates the connection and signals a failure. The default value for this property is 300000 milliseconds, or 5 minutes.

Cancelling a Query

Occasionally, an Impala query might run for an unexpectedly long time, tying up resources in the cluster. This section describes the options to terminate such runaway queries.

Setting a Time Limit on Query Execution

An Impala administrator can set a default value of the EXEC_TIME_LIMIT_S query option for a resource pool. If a user accidentally runs a large query that executes for longer than the limit, it will be automatically terminated after the time limit expires to free up resources.

You can override the default value per query or per session if you do not want to apply the default EXEC_TIME_LIMIT_S value to a specific query or a session. See EXEC_TIME_LIMIT_S Query Option (Impala 2.12 or higher only) for the details of the query option.

Interactively Cancelling a Query

You can cancel the query explicitly, independent of the timeout period, by going into the web UI for the impalad host (on port 25000 by default), and using the link on the /queries tab to cancel the running query.

Various client applications let you interactively cancel queries submitted or monitored through those applications. For example:
  • Press ^C in impala-shell.
  • Click Cancel from the Watchpage in Hue.