Troubleshooting Impala

Troubleshooting Impala SQL Syntax Issues

In general, if queries issued against Impala fail, you can try running these same queries against Hive.

If a query fails against both Impala and Hive, it is likely that there is a problem with your query or other elements of your environment:
- Review the Language Reference to ensure your query is valid.
- Check Impala Reserved Words to see if any database, table, column, or other object names in your query conflict with Impala reserved words. Quote those names with backticks (``) if so.
- Check Impala Built-In Functions to confirm whether Impala supports all the built-in functions being used by your query, and whether argument and return types are the same as you expect.
- Review the contents of the Impala logs for any information that may be useful in identifying the source of the problem.
If a query fails against Impala but not Hive, it is likely that there is a problem with your Impala installation.

Troubleshooting Crashes Caused by Memory Resource Limit

Under very high concurrency, Impala could encounter a serious error due to usage of various operating system resources. Errors similar to the following may be caused by operating system resource exhaustion:

F0629 08:20:02.956413 29088 llvm-codegen.cc:111] LLVM hit fatal error: Unable to allocate section memory!
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::thread_resource_error> >'

The KRPC implementation in Impala 2.12 / 3.0 greatly reduces thread counts and the chances of hitting a resource limit.

If you still get an error similar to the above in Impala 3.0 and higher, try increasing the max_map_count OS virtual memory parameter. max_map_count defines the maximum number of memory map areas that a process can use. Configure each host running an impalad daemon with the command to increase max_map_count to 8 GB.

echo 8000000 > /proc/sys/vm/max_map_count

To make the above settings durable, refer to your OS documentation. For example, on RHEL 6.x:

Add the following line to /etc/sysctl.conf:
```
vm.max_map_count=8000000
```
Run the following command:
```
sysctl -p
```

Troubleshooting I/O Capacity Problems

Impala queries are typically I/O-intensive. If there is an I/O problem with storage devices, or with HDFS itself, Impala queries could show slow response times with no obvious cause on the Impala side. Slow I/O on even a single Impala daemon could result in an overall slowdown, because queries involving clauses such as ORDER BY, GROUP BY, or JOIN do not start returning results until all executor Impala daemons have finished their work.

To test whether the Linux I/O system itself is performing as expected, run Linux commands like the following on each host Impala daemon is running:


$ sudo sysctl -w vm.drop_caches=3 vm.drop_caches=0
vm.drop_caches = 3
vm.drop_caches = 0
$ sudo dd if=/dev/sda bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.60373 s, 192 MB/s
$ sudo dd if=/dev/sdb bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.51145 s, 195 MB/s
$ sudo dd if=/dev/sdc bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.58096 s, 192 MB/s
$ sudo dd if=/dev/sdd bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.43924 s, 197 MB/s

On modern hardware, a throughput rate of less than 100 MB/s typically indicates a performance issue with the storage device. Correct the hardware problem before continuing with Impala tuning or benchmarking.

Impala Troubleshooting Quick Reference

The following table lists common problems and potential solutions.


Symptom	Explanation	Recommendation
Impala takes a long time to start.	Impala instances with large numbers of tables, partitions, or data files take longer to start because the metadata for these objects is broadcast to all impalad nodes and cached.	Adjust timeout and synchronicity settings.
Joins fail to complete.	There may be insufficient memory. During a join, data from the second, third, and so on sets to be joined is loaded into memory. If Impala chooses an inefficient join order or join mechanism, the query could exceed the total memory available.	Start by gathering statistics with the `COMPUTE STATS` statement for each table involved in the join. Consider specifying the `[SHUFFLE]` hint so that data from the joined tables is split up between nodes rather than broadcast to each node. If tuning at the SQL level is not sufficient, add more memory to your system or join smaller data sets.
Queries return incorrect results.	Impala metadata may be outdated after changes are performed in Hive.	Where possible, use the appropriate Impala statement (`INSERT`, `LOAD DATA`, `CREATE TABLE`, `ALTER TABLE`, `COMPUTE STATS`, and so on) rather than switching back and forth between Impala and Hive. Impala automatically broadcasts the results of DDL and DML operations to all Impala nodes in the cluster, but does not automatically recognize when such changes are made through Hive. After inserting data, adding a partition, or other operation in Hive, refresh the metadata for the table as described in REFRESH Statement.
Queries are slow to return results.	Some `impalad` instances may not have started. Using a browser, connect to the host running the Impala state store. Connect using an address of the form `http://hostname:port/metrics`. Note: Replace `hostname` and `port` with the hostname and port of your Impala state store host machine and web server port. The default port is 25010. The number of `impalad` instances listed should match the expected number of `impalad` instances installed in the cluster. There should also be one `impalad` instance installed on each DataNode.	Ensure Impala is installed on all DataNodes. Start any `impalad` instances that are not running.
Queries are slow to return results.	Impala may not be configured to use native checksumming. Native checksumming uses machine-specific instructions to compute checksums over HDFS data very quickly. Review Impala logs. If you find instances of "`INFO util.NativeCodeLoader: Loaded the native-hadoop`" messages, native checksumming is not enabled.	Ensure Impala is configured to use native checksumming as described in Post-Installation Configuration for Impala.
Queries are slow to return results.	Impala may not be configured to use data locality tracking.	Test Impala for data locality tracking and make configuration changes as necessary. Information on this process can be found in Post-Installation Configuration for Impala.
Attempts to complete Impala tasks such as executing INSERT-SELECT actions fail. The Impala logs include notes that files could not be opened due to permission denied.	This can be the result of permissions issues. For example, you could use the Hive shell as the hive user to create a table. After creating this table, you could attempt to complete some action, such as an INSERT-SELECT on the table. Because the table was created using one user and the INSERT-SELECT is attempted by another, this action may fail due to permissions issues.	In general, ensure the Impala user has sufficient permissions. In the preceding example, ensure the Impala user has sufficient permissions to the table that the Hive user created.
Impala fails to start up, with the impalad logs referring to errors connecting to the statestore service and attempts to re-register.	A large number of databases, tables, partitions, and so on can require metadata synchronization, particularly on startup, that takes longer than the default timeout for the statestore service.	Configure the statestore timeout value and possibly other settings related to the frequency of statestore updates and metadata loading. See Increasing the Statestore Timeout and Scalability Considerations for the Impala Statestore.