Specifies the default size for a memory buffer used when the spill-to-disk mechanism is activated, for example for queries against a large table with no statistics, or large join operations.
Type: integer
Default:
2097152 (2 MB)
Units: A numeric argument represents a size in bytes; you can also use a suffix
of m or mb for megabytes, or g or
gb for gigabytes. If you specify a value with unrecognized formats,
subsequent queries fail with an error.
Added in: Impala 2.10.0
Usage notes:
This query option sets an upper bound on the size of the internal buffer size that can be used during spill-to-disk operations. The actual size of the buffer is chosen by the query planner.
If overall query performance is limited by the time needed for spilling,
consider increasing the DEFAULT_SPILLABLE_BUFFER_SIZE setting.
Larger buffer sizes result in Impala issuing larger I/O requests to storage
devices, which might result in higher throughput, particularly on rotational
disks.
The tradeoff with a large value for this setting is increased memory usage during spill-to-disk operations. Reducing this value may reduce memory consumption.
To determine if the value for this setting is having an effect by capping the
spillable buffer size, you can see the buffer size chosen by the query planner for
a particular query. EXPLAIN the query while the setting
EXPLAIN_LEVEL=2 is in effect.
Examples:
set default_spillable_buffer_size=4MB;
Related information:
BUFFER_POOL_LIMIT Query Option, MAX_ROW_SIZE Query Option, MIN_SPILLABLE_BUFFER_SIZE Query Option, Scalability Considerations for Impala