Collaboration diagram for com.cloudera.impala.util.RequestPoolService:

Classes
class	LlamaConfWatcher

Public Member Functions
	RequestPoolService (final String fsAllocationPath, final String llamaSitePath)

void	start ()

void	stop ()

byte[]	resolveRequestPool (byte[] thriftResolvePoolParams) throws ImpalaException

byte[]	getPoolConfig (byte[] thriftPoolConfigParams) throws ImpalaException

Package Functions
TResolveRequestPoolResult	resolveRequestPool (TResolveRequestPoolParams resolvePoolParams)

TPoolConfigResult	getPoolConfig (String pool)

String	assignToPool (String requestedPool, String user) throws IOException

boolean	hasAccess (String pool, String user)

Static Package Functions
static URL	getURL (String path)

Package Attributes
final AllocationFileLoaderService	allocLoader_

final FileWatchService	llamaConfWatcher_

Static Package Attributes
static final Logger	LOG = LoggerFactory.getLogger(RequestPoolService.class)

static final String	LLAMA_MAX_PLACED_RESERVATIONS_KEY

static final int	LLAMA_MAX_PLACED_RESERVATIONS_DEFAULT = 200

static final String	LLAMA_MAX_QUEUED_RESERVATIONS_KEY

static final int	LLAMA_MAX_QUEUED_RESERVATIONS_DEFAULT = 200

static final String	LLAMA_PER_POOL_CONFIG_KEY_FORMAT = "%s.%s"

Private Member Functions
void	stopInternal ()

int	getLlamaPoolConfigValue (Configuration conf, String pool, String key, int defaultValue)

Private Attributes
final AtomicBoolean	running_

final AtomicReference < AllocationConfiguration >	allocationConf_

volatile Configuration	llamaConf_

final URL	llamaConfUrl_

Static Private Attributes
static final TBinaryProtocol.Factory	protocolFactory_

Detailed Description

Admission control utility class that provides user to request pool mapping, ACL enforcement, and pool configuration values. Pools are configured via a fair scheduler allocation file (fair-scheduler.xml) and Llama configuration (llama-site.xml). This class wraps a number of Hadoop classes to provide the user to pool mapping, authorization, and accessing memory resource limits, all of which are specified in the fair scheduler allocation file. The other pool limits are specified in the Llama configuration, and those properties are accessed via the standard Configuration API.

Both the allocation configuration and Llama configuration files are watched for changes and reloaded when necessary. The allocation file is watched/loaded using the Yarn AllocationFileLoaderService and the Llama configuration uses a subclass of the FileWatchService. There are two different mechanisms because there is different parsing/configuration code for the allocation file and the Llama configuration (which is a regular Hadoop conf file so it can use the Configuration class). start() and stop() will start/stop watching and reloading both of these files.

A single instance is created by the backend and lasts the duration of the process.

Definition at line 75 of file RequestPoolService.java.

Constructor & Destructor Documentation

com.cloudera.impala.util.RequestPoolService.RequestPoolService	(	final String	fsAllocationPath,
		final String	llamaSitePath
	)

inline

Creates a RequestPoolService instance with a configuration containing the specified fair-scheduler.xml and llama-site.xml.

Parameters

fsAllocationPath	path to the fair scheduler allocation file.
llamaSitePath	path to the Llama configuration file.

Definition at line 151 of file RequestPoolService.java.

References com.cloudera.impala.util.RequestPoolService.allocationConf_, com.cloudera.impala.util.RequestPoolService.allocLoader_, com.cloudera.impala.util.RequestPoolService.getURL(), com.cloudera.impala.util.RequestPoolService.llamaConf_, com.cloudera.impala.util.RequestPoolService.llamaConfUrl_, com.cloudera.impala.util.RequestPoolService.llamaConfWatcher_, and com.cloudera.impala.util.RequestPoolService.running_.

Member Function Documentation

String com.cloudera.impala.util.RequestPoolService.assignToPool	(	String	requestedPool,
		String	user
	)		throws IOException

inlinepackage

Resolves the actual pool to use via the allocation placement policy. The policy may change the requested pool.

Parameters

requestedPool	The requested pool. May not be null, an empty string indicates the policy should return the default pool for this user.
user	The user, must not be null or empty.

Returns: the actual pool to use, null if a pool could not be resolved.

Definition at line 379 of file RequestPoolService.java.

Referenced by com.cloudera.impala.util.RequestPoolService.resolveRequestPool().

int com.cloudera.impala.util.RequestPoolService.getLlamaPoolConfigValue	(	Configuration	conf,
		String	pool,
		String	key,
		int	defaultValue
	)

inlineprivate

Looks up the per-pool Llama config, first checking for a per-pool value, then a default set in the config, and lastly to the specified 'defaultValue'.

Parameters

conf	The Configuration to use, provided so the caller can ensure the same Configuration is used to look up multiple properties.

Definition at line 363 of file RequestPoolService.java.

References com.cloudera.impala.util.RequestPoolService.LLAMA_PER_POOL_CONFIG_KEY_FORMAT, and pool.

Referenced by com.cloudera.impala.util.RequestPoolService.getPoolConfig().

byte [] com.cloudera.impala.util.RequestPoolService.getPoolConfig ( byte[] thriftPoolConfigParams ) throws ImpalaException

inline

Gets the pool configuration values for the specified pool.

Parameters

thriftPoolConfigParams Serialized TPoolConfigParams

Returns: serialized TPoolConfigResult

Definition at line 319 of file RequestPoolService.java.

References com.cloudera.impala.util.RequestPoolService.protocolFactory_.

TPoolConfigResult com.cloudera.impala.util.RequestPoolService.getPoolConfig ( String pool )

inlinepackage

Definition at line 333 of file RequestPoolService.java.

References com.cloudera.impala.util.RequestPoolService.getLlamaPoolConfigValue(), com.cloudera.impala.util.RequestPoolService.LLAMA_MAX_PLACED_RESERVATIONS_DEFAULT, com.cloudera.impala.util.RequestPoolService.LLAMA_MAX_PLACED_RESERVATIONS_KEY, com.cloudera.impala.util.RequestPoolService.LLAMA_MAX_QUEUED_RESERVATIONS_DEFAULT, com.cloudera.impala.util.RequestPoolService.LLAMA_MAX_QUEUED_RESERVATIONS_KEY, com.cloudera.impala.util.RequestPoolService.llamaConf_, com.cloudera.impala.common.ByteUnits.MEGABYTE, and pool.

static URL com.cloudera.impala.util.RequestPoolService.getURL ( String path )

inlinestaticpackage

Returns a URL for the file if it exists, null otherwise.

Definition at line 185 of file RequestPoolService.java.

References path().

Referenced by com.cloudera.impala.util.RequestPoolService.RequestPoolService().

boolean com.cloudera.impala.util.RequestPoolService.hasAccess	(	String	pool,
		String	user
	)

inlinepackage

Indicates if a user has access to the pool.

Parameters

pool	the pool to check if the user has access to. NOTE: it should always be called with a pool returned by the assignToPool(String, String) method.
user	the user to check if it has access to the pool.

Returns: True if the user has access to the pool.

Definition at line 402 of file RequestPoolService.java.

References pool.

Referenced by com.cloudera.impala.util.RequestPoolService.resolveRequestPool().

byte [] com.cloudera.impala.util.RequestPoolService.resolveRequestPool ( byte[] thriftResolvePoolParams ) throws ImpalaException

inline

Resolves a user and pool to the pool specified by the allocation placement policy and checks if the user is authorized to submit requests.

Parameters

thriftResolvePoolParams Serialized TResolveRequestPoolParams

Returns: serialized TResolveRequestPoolResult

Definition at line 255 of file RequestPoolService.java.

References com.cloudera.impala.util.RequestPoolService.protocolFactory_.

TResolveRequestPoolResult com.cloudera.impala.util.RequestPoolService.resolveRequestPool ( TResolveRequestPoolParams resolvePoolParams )

inlinepackage

Definition at line 272 of file RequestPoolService.java.

References com.cloudera.impala.util.RequestPoolService.assignToPool(), com.cloudera.impala.util.RequestPoolService.hasAccess(), and pool.

void com.cloudera.impala.util.RequestPoolService.start ( )

inline

Starts the RequestPoolService instance. It does the initial loading of the configuration and starts the automatic reloading.

Definition at line 205 of file RequestPoolService.java.

References com.cloudera.impala.util.RequestPoolService.llamaConfWatcher_, and com.cloudera.impala.util.RequestPoolService.stopInternal().

void com.cloudera.impala.util.RequestPoolService.stop ( )

inline

Stops the RequestPoolService instance. Only used by tests.

Definition at line 232 of file RequestPoolService.java.

References com.cloudera.impala.util.RequestPoolService.stopInternal().

void com.cloudera.impala.util.RequestPoolService.stopInternal ( )

inlineprivate

Stops the RequestPoolService instance without checking the running state. Only called by stop() (which is only used in tests) or by start() if a failure occurs. Should not be called more than once.

Definition at line 242 of file RequestPoolService.java.

References com.cloudera.impala.util.RequestPoolService.llamaConfWatcher_.

Referenced by com.cloudera.impala.util.RequestPoolService.start(), and com.cloudera.impala.util.RequestPoolService.stop().