Only CPU-heavy threads need be managed using this class. More...

#include <query-resource-mgr.h>

Collaboration diagram for impala::QueryResourceMgr:

Public Types
typedef boost::function< void()>	VcoreAvailableCb

Public Member Functions
	QueryResourceMgr (const TUniqueId &reservation_id, const TNetworkAddress &local_resource_location, const TUniqueId &query_id)

void	InitVcoreAcquisition (int32_t init_vcores)

bool	IsVcoreOverSubscribed ()

void	NotifyThreadUsageChange (int delta)

int32_t	AddVcoreAvailableCb (const VcoreAvailableCb &callback)

void	RemoveVcoreAvailableCb (int32_t callback_id)
	Removes the callback with the given ID. More...

Status	CreateExpansionRequest (int64_t memory_mb, int64_t vcores, TResourceBrokerExpansionRequest *request)

void	Shutdown ()

	~QueryResourceMgr ()
	Waits for the VCore acquisition thread to stop. More...

Private Types
typedef boost::unordered_map < int32_t, VcoreAvailableCb >	CallbackMap
	List of callbacks to notify when a new VCore resource is available. More...

Private Member Functions
void	AcquireVcoreResources (boost::shared_ptr< AtomicInt< int16_t > > thread_in_expand, boost::shared_ptr< AtomicInt< int16_t > > early_exit)

bool	AboveVcoreSubscriptionThreshold ()

bool	ShouldExit ()
	Notifies acquire_cpu_thread_ that it should terminate. Does not block. More...

Private Attributes
TUniqueId	reservation_id_
	ID of the single reservation corresponding to this query. More...

TUniqueId	query_id_
	Query ID of the query this class manages resources for. More...

TNetworkAddress	local_resource_location_

boost::mutex	exit_lock_
	Used to control shutdown of AcquireCpuResources(). More...

bool	exit_

boost::mutex	callbacks_lock_
	Protects callbacks_ and callbacks_it_. More...

CallbackMap	callbacks_

CallbackMap::iterator	callbacks_it_
	Round-robin iterator to notify callbacks about new VCores one at a time. More...

int32_t	callback_count_

boost::mutex	threads_running_lock_
	Protects threads_running_, threads_changed_cv_ and vcores_. More...

boost::condition_variable	threads_changed_cv_
	Waited on by AcquireCpuResources(), and notified by NotifyThreadUsageChange(). More...

int64_t	threads_running_
	The number of threads we know to be running on behalf of this query. More...

int64_t	vcores_
	The number of VCores acquired for this node for this query. More...

float	max_vcore_oversubscription_ratio_

boost::scoped_ptr< Thread >	acquire_vcore_thread_
	Runs AcquireVcoreResources() after InitVcoreAcquisition() is called. More...

boost::shared_ptr< AtomicInt < int16_t > >	early_exit_

boost::shared_ptr< AtomicInt < int16_t > >	thread_in_expand_

Detailed Description

Only CPU-heavy threads need be managed using this class.

Tracks all the state necessary to create expansion requests for all fragments of a single query on a single node. Code that might need to expand the memory reservation for this query (i.e. MemTracker) can use this class to construct expansion requests that may then be submitted to the ResourceBroker. If InitCpuAcquisition() is called, this class will monitor the thread token to VCore ratio (thread consumers must use NotifyThreadUsageChange() to update the thread consumption count). If the ratio gets too high (see AboveVcoreSubscriptionThreshold() for details), we will try to acquire more VCore resources from Llama asynchronously. If the ratio passes a higher threshold (see IsVcoreOverSubscribed()), we say that the query fragments are currently oversubscribing their VCore resources. Threads are typically handed to a fragment by the thread resource manager, which deals in tokens. When a fragment wants to use a token to start a thread, it should only do so if the ratio of threads to VCores (which map directly onto cgroup shares) is not too large. If it is too large - i.e. the VCores are oversubscribed - the fragment should wait to spin up a new threads until more VCore resources are acquired as above. To help with this, each fragment may register one or more callbacks with their QueryResourceMgr; when more VCore resources are acquired the callbacks are invoked in round-robin fashion. The callback should try and re-acquire the previously untaken thread token, and then a new thread may be started.TODO: Handle reducing the number of VCores when threads finish. TODO: Consider combining more closely with ThreadResourceMgr. TODO: Add counters to RuntimeProfile to track VCores allocated etc.

Definition at line 94 of file query-resource-mgr.h.

Member Typedef Documentation

typedef boost::unordered_map<int32_t, VcoreAvailableCb> impala::QueryResourceMgr::CallbackMap

private

List of callbacks to notify when a new VCore resource is available.

Definition at line 161 of file query-resource-mgr.h.

typedef boost::function<void ()> impala::QueryResourceMgr::VcoreAvailableCb

All callbacks registered here are called in round-robin fashion when more VCores are acquired. Returns a unique ID that can be used as an argument to RemoveVcoreAvailableCb().

Definition at line 122 of file query-resource-mgr.h.

Constructor & Destructor Documentation

QueryResourceMgr::QueryResourceMgr	(	const TUniqueId &	reservation_id,
		const TNetworkAddress &	local_resource_location,
		const TUniqueId &	query_id
	)

Definition at line 76 of file query-resource-mgr.cc.

References max_vcore_oversubscription_ratio_.

QueryResourceMgr::~QueryResourceMgr ( )

Waits for the VCore acquisition thread to stop.

Definition at line 252 of file query-resource-mgr.cc.

References acquire_vcore_thread_, early_exit_, ShouldExit(), Shutdown(), and thread_in_expand_.

Member Function Documentation

bool QueryResourceMgr::AboveVcoreSubscriptionThreshold ( )

private

True if thread:VCore subscription is too high, meaning more VCores are required. Must be called holding threads_running_ lock.

Definition at line 136 of file query-resource-mgr.cc.

References max_vcore_oversubscription_ratio_, threads_running_, and vcores_.

Referenced by AcquireVcoreResources(), and NotifyThreadUsageChange().

void QueryResourceMgr::AcquireVcoreResources	(	boost::shared_ptr< AtomicInt< int16_t > >	thread_in_expand,
		boost::shared_ptr< AtomicInt< int16_t > >	early_exit
	)

private

Run as a thread owned by acquire_cpu_thread_. Waits for notification from NotifyThreadUsageChange(), then checks the subscription level to decide if more VCores are needed, and starts a new expansion request if so.

Definition at line 162 of file query-resource-mgr.cc.

References AboveVcoreSubscriptionThreshold(), callbacks_, callbacks_it_, callbacks_lock_, impala::ExecEnv::cgroups_mgr(), CreateExpansionRequest(), impala::ResourceBroker::Expand(), impala::Status::GetDetail(), impala::ExecEnv::GetInstance(), impala::Status::ok(), impala::PrintId(), query_id_, reservation_id_, impala::ExecEnv::resource_broker(), impala::CgroupsMgr::SetCpuShares(), ShouldExit(), impala::SleepForMs(), threads_changed_cv_, threads_running_lock_, impala::CgroupsMgr::UniqueIdToCgroup(), vcores_, impala::CgroupsMgr::VirtualCoresToCpuShares(), and VLOG_QUERY.

Referenced by InitVcoreAcquisition().

int32_t QueryResourceMgr::AddVcoreAvailableCb ( const VcoreAvailableCb & callback )

Definition at line 147 of file query-resource-mgr.cc.

References callback_count_, callbacks_, callbacks_it_, and callbacks_lock_.

Referenced by impala::HdfsScanNode::Open().

Status QueryResourceMgr::CreateExpansionRequest	(	int64_t	memory_mb,
		int64_t	vcores,
		TResourceBrokerExpansionRequest *	request
	)

Creates an expansion request for the reservation corresponding to this resource context.

Definition at line 107 of file query-resource-mgr.cc.

References DEFAULT_EXPANSION_REQUEST_TIMEOUT_MS, impala::ResourceResolver::GetResourceHostport(), local_resource_location_, impala::Status::OK, reservation_id_, and impala::TNetworkAddressToString().

Referenced by AcquireVcoreResources(), and impala::MemTracker::ExpandRmReservation().

void QueryResourceMgr::InitVcoreAcquisition ( int32_t init_vcores )

Must be called only once. Starts a separate thread to monitor thread consumption, which asks for more VCores from Llama periodically.

Definition at line 84 of file query-resource-mgr.cc.

References acquire_vcore_thread_, AcquireVcoreResources(), early_exit_, impala::PrintId(), query_id_, thread_in_expand_, and vcores_.

bool impala::QueryResourceMgr::IsVcoreOverSubscribed ( )

inline

Should be used to check if another thread token may be acquired by this query. Fragments may ignore this when acquiring a new CPU token, but the result will be a larger thread:VCore ratio. Note that this threshold is larger than the one in AboveVcoreSubscriptionThreshold(). We want to start acquiring more VCore allocations before we get so oversubscribed that adding new threads is considered a bad idea.

Definition at line 110 of file query-resource-mgr.h.

References max_vcore_oversubscription_ratio_, threads_running_, threads_running_lock_, and vcores_.

Referenced by impala::HdfsScanNode::ThreadTokenAvailableCb().

void QueryResourceMgr::NotifyThreadUsageChange ( int delta )

Called when thread consumption goes up or down. If the total consumption goes above a subscription threshold, the acquisition thread will be woken to ask for more VCores.

Definition at line 140 of file query-resource-mgr.cc.

References AboveVcoreSubscriptionThreshold(), threads_changed_cv_, threads_running_, and threads_running_lock_.

Referenced by impala::HdfsScanNode::ScannerThread(), and impala::HdfsScanNode::ThreadTokenAvailableCb().

void QueryResourceMgr::RemoveVcoreAvailableCb ( int32_t callback_id )

Removes the callback with the given ID.

Definition at line 154 of file query-resource-mgr.cc.

References callbacks_, callbacks_it_, and callbacks_lock_.

Referenced by impala::HdfsScanNode::Close().

bool QueryResourceMgr::ShouldExit ( )

private

Notifies acquire_cpu_thread_ that it should terminate. Does not block.

Definition at line 229 of file query-resource-mgr.cc.

References exit_, and exit_lock_.

Referenced by AcquireVcoreResources(), and ~QueryResourceMgr().

void QueryResourceMgr::Shutdown ( )

Sets the exit flag for the VCore acquisiton thread, but does not block. Also clears the set of callbacks, so that after Shutdown() has returned, no callback will be invoked.

Definition at line 234 of file query-resource-mgr.cc.

References callbacks_, callbacks_lock_, impala::ResourceBroker::ClearRequests(), exit_, exit_lock_, impala::ExecEnv::GetInstance(), reservation_id_, impala::ExecEnv::resource_broker(), and threads_changed_cv_.

Referenced by ~QueryResourceMgr().

Member Data Documentation

boost::scoped_ptr<Thread> impala::QueryResourceMgr::acquire_vcore_thread_

private

Runs AcquireVcoreResources() after InitVcoreAcquisition() is called.

Definition at line 189 of file query-resource-mgr.h.

Referenced by InitVcoreAcquisition(), and ~QueryResourceMgr().

int32_t impala::QueryResourceMgr::callback_count_

private

Total number of callbacks that were ever registered. Used to give each callback a unique ID so that they can be removed.

Definition at line 169 of file query-resource-mgr.h.

Referenced by AddVcoreAvailableCb().

CallbackMap impala::QueryResourceMgr::callbacks_

private

Definition at line 162 of file query-resource-mgr.h.

Referenced by AcquireVcoreResources(), AddVcoreAvailableCb(), RemoveVcoreAvailableCb(), and Shutdown().

CallbackMap::iterator impala::QueryResourceMgr::callbacks_it_

private

Round-robin iterator to notify callbacks about new VCores one at a time.

Definition at line 165 of file query-resource-mgr.h.

Referenced by AcquireVcoreResources(), AddVcoreAvailableCb(), and RemoveVcoreAvailableCb().

boost::mutex impala::QueryResourceMgr::callbacks_lock_

private

Protects callbacks_ and callbacks_it_.

Definition at line 158 of file query-resource-mgr.h.

Referenced by AcquireVcoreResources(), AddVcoreAvailableCb(), RemoveVcoreAvailableCb(), and Shutdown().

boost::shared_ptr<AtomicInt<int16_t> > impala::QueryResourceMgr::early_exit_

private

Signals to the vcore acquisition thread that it should exit after it exits from any pending Expand() call. Is a shared_ptr so that it will remain valid even after the parent QueryResourceMgr has been destroyed. TODO: Combine with ShouldExit(), and replace with AtomicBool when we have such a thing.

Definition at line 196 of file query-resource-mgr.h.

Referenced by InitVcoreAcquisition(), and ~QueryResourceMgr().

bool impala::QueryResourceMgr::exit_

private

Definition at line 155 of file query-resource-mgr.h.

Referenced by ShouldExit(), and Shutdown().

boost::mutex impala::QueryResourceMgr::exit_lock_

private

Used to control shutdown of AcquireCpuResources().

Definition at line 154 of file query-resource-mgr.h.

Referenced by ShouldExit(), and Shutdown().

TNetworkAddress impala::QueryResourceMgr::local_resource_location_

private

Network address of the local service registered with Llama. Usually corresponds to <local-address>:0, unless a pseudo-dstributed Llama is being used (see ResourceResolver::CreateLocalLlamaNodeMapping()).

Definition at line 151 of file query-resource-mgr.h.

Referenced by CreateExpansionRequest().

float impala::QueryResourceMgr::max_vcore_oversubscription_ratio_

private

Set to FLAGS_max_vcore_oversubscription_ratio in the constructor. If the ratio of threads to VCores exceeds this number, no more threads may be executed by this query until more VCore resources are acquired.

Definition at line 186 of file query-resource-mgr.h.

Referenced by AboveVcoreSubscriptionThreshold(), IsVcoreOverSubscribed(), and QueryResourceMgr().

TUniqueId impala::QueryResourceMgr::query_id_

private

Query ID of the query this class manages resources for.

Definition at line 146 of file query-resource-mgr.h.

Referenced by AcquireVcoreResources(), and InitVcoreAcquisition().

TUniqueId impala::QueryResourceMgr::reservation_id_

private

ID of the single reservation corresponding to this query.

Definition at line 143 of file query-resource-mgr.h.

Referenced by AcquireVcoreResources(), CreateExpansionRequest(), and Shutdown().

boost::shared_ptr<AtomicInt<int16_t> > impala::QueryResourceMgr::thread_in_expand_

private

Signals to the destructor that the vcore acquisition thread is currently in an Expand() RPC. If so, the destructor does not need to wait for the acquisition thread to exit.

Definition at line 201 of file query-resource-mgr.h.

Referenced by InitVcoreAcquisition(), and ~QueryResourceMgr().

boost::condition_variable impala::QueryResourceMgr::threads_changed_cv_

private

Waited on by AcquireCpuResources(), and notified by NotifyThreadUsageChange().

Definition at line 175 of file query-resource-mgr.h.

Referenced by AcquireVcoreResources(), NotifyThreadUsageChange(), and Shutdown().

int64_t impala::QueryResourceMgr::threads_running_

private

The number of threads we know to be running on behalf of this query.

Definition at line 178 of file query-resource-mgr.h.

Referenced by AboveVcoreSubscriptionThreshold(), IsVcoreOverSubscribed(), and NotifyThreadUsageChange().