Impala
Impalaistheopensource,nativeanalyticdatabaseforApacheHadoop.
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros
Public Member Functions | Private Member Functions | Private Attributes | Friends | List of all members
impala::ThreadResourceMgr::ResourcePool Class Reference

#include <thread-resource-mgr.h>

Collaboration diagram for impala::ThreadResourceMgr::ResourcePool:

Public Member Functions

void AcquireThreadToken ()
 
bool TryAcquireThreadToken (bool *is_reserved=NULL)
 
void ReserveOptionalTokens (int num)
 
void ReleaseThreadToken (bool required)
 
void SetThreadAvailableCb (ThreadAvailableCb fn)
 
int num_required_threads () const
 Returns the number of threads that are from AcquireThreadToken. More...
 
int num_optional_threads () const
 
int64_t num_threads () const
 
int num_reserved_optional_threads ()
 
bool optional_exceeded ()
 Returns true if the number of optional threads has now exceeded the quota. More...
 
int num_available_threads () const
 Returns the number of optional threads that can still be used. More...
 
int quota () const
 
void set_max_quota (int quota)
 

Private Member Functions

 ResourcePool (ThreadResourceMgr *parent)
 
void Reset ()
 Resets internal state. More...
 

Private Attributes

ThreadResourceMgr * parent_
 
int max_quota_
 
int num_reserved_optional_threads_
 
int64_t num_threads_
 
boost::mutex lock_
 
ThreadAvailableCb thread_available_fn_
 

Friends

class ThreadResourceMgr
 

Detailed Description

Pool abstraction for a single resource pool. TODO: this is not quite sufficient going forward. We need a hierarchy of pools, one for the entire query, and a sub pool for each component that needs threads, all of which share a quota. Currently, the way state is tracked here, it would be impossible to have two components both want optional threads (e.g. two things that have 1+ thread usage).

Definition at line 85 of file thread-resource-mgr.h.

Constructor & Destructor Documentation

ThreadResourceMgr::ResourcePool::ResourcePool ( ThreadResourceMgr *  parent)
private

Definition at line 45 of file thread-resource-mgr.cc.

Member Function Documentation

void impala::ThreadResourceMgr::ResourcePool::AcquireThreadToken ( )
inline

Acquire a thread for the pool. This will always succeed; the pool will go over the quota. Pools should use this API to reserve threads they need in order to make progress.

Definition at line 231 of file thread-resource-mgr.h.

References num_threads_.

Referenced by impala::TEST().

int impala::ThreadResourceMgr::ResourcePool::num_available_threads ( ) const
inline

Returns the number of optional threads that can still be used.

Definition at line 151 of file thread-resource-mgr.h.

References num_optional_threads(), num_reserved_optional_threads_, num_threads(), and quota().

Referenced by impala::ThreadResourceMgr::UpdatePoolQuotas().

int impala::ThreadResourceMgr::ResourcePool::num_optional_threads ( ) const
inline

Returns the number of thread resources returned by successful calls to TryAcquireThreadToken.

Definition at line 130 of file thread-resource-mgr.h.

References num_threads_.

Referenced by num_available_threads(), num_threads(), and impala::TEST().

int impala::ThreadResourceMgr::ResourcePool::num_required_threads ( ) const
inline

Returns the number of threads that are from AcquireThreadToken.

Definition at line 126 of file thread-resource-mgr.h.

References num_threads_.

Referenced by num_threads(), and impala::TEST().

int impala::ThreadResourceMgr::ResourcePool::num_reserved_optional_threads ( )
inline

Definition at line 138 of file thread-resource-mgr.h.

References num_reserved_optional_threads_.

int64_t impala::ThreadResourceMgr::ResourcePool::num_threads ( ) const
inline

Returns the total number of thread resources for this pool (i.e. num_optional_threads + num_required_threads).

Definition at line 134 of file thread-resource-mgr.h.

References num_optional_threads(), and num_required_threads().

Referenced by impala::NotifiedCounter::Notify(), num_available_threads(), optional_exceeded(), impala::PlanFragmentExecutor::Prepare(), and impala::TEST().

bool impala::ThreadResourceMgr::ResourcePool::optional_exceeded ( )
inline

Returns true if the number of optional threads has now exceeded the quota.

Definition at line 141 of file thread-resource-mgr.h.

References num_reserved_optional_threads_, num_threads(), num_threads_, and quota().

Referenced by impala::HdfsScanNode::ScannerThread().

int impala::ThreadResourceMgr::ResourcePool::quota ( ) const
inline

Returns the quota for this pool. Note this changes dynamically based on system load.

Definition at line 159 of file thread-resource-mgr.h.

References max_quota_, parent_, and impala::ThreadResourceMgr::per_pool_quota_.

Referenced by impala::NotifiedCounter::Notify(), num_available_threads(), optional_exceeded(), and set_max_quota().

void impala::ThreadResourceMgr::ResourcePool::ReleaseThreadToken ( bool  required)
inline

Release a thread for the pool. This must be called once for each call to AcquireThreadToken and each successful call to TryAcquireThreadToken If the thread token is from AcquireThreadToken, required must be true; false if from TryAcquireThreadToken. Must not be called from from ThreadAvailableCb.

We need to grab a lock before issuing the callback to prevent the callback from being removed while it is happening. Note: this is unlikely to be a big deal for performance currently since only scanner threads call this with any frequency and that only happens once when the scanner thread is complete. TODO: reconsider this.

Definition at line 255 of file thread-resource-mgr.h.

References impala::ThreadResourceMgr::lock_.

Referenced by impala::BlockingJoinNode::BuildSideThread(), impala::HdfsScanNode::ScannerThread(), and impala::TEST().

void ThreadResourceMgr::ResourcePool::ReserveOptionalTokens ( int  num)

Set a reserved optional number of threads for this pool. This can be used to implement that a component needs n+ number of threads. The first 'num' threads are guaranteed to be acquirable (via TryAcquireThreadToken) but anything beyond can fail. This can also be done with: if (pool->num_optional_threads() < num) AcquireThreadToken(); else TryAcquireThreadToken(); and similar tracking on the Release side but this is common enough to abstract it away.

Definition at line 56 of file thread-resource-mgr.cc.

Referenced by impala::HdfsScanNode::Open(), and impala::TEST().

void ThreadResourceMgr::ResourcePool::Reset ( )
private

Resets internal state.

Definition at line 49 of file thread-resource-mgr.cc.

Referenced by impala::ThreadResourceMgr::RegisterPool().

void impala::ThreadResourceMgr::ResourcePool::set_max_quota ( int  quota)
inline

Sets the max thread quota for this pool. The actual quota is the min of this value and the dynamic value.

Definition at line 163 of file thread-resource-mgr.h.

References max_quota_, and quota().

Referenced by impala::HdfsScanNode::Open().

void ThreadResourceMgr::ResourcePool::SetThreadAvailableCb ( ThreadAvailableCb  fn)

Add a callback to be notified when a thread is available. 'arg' is opaque and passed directly to the callback. The previous callback is no longer notified. TODO: rethink this. How we do coordinate when we have multiple places in the execution that all need threads (e.g. do we use that thread for the scanner or for the join).

Definition at line 90 of file thread-resource-mgr.cc.

References impala::ThreadResourceMgr::lock_.

Referenced by impala::HdfsScanNode::Close(), impala::HdfsScanNode::Open(), and impala::TEST().

bool impala::ThreadResourceMgr::ResourcePool::TryAcquireThreadToken ( bool *  is_reserved = NULL)
inline

Try to acquire a thread for this pool. If the pool is at the quota, this will return false and the pool should not run. Pools should use this API for resources they can use but don't need (e.g. scanner threads).

Definition at line 235 of file thread-resource-mgr.h.

Referenced by impala::BlockingJoinNode::Open(), impala::TEST(), and impala::HdfsScanNode::ThreadTokenAvailableCb().

Friends And Related Function Documentation

friend class ThreadResourceMgr
friend

Definition at line 166 of file thread-resource-mgr.h.

Member Data Documentation

boost::mutex impala::ThreadResourceMgr::ResourcePool::lock_
private

Lock for the fields below. This lock is taken when the callback function is called. TODO: reconsider this.

Definition at line 187 of file thread-resource-mgr.h.

Referenced by impala::ThreadResourceMgr::UpdatePoolQuotas().

int impala::ThreadResourceMgr::ResourcePool::max_quota_
private

Definition at line 175 of file thread-resource-mgr.h.

Referenced by quota(), and set_max_quota().

int impala::ThreadResourceMgr::ResourcePool::num_reserved_optional_threads_
private

Definition at line 176 of file thread-resource-mgr.h.

Referenced by num_available_threads(), num_reserved_optional_threads(), and optional_exceeded().

int64_t impala::ThreadResourceMgr::ResourcePool::num_threads_
private

A single 64 bit value to store both the number of optional and required threads. This is combined to allow using compare and swap operations. The number of required threads is the lower 32 bits and the number of optional threads is the upper 32 bits.

Definition at line 182 of file thread-resource-mgr.h.

Referenced by AcquireThreadToken(), num_optional_threads(), num_required_threads(), and optional_exceeded().

ThreadResourceMgr* impala::ThreadResourceMgr::ResourcePool::parent_
private

Definition at line 173 of file thread-resource-mgr.h.

Referenced by quota().

ThreadAvailableCb impala::ThreadResourceMgr::ResourcePool::thread_available_fn_
private

Definition at line 189 of file thread-resource-mgr.h.

Referenced by impala::ThreadResourceMgr::UpdatePoolQuotas().

The documentation for this class was generated from the following files: