#include <thread-pool.h>

Collaboration diagram for impala::ThreadPool< T >:

Public Types
typedef boost::function< void(int thread_id, const T &workitem)>	WorkFunction

Public Member Functions
	ThreadPool (const std::string &group, const std::string &thread_prefix, uint32_t num_threads, uint32_t queue_size, const WorkFunction &work_function)

	~ThreadPool ()

bool	Offer (const T &work)

void	Shutdown ()

void	Join ()

uint32_t	GetQueueSize () const

void	DrainAndShutdown ()

Status	AssignToCgroup (const std::string &cgroup)

Private Member Functions
void	WorkerThread (int thread_id)

bool	IsShutdown ()
	Returns value of shutdown_ under a lock, forcing visibility to threads in the pool. More...

Private Attributes
WorkFunction	work_function_
	User-supplied method to call to process each work item. More...

BlockingQueue< T >	work_queue_

ThreadGroup	threads_
	Collection of worker threads that process work from the queue. More...

boost::mutex	lock_
	Guards shutdown_ and empty_cv_. More...

bool	shutdown_
	Set to true when threads should stop doing work and terminate. More...

boost::condition_variable	empty_cv_
	Signalled when the queue becomes empty. More...

Detailed Description

template<typename T>
class impala::ThreadPool< T >

Simple threadpool which processes items (of type T) in parallel which were placed on a blocking queue by Offer(). Each item is processed by a single user-supplied method.

Definition at line 30 of file thread-pool.h.

Member Typedef Documentation

template<typename T>

typedef boost::function<void (int thread_id, const T& workitem)> impala::ThreadPool< T >::WorkFunction

Signature of a work-processing function. Takes the integer id of the thread which is calling it (ids run from 0 to num_threads - 1) and a reference to the item to process.

Definition at line 35 of file thread-pool.h.

Constructor & Destructor Documentation

template<typename T>

impala::ThreadPool< T >::ThreadPool	(	const std::string &	group,
		const std::string &	thread_prefix,
		uint32_t	num_threads,
		uint32_t	queue_size,
		const WorkFunction &	work_function
	)

inline

Creates a new thread pool and start num_threads threads. – num_threads: how many threads are part of this pool – queue_size: the maximum size of the queue on which work items are offered. If the queue exceeds this size, subsequent calls to Offer will block until there is capacity available. – work_function: the function to run every time an item is consumed from the queue

Definition at line 43 of file thread-pool.h.

template<typename T>

impala::ThreadPool< T >::~ThreadPool ( )

inline

Destructor ensures that all threads are terminated before this object is freed (otherwise they may continue to run and reference member variables)

Definition at line 58 of file thread-pool.h.

Member Function Documentation

template<typename T>

Status impala::ThreadPool< T >::AssignToCgroup ( const std::string & cgroup )

inline

Definition at line 114 of file thread-pool.h.

template<typename T>

void impala::ThreadPool< T >::DrainAndShutdown ( )

inline

Blocks until the work queue is empty, and then calls Shutdown to stop the worker threads and Join to wait until they are finished. Any work Offer()'ed during DrainAndShutdown may or may not be processed.

Definition at line 103 of file thread-pool.h.

Referenced by impala::TEST().

template<typename T>

uint32_t impala::ThreadPool< T >::GetQueueSize ( ) const

inline

Definition at line 96 of file thread-pool.h.

Referenced by impala::Statestore::OfferUpdate(), and impala::TEST().

template<typename T>

bool impala::ThreadPool< T >::IsShutdown ( )

inlineprivate

Returns value of shutdown_ under a lock, forcing visibility to threads in the pool.

Definition at line 138 of file thread-pool.h.

Referenced by impala::ThreadPool< TRowBatch * >::WorkerThread().

template<typename T>

void impala::ThreadPool< T >::Join ( )

inline

Blocks until all threads are finished. Shutdown does not need to have been called, since it may be called on a separate thread.

Definition at line 92 of file thread-pool.h.

Referenced by impala::ThreadPool< TRowBatch * >::DrainAndShutdown(), impala::Statestore::MainLoop(), and impala::ThreadPool< TRowBatch * >::~ThreadPool().

template<typename T>

bool impala::ThreadPool< T >::Offer ( const T & work )

inline

Blocking operation that puts a work item on the queue. If the queue is full, blocks until there is capacity available. 'work' is copied into the work queue, but may be referenced at any time in the future. Therefore the caller needs to ensure that any data referenced by work (if T is, e.g., a pointer type) remains valid until work has been processed, and it's up to the caller to provide their own signalling mechanism to detect this (or to wait until after DrainAndShutdown returns). Returns true if the work item was successfully added to the queue, false otherwise (which typically means that the thread pool has already been shut down).

Definition at line 74 of file thread-pool.h.

Referenced by impala::HdfsOperationSet::Execute(), impala::Statestore::OfferUpdate(), and impala::TEST().

template<typename T>

void impala::ThreadPool< T >::Shutdown ( )

inline

Shuts the thread pool down, causing the work queue to cease accepting offered work and the worker threads to terminate once they have processed their current work item. Returns once the shutdown flag has been set, does not wait for the threads to terminate.

Definition at line 82 of file thread-pool.h.

Referenced by impala::ThreadPool< TRowBatch * >::DrainAndShutdown(), impala::Statestore::SetExitFlag(), and impala::ThreadPool< TRowBatch * >::~ThreadPool().

template<typename T>

void impala::ThreadPool< T >::WorkerThread ( int thread_id )

inlineprivate

Driver method for each thread in the pool. Continues to read work from the queue until the pool is shutdown.

Take lock to ensure that DrainAndShutdown() cannot be between checking GetSize() and wait()'ing when the condition variable is notified. (It will hang if we notify right before calling wait().)

Definition at line 121 of file thread-pool.h.