impala::DiskIoMgr::RequestContext::PerDiskState Class Reference

Collaboration diagram for impala::DiskIoMgr::RequestContext::PerDiskState:

Public Member Functions
bool	done () const
void	set_done (bool b)
int	num_remaining_ranges () const
int &	num_remaining_ranges ()
ScanRange *	next_scan_range_to_start ()
void	set_next_scan_range_to_start (ScanRange *range)
bool	is_on_queue () const
int	num_threads_in_op () const
const InternalQueue< ScanRange > *	unstarted_scan_ranges () const
const InternalQueue< WriteRange > *	unstarted_write_ranges () const
const InternalQueue < RequestRange > *	in_flight_ranges () const
InternalQueue< ScanRange > *	unstarted_scan_ranges ()
InternalQueue< WriteRange > *	unstarted_write_ranges ()
InternalQueue< RequestRange > *	in_flight_ranges ()
	PerDiskState ()
void	ScheduleContext (RequestContext *context, int disk_id)
void	IncrementRequestThreadAndDequeue ()
void	DecrementRequestThread ()
void	DecrementRequestThreadAndCheckDone (RequestContext *context)
void	Reset ()

Private Attributes
bool	done_
bool	is_on_queue_
int	num_remaining_ranges_
InternalQueue< ScanRange >	unstarted_scan_ranges_
InternalQueue< RequestRange >	in_flight_ranges_
ScanRange *	next_scan_range_to_start_
AtomicInt< int >	num_threads_in_op_
InternalQueue< WriteRange >	unstarted_write_ranges_

Detailed Description

Struct containing state per disk. See comments in the disk read loop on how they are used.

Definition at line 302 of file disk-io-mgr-internal.h.

Constructor & Destructor Documentation

impala::DiskIoMgr::RequestContext::PerDiskState::PerDiskState ( )

inline

Definition at line 346 of file disk-io-mgr-internal.h.

References Reset().

Member Function Documentation

void impala::DiskIoMgr::RequestContext::PerDiskState::DecrementRequestThread ( )

inline

Definition at line 367 of file disk-io-mgr-internal.h.

References num_threads_in_op_.

Referenced by impala::DiskIoMgr::GetNextRequestRange(), impala::DiskIoMgr::HandleReadFinished(), impala::DiskIoMgr::HandleWriteFinished(), and impala::DiskIoMgr::ReadRange().

void impala::DiskIoMgr::RequestContext::PerDiskState::DecrementRequestThreadAndCheckDone ( RequestContext *  context )

inline

Decrement request thread count and do final cleanup if this is the last thread. RequestContext lock must be taken before this.

Definition at line 373 of file disk-io-mgr-internal.h.

References impala::DiskIoMgr::RequestContext::DecrementDiskRefCount(), done_, is_on_queue_, and num_threads_in_op_.

Referenced by impala::DiskIoMgr::GetNextRequestRange(), impala::DiskIoMgr::HandleReadFinished(), impala::DiskIoMgr::HandleWriteFinished(), and impala::DiskIoMgr::ReadRange().

bool impala::DiskIoMgr::RequestContext::PerDiskState::done ( ) const

inline

Definition at line 304 of file disk-io-mgr-internal.h.

References done_.

Referenced by impala::DiskIoMgr::RequestContext::AddRequestRange(), and impala::DiskIoMgr::RequestContext::Validate().

const InternalQueue<RequestRange>* impala::DiskIoMgr::RequestContext::PerDiskState::in_flight_ranges ( ) const

inline

Definition at line 336 of file disk-io-mgr-internal.h.

References in_flight_ranges_.

Referenced by impala::DiskIoMgr::RequestContext::Cancel(), impala::DiskIoMgr::GetNextRequestRange(), impala::DiskIoMgr::RequestContext::ScheduleScanRange(), and impala::DiskIoMgr::RequestContext::Validate().

InternalQueue<RequestRange>* impala::DiskIoMgr::RequestContext::PerDiskState::in_flight_ranges ( )

inline

Definition at line 344 of file disk-io-mgr-internal.h.

References in_flight_ranges_.

void impala::DiskIoMgr::RequestContext::PerDiskState::IncrementRequestThreadAndDequeue ( )

inline

Increment the ref count on reader. We need to track the number of threads per reader per disk that are in the unlocked hdfs read code section. This is updated by multiple threads without a lock so we need to use an atomic int.

Definition at line 362 of file disk-io-mgr-internal.h.

References is_on_queue_, and num_threads_in_op_.

Referenced by impala::DiskIoMgr::GetNextRequestRange().

bool impala::DiskIoMgr::RequestContext::PerDiskState::is_on_queue ( ) const

inline

We need to have a memory barrier to prevent this load from being reordered with num_threads_in_op(), since these variables are set without the reader lock taken

Definition at line 318 of file disk-io-mgr-internal.h.

References is_on_queue_.

Referenced by impala::DiskIoMgr::RequestContext::Validate().

ScanRange* impala::DiskIoMgr::RequestContext::PerDiskState::next_scan_range_to_start ( )

inline

Definition at line 310 of file disk-io-mgr-internal.h.

References next_scan_range_to_start_.

Referenced by impala::DiskIoMgr::RequestContext::AddRequestRange(), and impala::DiskIoMgr::GetNextRequestRange().

int impala::DiskIoMgr::RequestContext::PerDiskState::num_remaining_ranges ( ) const

inline

Definition at line 307 of file disk-io-mgr-internal.h.

References num_remaining_ranges_.

Referenced by impala::DiskIoMgr::RequestContext::AddRequestRange(), impala::DiskIoMgr::GetNextRequestRange(), impala::DiskIoMgr::HandleReadFinished(), impala::DiskIoMgr::HandleWriteFinished(), and impala::DiskIoMgr::RequestContext::Validate().

int& impala::DiskIoMgr::RequestContext::PerDiskState::num_remaining_ranges ( )

inline

Definition at line 308 of file disk-io-mgr-internal.h.

References num_remaining_ranges_.

int impala::DiskIoMgr::RequestContext::PerDiskState::num_threads_in_op ( ) const

inline

Definition at line 324 of file disk-io-mgr-internal.h.

References num_threads_in_op_.

Referenced by impala::DiskIoMgr::HandleReadFinished(), and impala::DiskIoMgr::RequestContext::Validate().

void impala::DiskIoMgr::RequestContext::PerDiskState::Reset ( )

inline

Definition at line 384 of file disk-io-mgr-internal.h.

References done_, in_flight_ranges_, is_on_queue_, next_scan_range_to_start_, num_remaining_ranges_, num_threads_in_op_, unstarted_scan_ranges_, and unstarted_write_ranges_.

Referenced by PerDiskState().

void impala::DiskIoMgr::RequestContext::PerDiskState::ScheduleContext ( RequestContext *  context, int  disk_id  )

inline

Schedules the request context on this disk if it's not already on the queue. Context lock must be taken before this.

Definition at line 352 of file disk-io-mgr-internal.h.

References impala::DiskIoMgr::disk_queues_, done_, is_on_queue_, and impala::DiskIoMgr::RequestContext::parent_.

Referenced by impala::DiskIoMgr::RequestContext::AddRequestRange(), impala::DiskIoMgr::RequestContext::Cancel(), impala::DiskIoMgr::GetNextRequestRange(), and impala::DiskIoMgr::RequestContext::ScheduleScanRange().

void impala::DiskIoMgr::RequestContext::PerDiskState::set_done ( bool  b )

inline

Definition at line 305 of file disk-io-mgr-internal.h.

References done_.

Referenced by impala::DiskIoMgr::RequestContext::AddRequestRange().

void impala::DiskIoMgr::RequestContext::PerDiskState::set_next_scan_range_to_start ( ScanRange *  range )

inline

Definition at line 311 of file disk-io-mgr-internal.h.

References next_scan_range_to_start_.

Referenced by impala::DiskIoMgr::GetNextRequestRange().

const InternalQueue<ScanRange>* impala::DiskIoMgr::RequestContext::PerDiskState::unstarted_scan_ranges ( ) const

inline

Definition at line 330 of file disk-io-mgr-internal.h.

References unstarted_scan_ranges_.

Referenced by impala::DiskIoMgr::RequestContext::AddRequestRange(), impala::DiskIoMgr::RequestContext::Cancel(), impala::DiskIoMgr::GetNextRequestRange(), and impala::DiskIoMgr::RequestContext::Validate().

InternalQueue<ScanRange>* impala::DiskIoMgr::RequestContext::PerDiskState::unstarted_scan_ranges ( )

inline

Definition at line 340 of file disk-io-mgr-internal.h.

References unstarted_scan_ranges_.

const InternalQueue<WriteRange>* impala::DiskIoMgr::RequestContext::PerDiskState::unstarted_write_ranges ( ) const

inline

Definition at line 333 of file disk-io-mgr-internal.h.

References unstarted_write_ranges_.

Referenced by impala::DiskIoMgr::RequestContext::AddRequestRange(), impala::DiskIoMgr::RequestContext::Cancel(), and impala::DiskIoMgr::GetNextRequestRange().

InternalQueue<WriteRange>* impala::DiskIoMgr::RequestContext::PerDiskState::unstarted_write_ranges ( )

inline

Definition at line 341 of file disk-io-mgr-internal.h.

References unstarted_write_ranges_.

Member Data Documentation

bool impala::DiskIoMgr::RequestContext::PerDiskState::done_

private

If true, this disk is all done for this request context, including any cleanup. If done is true, it means that this request must not be on this disk's queue AND there are no threads currently working on this context. To satisfy this, only the last thread (per disk) can set this to true.

Definition at line 401 of file disk-io-mgr-internal.h.

Referenced by DecrementRequestThreadAndCheckDone(), done(), Reset(), ScheduleContext(), and set_done().

InternalQueue<RequestRange> impala::DiskIoMgr::RequestContext::PerDiskState::in_flight_ranges_

private

Queue of pending IO requests for this disk in the order that they will be processed. A ScanRange is added to this queue when it is returned in GetNextRange(), or when it is added with schedule_immediately = true. A WriteRange is added to this queue from unstarted_write_ranges_ for each invocation of GetNextRequestRange() in WorkLoop(). The size of this queue is always less than or equal to num_remaining_ranges.

Definition at line 433 of file disk-io-mgr-internal.h.

Referenced by in_flight_ranges(), and Reset().

bool impala::DiskIoMgr::RequestContext::PerDiskState::is_on_queue_

private

For each disk, keeps track if the context is on this disk's queue, indicating the disk must do some work for this context. The disk needs to do work in 4 cases: 1) in_flight_ranges is not empty, the disk needs to read for this reader. 2) next_range_to_start is NULL, the disk needs to prepare a scan range to be read next. 3) the reader has been cancelled and this disk needs to participate in the cleanup. 4) A write range is added to queue. In general, we only want to put a context on the disk queue if there is something useful that can be done. If there's nothing useful, the disk queue will wake up and then remove the reader from the queue. Doing this causes thrashing of the threads.

Definition at line 415 of file disk-io-mgr-internal.h.

Referenced by DecrementRequestThreadAndCheckDone(), IncrementRequestThreadAndDequeue(), is_on_queue(), Reset(), and ScheduleContext().

ScanRange* impala::DiskIoMgr::RequestContext::PerDiskState::next_scan_range_to_start_

private

The next range to start for this reader on this disk. Each disk (for each reader) picks the next range to start. The range is set here and also added to the ready_to_start_ranges_ queue. The reader pulls from the queue in FIFO order, so the ranges from different disks are round-robined. When the range is pulled off the ready_to_start_ranges_ queue, it sets this variable to NULL, so the disk knows to populate it again and add it to ready_to_start_ranges_ i.e. it is used as a flag by DiskIoMgr::GetNextScanRange to determine if it needs to add another range to ready_to_start_ranges_.

Definition at line 443 of file disk-io-mgr-internal.h.

Referenced by next_scan_range_to_start(), Reset(), and set_next_scan_range_to_start().

int impala::DiskIoMgr::RequestContext::PerDiskState::num_remaining_ranges_

private

For each disks, the number of request ranges that have not been fully read. In the non-cancellation path, this will hit 0, and done will be set to true by the disk thread. This is undefined in the cancellation path (the various threads notice by looking at the RequestContext's state_).

Definition at line 421 of file disk-io-mgr-internal.h.

Referenced by num_remaining_ranges(), and Reset().

AtomicInt<int> impala::DiskIoMgr::RequestContext::PerDiskState::num_threads_in_op_

private

For each disk, the number of threads issuing the underlying read/write on behalf of this context. There are a few places where we release the context lock, do some work, and then grab the lock again. Because we don't hold the lock for the entire operation, we need this ref count to keep track of which thread should do final resource cleanup during cancellation. Only the thread that sees the count at 0 should do the final cleanup.

Definition at line 451 of file disk-io-mgr-internal.h.

Referenced by DecrementRequestThread(), DecrementRequestThreadAndCheckDone(), IncrementRequestThreadAndDequeue(), num_threads_in_op(), and Reset().

InternalQueue<ScanRange> impala::DiskIoMgr::RequestContext::PerDiskState::unstarted_scan_ranges_

private

Queue of ranges that have not started being read. This list is exclusive with in_flight_ranges.

Definition at line 425 of file disk-io-mgr-internal.h.

Referenced by Reset(), and unstarted_scan_ranges().

InternalQueue<WriteRange> impala::DiskIoMgr::RequestContext::PerDiskState::unstarted_write_ranges_

private

Queue of write ranges to process for this disk. A write range is always added to in_flight_ranges_ in GetNextRequestRange(). There is a separate unstarted_read_ranges_ and unstarted_write_ranges_ to alternate between reads and writes. (Otherwise, since next_scan_range_to_start is set in GetNextRequestRange() whenever it is null, repeated calls to GetNextRequestRange() and GetNextRange() may result in only reads being processed)

Definition at line 459 of file disk-io-mgr-internal.h.

Referenced by Reset(), and unstarted_write_ranges().

---

The documentation for this class was generated from the following file:

• be/src/runtime/disk-io-mgr-internal.h