#include <buffered-block-mgr.h>

Inheritance diagram for impala::BufferedBlockMgr::Block:

Collaboration diagram for impala::BufferedBlockMgr::Block:

Public Member Functions
Status	Pin (bool pinned, Block release_block=NULL, bool unpin=true)

Status	Unpin ()

Status	Delete ()

void	AddRow ()

int	num_rows () const

template<typename T >
T *	Allocate (int size)
	Allocates the specified number of bytes from this block. More...

int	BytesRemaining () const
	Return the number of remaining bytes that can be allocated in this block. More...

void	ReturnAllocation (int size)
	Return size bytes from the most recent allocation. More...

uint8_t *	buffer () const

int64_t	valid_data_len () const
	Return the number of bytes allocated in this block. More...

int64_t	buffer_len () const

bool	is_max_size () const

bool	is_pinned () const

std::string	DebugString () const
	Debug helper method to print the state of a block. More...

T *	Next () const
	Returns the Next/Prev node or NULL if this is the end/front. More...

T *	Prev () const

Private Member Functions
	Block (BufferedBlockMgr *block_mgr)

void	Init ()
	Initialize the state of a block and set the number of bytes allocated to 0. More...

bool	Validate () const

Private Attributes
BufferDescriptor *	buffer_desc_

BufferedBlockMgr *	block_mgr_
	Parent block manager object. Responsible for maintaining the state of the block. More...

Client *	client_
	The client that owns this block. More...

DiskIoMgr::WriteRange *	write_range_

int64_t	valid_data_len_
	Length of valid (i.e. allocated) data within the block. More...

int	num_rows_
	Number of rows in this block. More...

boost::scoped_array< uint8_t >	encrypted_write_buffer_

uint8_t	key_ [32]
	If encryption_ is on, a AES 256-bit key. Regenerated on each write. More...

uint8_t	iv_ [AES_BLOCK_SIZE]

uint8_t	hash_ [SHA256_DIGEST_LENGTH]

bool	is_pinned_
	is_pinned_ is true while the block is pinned by a client. More...

bool	in_write_

bool	is_deleted_
	True if the block is deleted by the client. More...

boost::condition_variable	write_complete_cv_

bool	client_local_

Friends
class	BufferedBlockMgr

Detailed Description

A fixed-size block of data that may be be persisted to disk. The state of the block is maintained by the block manager and is described by 3 bools: is_pinned_ = True if the block is pinned. The block has a non-null buffer_desc_, buffer_desc_ cannot be in the free buffer list and the block cannot be in unused_blocks_ or unpinned_blocks_. Newly allocated blocks are pinned. in_write_ = True if a write has been issued but not completed for this block. The block cannot be in the unpinned_blocks_ and must have a non-null buffer_desc_ that's not in the free buffer list. It may be pinned or unpinned. is_deleted_ = True if Delete() has been called on a block. After this, no API call is valid on the block. Pin() and Unpin() can be invoked on a block any number of times before Delete(). When a pinned block is unpinned for the first time, it is added to the unpinned_blocks_ list and its buffer is removed from the free list. If it is pinned or deleted at any time while it is on the unpinned list, it is simply removed from that list. When it is dequeued from that list and enqueued for writing, in_write_ is set to true. The block may be pinned, unpinned or deleted while in_write_ is true. After the write has completed, the block's buffer will be returned to the free buffer list if it is no longer pinned, and the block itself will be put on the unused blocks list if Delete() was called. A block MUST have a non-null buffer_desc_ if a) is_pinned_ is true (i.e. the client is using it), or b) in_write_ is true, (i.e. IO mgr is using it), or c) It is on the unpinned list (buffer has not been persisted.) In addition to the block manager API, Block exposes Allocate(), ReturnAllocation() and BytesRemaining() to allocate and free memory within a block, and buffer() and valid_data_len() to read/write the contents of a block. These are not thread-safe.

Definition at line 136 of file buffered-block-mgr.h.

Constructor & Destructor Documentation

impala::BufferedBlockMgr::Block::Block ( BufferedBlockMgr * block_mgr )

private

Definition at line 113 of file buffered-block-mgr.cc.

Member Function Documentation

void impala::BufferedBlockMgr::Block::AddRow ( )

inline

Definition at line 156 of file buffered-block-mgr.h.

References num_rows_.

Referenced by impala::BufferedTupleStream::AllocateRow(), and impala::BufferedTupleStream::DeepCopyInternal().

template<typename T >

T* impala::BufferedBlockMgr::Block::Allocate ( int size )

inline

Allocates the specified number of bytes from this block.

Definition at line 160 of file buffered-block-mgr.h.

References impala::BufferedBlockMgr::BufferDescriptor::buffer, buffer_desc_, BytesRemaining(), and valid_data_len_.

Referenced by impala::Sorter::Run::AddBatch(), impala::BufferedBlockMgrTest::AllocateBlocks(), impala::BufferedTupleStream::AllocateRow(), impala::BufferedTupleStream::DeepCopyInternal(), impala::HashTable::GrowNodeArray(), impala::BufferedBlockMgrTest::MakeRandomSizeData(), impala::BufferedTupleStream::NewBlockForWrite(), and impala::Sorter::Run::UnpinAllBlocks().

uint8_t* impala::BufferedBlockMgr::Block::buffer ( ) const

inline

Pointer to start of the block data in memory. Only guaranteed to be valid if the block is pinned.

Definition at line 181 of file buffered-block-mgr.h.

References impala::BufferedBlockMgr::BufferDescriptor::buffer, and buffer_desc_.

Referenced by impala::BufferedBlockMgr::Decrypt(), impala::BufferedTupleStream::DeepCopyInternal(), impala::BufferedBlockMgr::Encrypt(), impala::Sorter::Run::GetNext(), impala::BufferedTupleStream::NewBlockForWrite(), impala::BufferedBlockMgr::PinBlock(), impala::BufferedBlockMgr::SetHash(), impala::TEST_F(), impala::Sorter::Run::UnpinAllBlocks(), impala::BufferedBlockMgrTest::ValidateBlock(), impala::BufferedBlockMgrTest::ValidateRandomSizeData(), impala::BufferedBlockMgr::VerifyHash(), and impala::BufferedBlockMgr::WriteUnpinnedBlock().

int64_t impala::BufferedBlockMgr::Block::buffer_len ( ) const

inline

Returns the length of the underlying buffer. Only callable if the block is pinned.

Definition at line 191 of file buffered-block-mgr.h.

References buffer_desc_, is_pinned(), and impala::BufferedBlockMgr::BufferDescriptor::len.

int impala::BufferedBlockMgr::Block::BytesRemaining ( ) const

inline

Return the number of remaining bytes that can be allocated in this block.

Definition at line 168 of file buffered-block-mgr.h.

References buffer_desc_, impala::BufferedBlockMgr::BufferDescriptor::len, and valid_data_len_.

Referenced by impala::Sorter::Run::AddBatch(), Allocate(), impala::BufferedTupleStream::AllocateRow(), impala::BufferedTupleStream::DeepCopyInternal(), impala::TEST_F(), and impala::Sorter::Run::UnpinAllBlocks().

string impala::BufferedBlockMgr::Block::DebugString ( ) const

Debug helper method to print the state of a block.

Definition at line 168 of file buffered-block-mgr.cc.

Referenced by impala::BufferedTupleStream::DeepCopyInternal(), impala::BufferedBlockMgr::DeleteBlock(), impala::BufferedBlockMgr::FindBufferForBlock(), impala::BufferedBlockMgr::GetNewBlock(), impala::BufferedBlockMgr::PinBlock(), impala::BufferedBlockMgr::ReturnUnusedBlock(), impala::BufferedBlockMgr::UnpinBlock(), impala::BufferedBlockMgr::Validate(), impala::BufferedBlockMgr::WriteComplete(), and impala::BufferedBlockMgr::WriteUnpinnedBlock().

Status impala::BufferedBlockMgr::Block::Delete ( )

Delete a block. Its buffer is released and on-disk location can be over-written. Non-blocking.

Definition at line 130 of file buffered-block-mgr.cc.

Referenced by impala::Sorter::Run::DeleteAllBlocks(), impala::BufferedBlockMgr::DeleteOrUnpinBlock(), impala::Sorter::MergeIntermediateRuns(), impala::BufferedTupleStream::NextBlockForRead(), impala::Sorter::SortRun(), impala::TEST_F(), and impala::Sorter::Run::UnpinAllBlocks().

void impala::BufferedBlockMgr::Block::Init ( )

private

Initialize the state of a block and set the number of bytes allocated to 0.

Definition at line 134 of file buffered-block-mgr.cc.

Referenced by impala::BufferedBlockMgr::GetUnusedBlock(), and impala::BufferedBlockMgr::ReturnUnusedBlock().

bool impala::BufferedBlockMgr::Block::is_max_size ( ) const

inline

Returns true if this block is the max block size. Only callable if the block is pinned.

Definition at line 198 of file buffered-block-mgr.h.

References block_mgr_, buffer_desc_, is_pinned(), impala::BufferedBlockMgr::BufferDescriptor::len, and impala::BufferedBlockMgr::max_block_size().

Referenced by impala::BufferedBlockMgr::DeleteBlock(), impala::BufferedTupleStream::NewBlockForWrite(), impala::BufferedTupleStream::NextBlockForRead(), and impala::BufferedTupleStream::UnpinBlock().

bool impala::BufferedBlockMgr::Block::is_pinned ( ) const

inline

Definition at line 203 of file buffered-block-mgr.h.

References is_pinned_.

Referenced by impala::BufferedTupleStream::AllocateRow(), buffer_len(), impala::BufferedTupleStream::DeepCopyInternal(), impala::BufferedBlockMgr::GetNewBlock(), is_max_size(), impala::BufferedTupleStream::NewBlockForWrite(), impala::BufferedTupleStream::PrepareForRead(), impala::TEST_F(), impala::BufferedTupleStream::UnpinBlock(), and impala::BufferedTupleStream::UnpinStream().

template<typename T>

T* impala::InternalQueue< T >::Node::Next ( ) const

inlineinherited

Returns the Next/Prev node or NULL if this is the end/front.

Definition at line 48 of file internal-queue.h.

References impala::InternalQueue< T >::lock_, impala::InternalQueue< T >::Node::next, and impala::InternalQueue< T >::Node::parent_queue.

Referenced by impala::TEST(), and impala::BufferedBlockMgr::Validate().

int impala::BufferedBlockMgr::Block::num_rows ( ) const

inline

Definition at line 157 of file buffered-block-mgr.h.

References num_rows_.

Referenced by impala::BufferedTupleStream::NewBlockForWrite().

Status impala::BufferedBlockMgr::Block::Pin	(	bool *	pinned,
		Block *	release_block = `NULL`,
		bool	unpin = `true`
	)

Pins a block in memory–assigns a free buffer to a block and reads it from disk if necessary. If there are no free blocks and no unpinned blocks, '*pinned' is set to false and the block is not pinned. If 'release_block' is non-NULL, if there is memory pressure, this block will be pinned using the buffer from 'release_block'. If 'unpin' is true, 'release_block' will be unpinned (regardless of whether or not the buffer was used for this block). If 'unpin' is false, 'release_block' is deleted. 'release_block' must be pinned.

Definition at line 122 of file buffered-block-mgr.cc.

Referenced by impala::Sorter::Run::GetNext().

template<typename T>

T* impala::InternalQueue< T >::Node::Prev ( ) const

inlineinherited

Definition at line 52 of file internal-queue.h.

References impala::InternalQueue< T >::lock_, impala::InternalQueue< T >::Node::parent_queue, and impala::InternalQueue< T >::Node::prev.

Referenced by impala::TEST().

void impala::BufferedBlockMgr::Block::ReturnAllocation ( int size )

inline

Return size bytes from the most recent allocation.

Definition at line 174 of file buffered-block-mgr.h.

References valid_data_len_.

Referenced by impala::Sorter::Run::AddBatch(), and impala::BufferedTupleStream::DeepCopyInternal().

Status impala::BufferedBlockMgr::Block::Unpin ( )

Unpins a block by adding it to the list of unpinned blocks maintained by the block manager. An unpinned block must be flushed before its buffer is released or assigned to a different block. Is non-blocking.

Definition at line 126 of file buffered-block-mgr.cc.

Referenced by impala::BufferedBlockMgr::DeleteOrUnpinBlock(), impala::BufferedBlockMgr::GetNewBlock(), impala::Sorter::MergeIntermediateRuns(), impala::TEST_F(), impala::Sorter::Run::UnpinAllBlocks(), and impala::BufferedTupleStream::UnpinBlock().

int64_t impala::BufferedBlockMgr::Block::valid_data_len ( ) const

inline

Return the number of bytes allocated in this block.

Definition at line 187 of file buffered-block-mgr.h.

References valid_data_len_.

Referenced by impala::Sorter::Run::GetNext(), impala::Sorter::MergeIntermediateRuns(), impala::Sorter::SortRun(), impala::Sorter::Run::UnpinAllBlocks(), impala::BufferedBlockMgrTest::ValidateBlock(), and impala::BufferedBlockMgrTest::ValidateRandomSizeData().

bool impala::BufferedBlockMgr::Block::Validate ( ) const

private

Debug helper method to validate the state of a block. block_mgr_ lock must already be taken.

Definition at line 144 of file buffered-block-mgr.cc.

References impala::BufferedBlockMgr::DebugString().

Referenced by impala::BufferedBlockMgr::DeleteBlock(), impala::BufferedBlockMgr::FindBufferForBlock(), impala::BufferedBlockMgr::GetNewBlock(), impala::BufferedBlockMgr::UnpinBlock(), impala::BufferedBlockMgr::Validate(), and impala::BufferedBlockMgr::WriteUnpinnedBlock().

Friends And Related Function Documentation

friend class BufferedBlockMgr

friend

Definition at line 209 of file buffered-block-mgr.h.

Member Data Documentation

BufferedBlockMgr* impala::BufferedBlockMgr::Block::block_mgr_

private

Parent block manager object. Responsible for maintaining the state of the block.

Definition at line 225 of file buffered-block-mgr.h.

Referenced by is_max_size().

BufferDescriptor* impala::BufferedBlockMgr::Block::buffer_desc_

private

Pointer to the buffer associated with the block. NULL if the block is not in memory and cannot be changed while the block is pinned or being written.

Definition at line 222 of file buffered-block-mgr.h.

Client* impala::BufferedBlockMgr::Block::client_

private

The client that owns this block.

Definition at line 228 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::DeleteBlock(), impala::BufferedBlockMgr::FindBufferForBlock(), impala::BufferedBlockMgr::GetNewBlock(), impala::BufferedBlockMgr::GetUnusedBlock(), impala::BufferedBlockMgr::PinBlock(), impala::BufferedBlockMgr::UnpinBlock(), and impala::BufferedBlockMgr::WriteComplete().

bool impala::BufferedBlockMgr::Block::client_local_

private

If true, this block is being written out so the underlying buffer can be transferred to another block from the same client. We don't want this buffer getting picked up by another client.

Definition at line 288 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::TransferBuffer(), impala::BufferedBlockMgr::WriteComplete(), and impala::BufferedBlockMgr::WriteUnpinnedBlocks().

boost::scoped_array<uint8_t> impala::BufferedBlockMgr::Block::encrypted_write_buffer_

private

If encryption_ is on, in the write path we allocate a new buffer to hold encrypted data while it's being written to disk. The read path, having no references to the data, can be decrypted in place.

Definition at line 247 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::Encrypt(), impala::BufferedBlockMgr::EncryptDone(), and impala::BufferedBlockMgr::SetHash().

uint8_t impala::BufferedBlockMgr::Block::hash_[SHA256_DIGEST_LENGTH]

private

If integrity_ is on, our SHA256 hash of the data being written. Filled in on writes; verified on reads. This is calculated after encryption.

Definition at line 263 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::SetHash(), and impala::BufferedBlockMgr::VerifyHash().

bool impala::BufferedBlockMgr::Block::in_write_

private

in_write_ is set to true when the block is enqueued for writing via DiskIoMgr, and set to false when the write is complete.

Definition at line 274 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::DeleteBlock(), impala::BufferedBlockMgr::Encrypt(), impala::BufferedBlockMgr::FindBufferForBlock(), impala::BufferedBlockMgr::PinBlock(), impala::BufferedBlockMgr::TransferBuffer(), impala::BufferedBlockMgr::UnpinBlock(), impala::BufferedBlockMgr::Validate(), impala::BufferedBlockMgr::WriteComplete(), and impala::BufferedBlockMgr::WriteUnpinnedBlock().

bool impala::BufferedBlockMgr::Block::is_deleted_

private

True if the block is deleted by the client.

Definition at line 277 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::DeleteBlock(), impala::BufferedBlockMgr::FindBufferForBlock(), impala::BufferedBlockMgr::GetNewBlock(), impala::BufferedBlockMgr::PinBlock(), impala::BufferedBlockMgr::ReturnUnusedBlock(), impala::BufferedBlockMgr::TransferBuffer(), impala::BufferedBlockMgr::UnpinBlock(), and impala::BufferedBlockMgr::WriteComplete().

bool impala::BufferedBlockMgr::Block::is_pinned_

private

is_pinned_ is true while the block is pinned by a client.

Block state variables. The block's buffer can be freed only if is_pinned_ and in_write_ are both false. TODO: this might be better expressed as an enum.

Definition at line 270 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::DeleteBlock(), impala::BufferedBlockMgr::Encrypt(), impala::BufferedBlockMgr::FindBufferForBlock(), impala::BufferedBlockMgr::GetNewBlock(), is_pinned(), impala::BufferedBlockMgr::PinBlock(), impala::BufferedBlockMgr::ReturnUnusedBlock(), impala::BufferedBlockMgr::TransferBuffer(), impala::BufferedBlockMgr::UnpinBlock(), impala::BufferedBlockMgr::Validate(), impala::BufferedBlockMgr::WriteComplete(), and impala::BufferedBlockMgr::WriteUnpinnedBlock().

uint8_t impala::BufferedBlockMgr::Block::iv_[AES_BLOCK_SIZE]

private

If encryption_ is on, the IV to use. IV stands for Initialization Vector, and is used as an input to the cipher as the "block to supply before the first block of plaintext". This is required because all ciphers (except the weak ECB) are built such that each block depends on the output from the previous block. Since the first block doesn't have a previous block, we supply this IV. Think of it as starting off the chain of encryption. This IV is also regenerated on each write.

Definition at line 259 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::Decrypt(), and impala::BufferedBlockMgr::Encrypt().

uint8_t impala::BufferedBlockMgr::Block::key_[32]

private

If encryption_ is on, a AES 256-bit key. Regenerated on each write.

Definition at line 250 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::Decrypt(), and impala::BufferedBlockMgr::Encrypt().

int impala::BufferedBlockMgr::Block::num_rows_

private

Number of rows in this block.

Definition at line 242 of file buffered-block-mgr.h.

Referenced by AddRow(), and num_rows().

int64_t impala::BufferedBlockMgr::Block::valid_data_len_

private

Length of valid (i.e. allocated) data within the block.

Definition at line 239 of file buffered-block-mgr.h.

Referenced by Allocate(), BytesRemaining(), impala::BufferedBlockMgr::Decrypt(), impala::BufferedBlockMgr::Encrypt(), impala::BufferedBlockMgr::PinBlock(), ReturnAllocation(), impala::BufferedBlockMgr::SetHash(), valid_data_len(), impala::BufferedBlockMgr::VerifyHash(), and impala::BufferedBlockMgr::WriteUnpinnedBlock().

boost::condition_variable impala::BufferedBlockMgr::Block::write_complete_cv_

private

Condition variable for when there is a specific client waiting for this block. Only used if client_local_ is true. TODO: Currently we use block_mgr_->lock_ for this condvar. There is no reason to use that lock_ that is already overloaded, see IMPALA-1883.

Definition at line 283 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::TransferBuffer(), and impala::BufferedBlockMgr::WriteComplete().

DiskIoMgr::WriteRange* impala::BufferedBlockMgr::Block::write_range_

private

WriteRange object representing the on-disk location used to persist a block. Is created the first time a block is persisted, and retained until the block object is destroyed. The file location and offset in write_range_ are valid throughout the lifetime of this object, but the data and length in the write_range_ are only valid while the block is being written. write_range_ instance is owned by the block manager.

Definition at line 236 of file buffered-block-mgr.h.

Referenced by impala::BufferedBlockMgr::PinBlock(), and impala::BufferedBlockMgr::WriteUnpinnedBlock().

The documentation for this class was generated from the following files:

be/src/runtime/buffered-block-mgr.h
be/src/runtime/buffered-block-mgr.cc

Public Member Functions

Private Member Functions

Private Attributes

Friends

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Friends And Related Function Documentation

Member Data Documentation