#include <data-stream-mgr.h>

Collaboration diagram for impala::DataStreamMgr:

Classes
struct	ComparisonOp
	less-than ordering for pair<TUniqueId, PlanNodeId> More...

Public Member Functions
	DataStreamMgr ()

boost::shared_ptr < DataStreamRecvr >	CreateRecvr (RuntimeState state, const RowDescriptor &row_desc, const TUniqueId &fragment_instance_id, PlanNodeId dest_node_id, int num_senders, int buffer_size, RuntimeProfile profile, bool is_merging)

Status	AddData (const TUniqueId &fragment_instance_id, PlanNodeId dest_node_id, const TRowBatch &thrift_batch, int sender_id)

Status	CloseSender (const TUniqueId &fragment_instance_id, PlanNodeId dest_node_id, int sender_id)

void	Cancel (const TUniqueId &fragment_instance_id)
	Closes all receivers registered for fragment_instance_id immediately. More...

Private Types
typedef boost::unordered_multimap < uint32_t, boost::shared_ptr < DataStreamRecvr > >	StreamMap

typedef std::set< std::pair < TUniqueId, PlanNodeId > , ComparisonOp >	FragmentStreamSet
	ordered set of registered streams' fragment instance id/node id More...

Private Member Functions
boost::shared_ptr < DataStreamRecvr >	FindRecvr (const TUniqueId &fragment_instance_id, PlanNodeId node_id, bool acquire_lock=true)

Status	DeregisterRecvr (const TUniqueId &fragment_instance_id, PlanNodeId node_id)
	Remove receiver block for fragment_instance_id/node_id from the map. More...

uint32_t	GetHashValue (const TUniqueId &fragment_instance_id, PlanNodeId node_id)

Private Attributes
boost::mutex	lock_
	protects all fields below More...

StreamMap	receiver_map_

FragmentStreamSet	fragment_stream_set_

Friends
class	DataStreamRecvr

Detailed Description

Singleton class which manages all incoming data streams at a backend node. It provides both producer and consumer functionality for each data stream.

ImpalaBackend service threads use this to add incoming data to streams in response to TransmitData rpcs (AddData()) or to signal end-of-stream conditions (CloseSender()).
Exchange nodes extract data from an incoming stream via a DataStreamRecvr, which is created with CreateRecvr(). DataStreamMgr also allows asynchronous cancellation of streams via Cancel() which unblocks all DataStreamRecvr::GetBatch() calls that are made on behalf of the cancelled fragment id. TODO: The recv buffers used in DataStreamRecvr should count against per-query memory limits.

Definition at line 56 of file data-stream-mgr.h.

Member Typedef Documentation

typedef std::set<std::pair<TUniqueId, PlanNodeId>, ComparisonOp > impala::DataStreamMgr::FragmentStreamSet

private

ordered set of registered streams' fragment instance id/node id

Definition at line 126 of file data-stream-mgr.h.

typedef boost::unordered_multimap<uint32_t, boost::shared_ptr<DataStreamRecvr> > impala::DataStreamMgr::StreamMap

private

map from hash value of fragment instance id/node id pair to stream receivers; Ownership of the stream revcr is shared between this instance and the caller of CreateRecvr(). we don't want to create a map<pair<TUniqueId, PlanNodeId>, DataStreamRecvr*>, because that requires a bunch of copying of ids for lookup

Definition at line 105 of file data-stream-mgr.h.

Constructor & Destructor Documentation

impala::DataStreamMgr::DataStreamMgr ( )

inline

Definition at line 58 of file data-stream-mgr.h.

Member Function Documentation

Status impala::DataStreamMgr::AddData	(	const TUniqueId &	fragment_instance_id,
		PlanNodeId	dest_node_id,
		const TRowBatch &	thrift_batch,
		int	sender_id
	)

Adds a row batch to the recvr identified by fragment_instance_id/dest_node_id if the recvr has not been cancelled. sender_id identifies the sender instance from which the data came. The call blocks if this ends up pushing the stream over its buffering limit; it unblocks when the consumer removed enough data to make space for row_batch. TODO: enforce per-sender quotas (something like 200% of buffer_size/#senders), so that a single sender can't flood the buffer and stall everybody else. Returns OK if successful, error status otherwise.

Definition at line 85 of file data-stream-mgr.cc.

References impala::OK, and VLOG_ROW.

Referenced by impala::ImpalaServer::TransmitData().

void impala::DataStreamMgr::Cancel ( const TUniqueId & fragment_instance_id )

Closes all receivers registered for fragment_instance_id immediately.

Definition at line 155 of file data-stream-mgr.cc.

References lock_, and VLOG_QUERY.

Status impala::DataStreamMgr::CloseSender	(	const TUniqueId &	fragment_instance_id,
		PlanNodeId	dest_node_id,
		int	sender_id
	)

Notifies the recvr associated with the fragment/node id that the specified sender has closed. Returns OK if successful, error status otherwise.

Definition at line 107 of file data-stream-mgr.cc.

References impala::OK, and VLOG_FILE.

Referenced by impala::ImpalaServer::TransmitData().

shared_ptr< DataStreamRecvr > impala::DataStreamMgr::CreateRecvr	(	RuntimeState *	state,
		const RowDescriptor &	row_desc,
		const TUniqueId &	fragment_instance_id,
		PlanNodeId	dest_node_id,
		int	num_senders,
		int	buffer_size,
		RuntimeProfile *	profile,
		bool	is_merging
	)

Create a receiver for a specific fragment_instance_id/node_id destination; If is_merging is true, the receiver maintains a separate queue of incoming row batches for each sender and merges the sorted streams from each sender into a single stream. Ownership of the receiver is shared between this DataStream mgr instance and the caller.

Definition at line 46 of file data-stream-mgr.cc.

References impala::hash_value(), impala::RuntimeState::instance_mem_tracker(), lock_, impala::row_desc(), and VLOG_FILE.

Referenced by impala::ExchangeNode::Prepare().

Status impala::DataStreamMgr::DeregisterRecvr	(	const TUniqueId &	fragment_instance_id,
		PlanNodeId	node_id
	)

private

Remove receiver block for fragment_instance_id/node_id from the map.

Definition at line 126 of file data-stream-mgr.cc.

References impala::hash_value(), lock_, impala::OK, and VLOG_QUERY.

Referenced by impala::DataStreamRecvr::Close().

shared_ptr< DataStreamRecvr > impala::DataStreamMgr::FindRecvr	(	const TUniqueId &	fragment_instance_id,
		PlanNodeId	node_id,
		bool	acquire_lock = `true`
	)

private

Return the receiver for given fragment_instance_id/node_id, or NULL if not found. If 'acquire_lock' is false, assumes lock_ is already being held and won't try to acquire it.

Definition at line 64 of file data-stream-mgr.cc.

References impala::hash_value(), lock_, and VLOG_ROW.

uint32_t impala::DataStreamMgr::GetHashValue	(	const TUniqueId &	fragment_instance_id,
		PlanNodeId	node_id
	)

inlineprivate

Definition at line 38 of file data-stream-mgr.cc.

References impala::TYPE_BIGINT, and impala::TYPE_INT.

Friends And Related Function Documentation

friend class DataStreamRecvr

friend

Definition at line 94 of file data-stream-mgr.h.

Member Data Documentation

FragmentStreamSet impala::DataStreamMgr::fragment_stream_set_

private

Definition at line 127 of file data-stream-mgr.h.

boost::mutex impala::DataStreamMgr::lock_

private

protects all fields below

Definition at line 97 of file data-stream-mgr.h.

StreamMap impala::DataStreamMgr::receiver_map_

private

Definition at line 106 of file data-stream-mgr.h.

The documentation for this class was generated from the following files:

be/src/runtime/data-stream-mgr.h
be/src/runtime/data-stream-mgr.cc

Classes

Public Member Functions

Private Types

Private Member Functions

Private Attributes

Friends

Detailed Description

Member Typedef Documentation

Constructor & Destructor Documentation

Member Function Documentation

Friends And Related Function Documentation

Member Data Documentation