#include <hdfs-bulk-ops.h>

Collaboration diagram for impala::HdfsOperationSet:

Public Types
typedef std::pair< const HdfsOp *, std::string >	Error

typedef std::vector< Error >	Errors

Public Member Functions
	HdfsOperationSet (hdfsFS *hdfs_connection)

void	Add (HdfsOpType op, const std::string &src)

void	Add (HdfsOpType op, const std::string &src, const std::string &dst)
	Add an operation that takes two parameters (e.g. RENAME) More...

void	Add (HdfsOpType op, const std::string &src, short permissions)
	Add an operation that takes a permission argument (i.e. CHMOD) More...

bool	Execute (HdfsOpThreadPool *pool, bool abort_on_error)

const Errors &	errors ()

hdfsFS *	hdfs_connection () const

Private Member Functions
void	MarkOneOpDone ()

void	AddError (const std::string &err, const HdfsOp *op)
	Called by HdfsOp to record an error. More...

bool	ShouldAbort ()

Private Attributes
std::vector< HdfsOp >	ops_
	The set of operations to be submitted to HDFS. More...

Promise< bool >	promise_

AtomicInt< int64_t >	num_ops_

hdfsFS *	hdfs_connection_
	HDFS connection shared between all operations. Not owned by this class. More...

boost::mutex	errors_lock_
	Protects errors_ and abort_on_error_ during Execute. More...

Errors	errors_
	All errors produced during Execute. More...

bool	abort_on_error_
	True if a single error should cause any subsequent operations to become no-ops. More...

Friends
class	HdfsOp

Detailed Description

This class contains a set of operations to be executed in parallel on an HdfsOpThreadPool. These operations may not be executed in the order that they are added.

Definition at line 87 of file hdfs-bulk-ops.h.

Member Typedef Documentation

typedef std::pair<const HdfsOp*, std::string> impala::HdfsOperationSet::Error

Definition at line 109 of file hdfs-bulk-ops.h.

typedef std::vector<Error> impala::HdfsOperationSet::Errors

Definition at line 110 of file hdfs-bulk-ops.h.

Constructor & Destructor Documentation

HdfsOperationSet::HdfsOperationSet ( hdfsFS * hdfs_connection )

Constructs a new operation set. The hdfsFS parameter is shared between all operations, and is not owned by this class (and therefore should remain valid until Execute returns).

Definition at line 125 of file hdfs-bulk-ops.cc.

Member Function Documentation

void impala::HdfsOperationSet::Add	(	HdfsOpType	op,
		const std::string &	src
	)

Add an operation that takes only a single 'src' parameter (e.g. DELETE, CREATE_DIR, DELETE_THEN_CREATE)

void impala::HdfsOperationSet::Add	(	HdfsOpType	op,
		const std::string &	src,
		const std::string &	dst
	)

Add an operation that takes two parameters (e.g. RENAME)

void impala::HdfsOperationSet::Add	(	HdfsOpType	op,
		const std::string &	src,
		short	permissions
	)

Add an operation that takes a permission argument (i.e. CHMOD)

void HdfsOperationSet::AddError	(	const std::string &	err,
		const HdfsOp *	op
	)

private

Called by HdfsOp to record an error.

Definition at line 157 of file hdfs-bulk-ops.cc.

References errors_, and errors_lock_.

Referenced by impala::HdfsOp::Execute().

const Errors& impala::HdfsOperationSet::errors ( )

inline

Returns the (possible zero-length) list of errors during op execution. Not valid until Execute has returned.

Definition at line 114 of file hdfs-bulk-ops.h.

References errors_.

Referenced by MarkOneOpDone().

bool HdfsOperationSet::Execute	(	HdfsOpThreadPool *	pool,
		bool	abort_on_error
	)

Run all operations on the given pool, blocking until all are complete. Returns false if there were any errors, true otherwise. If abort_on_error is true, execution will finish after the first error seen.

Definition at line 129 of file hdfs-bulk-ops.cc.

References abort_on_error_, errors_lock_, impala::Promise< T >::Get(), num_ops_, impala::ThreadPool< T >::Offer(), ops_, and promise_.

hdfsFS* impala::HdfsOperationSet::hdfs_connection ( ) const

inline

Definition at line 116 of file hdfs-bulk-ops.h.

References hdfs_connection_.

Referenced by impala::HdfsOp::Execute().

void HdfsOperationSet::MarkOneOpDone ( )

private

Called by HdfsOp to signal its completion. When the last op has finished, this method signals Execute() so that it can return.

Definition at line 162 of file hdfs-bulk-ops.cc.

References errors(), num_ops_, promise_, impala::Promise< T >::Set(), and impala::AtomicInt< T >::UpdateAndFetch().

Referenced by impala::HdfsOp::Execute().

bool HdfsOperationSet::ShouldAbort ( )

private

Called by HdfsOp at the start of execution to decide whether to continue. Returns true iff abort_on_error_ is true and at least one error has been recorded.

Definition at line 168 of file hdfs-bulk-ops.cc.

References abort_on_error_, errors_, and errors_lock_.

Referenced by impala::HdfsOp::Execute().