impala::HdfsFsCache Class Reference

#include <hdfs-fs-cache.h>

Collaboration diagram for impala::HdfsFsCache:

Public Types
typedef boost::unordered_map < std::string, hdfsFS >	HdfsFsMap

Public Member Functions
Status	GetLocalConnection (hdfsFS *fs)
	Get connection to the local filesystem. More...
Status	GetConnection (const std::string &path, hdfsFS *fs, HdfsFsMap *local_cache=NULL)

Static Public Member Functions
static HdfsFsCache *	instance ()
static void	Init ()
	Initializes the cache. Must be called before any other APIs. More...
static string	GetNameNodeFromPath (const string &path, string *err)

Private Member Functions
	HdfsFsCache ()
	HdfsFsCache (HdfsFsCache const &l)
HdfsFsCache &	operator= (HdfsFsCache const &l)

Private Attributes
boost::mutex	lock_
HdfsFsMap	fs_map_

Static Private Attributes
static boost::scoped_ptr < HdfsFsCache >	instance_
	Singleton instance. Instantiated in Init(). More...

Detailed Description

A (process-wide) cache of hdfsFS objects. These connections are shared across all threads and kept open until the process terminates. These connections are leaked, i.e. we never call hdfsDisconnect(). Calls to hdfsDisconnect() by individual threads would terminate all other connections handed out via hdfsConnect() to the same URI, and there is no simple, safe way to call hdfsDisconnect() when process terminates (the proper solution is likely to create a signal handler to detect when the process is killed, but we would still leak when impalad crashes).

Definition at line 39 of file hdfs-fs-cache.h.

Member Typedef Documentation

typedef boost::unordered_map<std::string, hdfsFS> impala::HdfsFsCache::HdfsFsMap

Definition at line 41 of file hdfs-fs-cache.h.

Constructor & Destructor Documentation

impala::HdfsFsCache::HdfsFsCache ( )

inlineprivate

Definition at line 69 of file hdfs-fs-cache.h.

impala::HdfsFsCache::HdfsFsCache ( HdfsFsCache const &  l )

private

Member Function Documentation

Status impala::HdfsFsCache::GetConnection	(	const std::string &	path,
		hdfsFS *	fs,
		HdfsFsMap *	local_cache = NULL
	)

Get connection to specific fs by specifying a path. Optionally, a local cache can be provided so that the process-wide lock can be avoided on subsequent calls for the same filesystem. The caller is responsible for synchronizing the local cache (e.g. by passing a thread-local cache).

Definition at line 39 of file hdfs-fs-cache.cc.

References impala::GetHdfsErrorMsg(), lock_, and impala::OK.

Status impala::HdfsFsCache::GetLocalConnection

(

hdfsFS *

fs

)

Get connection to the local filesystem.

Definition at line 78 of file hdfs-fs-cache.cc.

string impala::HdfsFsCache::GetNameNodeFromPath ( const string &  path, string *  err  )

static

Get NameNode info from path, set error message if path is not valid. Exposed as a static method for testing purpose.

Definition at line 82 of file hdfs-fs-cache.cc.

void impala::HdfsFsCache::Init ( )

static

Initializes the cache. Must be called before any other APIs.

Definition at line 34 of file hdfs-fs-cache.cc.

Referenced by impala::InitCommonRuntime().

static HdfsFsCache* impala::HdfsFsCache::instance ( )

inlinestatic

Definition at line 43 of file hdfs-fs-cache.h.

References instance_.

Referenced by impala::HdfsScanNode::Prepare(), and impala::HdfsTableSink::Prepare().

HdfsFsCache& impala::HdfsFsCache::operator= ( HdfsFsCache const &  l )

private

Member Data Documentation

HdfsFsMap impala::HdfsFsCache::fs_map_

private

Definition at line 67 of file hdfs-fs-cache.h.

scoped_ptr< HdfsFsCache > impala::HdfsFsCache::instance_

staticprivate

Singleton instance. Instantiated in Init().

Definition at line 64 of file hdfs-fs-cache.h.

Referenced by instance().

boost::mutex impala::HdfsFsCache::lock_

private

Definition at line 66 of file hdfs-fs-cache.h.

---

The documentation for this class was generated from the following files:

• be/src/runtime/hdfs-fs-cache.h
• be/src/runtime/hdfs-fs-cache.cc