Collaboration diagram for com.cloudera.impala.catalog.TableLoadingMgr:

Classes
class	LoadRequest

Public Member Functions
	TableLoadingMgr (CatalogServiceCatalog catalog, int numLoadingThreads)

void	prioritizeLoad (TTableName tblName)

void	backgroundLoad (TTableName tblName)

void	watchCacheDirs (List< Long > cacheDirIds, final TTableName tblName)

LoadRequest	loadAsync (final TTableName tblName, final Table previousTbl) throws DatabaseNotFoundException

Package Attributes
ExecutorService	asyncRefreshThread_ = Executors.newSingleThreadExecutor()

Private Member Functions
void	startTableLoadingThreads ()

void	loadNextTable () throws InterruptedException

void	execAsyncRefreshWork (TTableName tblName)

boolean	waitForCacheDirs (TTableName tblName)

Private Attributes
final LinkedBlockingDeque < TTableName >	tableLoadingDeque_

final Set< TTableName >	tableLoadingSet_

final ConcurrentHashMap < TTableName, FutureTask < Table > >	loadingTables_

final Map< TTableName, List < Long > >	pendingTableCacheDirs_ = Maps.newHashMap()

final int	numLoadingThreads_

final ExecutorService	tblLoadingPool_

final LinkedBlockingQueue < TTableName >	refreshThreadWork_

final CatalogServiceCatalog	catalog_

final TableLoader	tblLoader_

Static Private Attributes
static final Logger	LOG = Logger.getLogger(TableLoadingMgr.class)

Detailed Description

Class that manages scheduling the loading of table metadata from the Hive Metastore and the Hadoop NameNode. Loads tables using a pool of table loading threads. New load requests can be submitted using loadAsync(), which will schedule the load when the next thread becomes available. Also manages prioritized background table loading by reading from a deque of table names to determine which table to load next. Tables added to the head of the deque will be loaded before tables added to the tail, so the loading order can be prioritized (see prioritizeLoad()/backgroundLoad()).

Definition at line 47 of file TableLoadingMgr.java.

Constructor & Destructor Documentation

com.cloudera.impala.catalog.TableLoadingMgr.TableLoadingMgr	(	CatalogServiceCatalog	catalog,
		int	numLoadingThreads
	)

inline

Definition at line 151 of file TableLoadingMgr.java.

References com.cloudera.impala.catalog.TableLoadingMgr.catalog_, com.cloudera.impala.catalog.TableLoadingMgr.execAsyncRefreshWork(), com.cloudera.impala.catalog.TableLoadingMgr.numLoadingThreads_, com.cloudera.impala.catalog.TableLoadingMgr.refreshThreadWork_, com.cloudera.impala.catalog.TableLoadingMgr.startTableLoadingThreads(), com.cloudera.impala.catalog.TableLoadingMgr.tblLoader_, and com.cloudera.impala.catalog.TableLoadingMgr.tblLoadingPool_.

Member Function Documentation

void com.cloudera.impala.catalog.TableLoadingMgr.backgroundLoad ( TTableName tblName )

inline

Submits a single table for background (low priority) loading.

Definition at line 182 of file TableLoadingMgr.java.

References com.cloudera.impala.catalog.TableLoadingMgr.tableLoadingSet_.

void com.cloudera.impala.catalog.TableLoadingMgr.execAsyncRefreshWork ( TTableName tblName )

inlineprivate

Executes all async refresh work for the specified table name.

Definition at line 296 of file TableLoadingMgr.java.

References com.cloudera.impala.catalog.TableLoadingMgr.waitForCacheDirs().

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.TableLoadingMgr().

LoadRequest com.cloudera.impala.catalog.TableLoadingMgr.loadAsync	(	final TTableName	tblName,
		final Table	previousTbl
	)		throws DatabaseNotFoundException

inline

Loads a table asynchronously, returning a LoadRequest that can be used to get the result (a Table). If there is already a load in flight for this table name, the same underlying loading task (Future) will be used, helping to prevent duplicate loads of the same table. Can also be used to perform an incremental refresh of an existing table, by passing the previous Table value in previousTbl. This may speedup the loading process, but may return a stale object.

Definition at line 221 of file TableLoadingMgr.java.

void com.cloudera.impala.catalog.TableLoadingMgr.loadNextTable ( ) throws InterruptedException

inlineprivate

Gets the next table name to load off the head of the table loading queue. If the queue is empty, this will block until a new table is added.

Definition at line 278 of file TableLoadingMgr.java.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.startTableLoadingThreads().

void com.cloudera.impala.catalog.TableLoadingMgr.prioritizeLoad ( TTableName tblName )

inline

Prioritizes the loading of the given table.

Definition at line 174 of file TableLoadingMgr.java.

void com.cloudera.impala.catalog.TableLoadingMgr.startTableLoadingThreads ( )

inlineprivate

Starts table loading threads in a fixed sized thread pool with a size defined by NUM_TBL_LOADING_THREADS. Each thread polls the tableLoadingDeque_ for new tables to load.

Definition at line 251 of file TableLoadingMgr.java.

References com.cloudera.impala.catalog.TableLoadingMgr.loadNextTable(), and com.cloudera.impala.catalog.TableLoadingMgr.numLoadingThreads_.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.TableLoadingMgr().

boolean com.cloudera.impala.catalog.TableLoadingMgr.waitForCacheDirs ( TTableName tblName )

inlineprivate

Waits for all pending cache directives on a table to complete. Returns true if a refresh is needed and false if a refresh is not needed.

Definition at line 310 of file TableLoadingMgr.java.

References com.cloudera.impala.catalog.TableLoadingMgr.pendingTableCacheDirs_.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.execAsyncRefreshWork().

void com.cloudera.impala.catalog.TableLoadingMgr.watchCacheDirs	(	List< Long >	cacheDirIds,
		final TTableName	tblName
	)

inline

Adds a list of cache directive IDs to watch for the given table name. The asyncRefreshThread_ will process the cache directives and once all directives complete (data has been cached or no progress is being made), the asyncRefreshThread_ will refresh the table metadata. After processing the request the watch will be deleted.

Definition at line 197 of file TableLoadingMgr.java.

References com.cloudera.impala.catalog.TableLoadingMgr.pendingTableCacheDirs_.

Member Data Documentation

ExecutorService com.cloudera.impala.catalog.TableLoadingMgr.asyncRefreshThread_ = Executors.newSingleThreadExecutor()

package

Definition at line 141 of file TableLoadingMgr.java.

final CatalogServiceCatalog com.cloudera.impala.catalog.TableLoadingMgr.catalog_

private

Definition at line 148 of file TableLoadingMgr.java.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.LoadRequest.get(), and com.cloudera.impala.catalog.TableLoadingMgr.TableLoadingMgr().

final ConcurrentHashMap<TTableName, FutureTask<Table> > com.cloudera.impala.catalog.TableLoadingMgr.loadingTables_

private

Initial value:

=

new ConcurrentHashMap<TTableName, FutureTask<Table>>()

Definition at line 113 of file TableLoadingMgr.java.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.LoadRequest.close().

final Logger com.cloudera.impala.catalog.TableLoadingMgr.LOG = Logger.getLogger(TableLoadingMgr.class)

staticprivate

Definition at line 91 of file TableLoadingMgr.java.

final int com.cloudera.impala.catalog.TableLoadingMgr.numLoadingThreads_

private

Definition at line 127 of file TableLoadingMgr.java.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.startTableLoadingThreads(), and com.cloudera.impala.catalog.TableLoadingMgr.TableLoadingMgr().

final Map<TTableName, List<Long> > com.cloudera.impala.catalog.TableLoadingMgr.pendingTableCacheDirs_ = Maps.newHashMap()

private

Definition at line 122 of file TableLoadingMgr.java.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.waitForCacheDirs(), and com.cloudera.impala.catalog.TableLoadingMgr.watchCacheDirs().

final LinkedBlockingQueue<TTableName> com.cloudera.impala.catalog.TableLoadingMgr.refreshThreadWork_

private

Initial value:

=

new LinkedBlockingQueue<TTableName>()

Definition at line 145 of file TableLoadingMgr.java.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.TableLoadingMgr().

final LinkedBlockingDeque<TTableName> com.cloudera.impala.catalog.TableLoadingMgr.tableLoadingDeque_

private

Initial value:

=

new LinkedBlockingDeque<TTableName>()

Definition at line 99 of file TableLoadingMgr.java.

final Set<TTableName> com.cloudera.impala.catalog.TableLoadingMgr.tableLoadingSet_

private

Initial value:

=

Collections.synchronizedSet(new HashSet<TTableName>())

Definition at line 108 of file TableLoadingMgr.java.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.backgroundLoad().

final TableLoader com.cloudera.impala.catalog.TableLoadingMgr.tblLoader_

private

Definition at line 149 of file TableLoadingMgr.java.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.TableLoadingMgr().

final ExecutorService com.cloudera.impala.catalog.TableLoadingMgr.tblLoadingPool_

private

Definition at line 133 of file TableLoadingMgr.java.

Referenced by com.cloudera.impala.catalog.TableLoadingMgr.TableLoadingMgr().

The documentation for this class was generated from the following file:

fe/src/main/java/com/cloudera/impala/catalog/TableLoadingMgr.java

Classes

Public Member Functions

Package Attributes

Private Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation