com.cloudera.impala.catalog.CatalogServiceCatalog Class Reference

Inheritance diagram for com.cloudera.impala.catalog.CatalogServiceCatalog:

Collaboration diagram for com.cloudera.impala.catalog.CatalogServiceCatalog:

Classes
class	CachePoolReader

Public Member Functions
	CatalogServiceCatalog (boolean loadInBackground, int numLoadingThreads, SentryConfig sentryConfig, TUniqueId catalogServiceId)
void	watchCacheDirs (List< Long > dirIds, TTableName tblName)
void	prioritizeLoad (List< TCatalogObject > objectDescs)
TGetAllCatalogObjectsResponse	getCatalogObjects (long fromVersion)
List< Function >	getFunctions (String dbName) throws DatabaseNotFoundException
void	reset () throws CatalogException
Db	addDb (String dbName) throws ImpalaException
Db	removeDb (String dbName)
Table	addTable (String dbName, String tblName) throws TableNotFoundException
Table	getOrLoadTable (String dbName, String tblName) throws CatalogException
Table	removeTable (String dbName, String tblName)
Function	removeFunction (Function desc)
boolean	addFunction (Function fn)
boolean	addDataSource (DataSource dataSource)
DataSource	removeDataSource (String dataSourceName)
void	updateLastDdlTime (TTableName tblName, long ddlTime)
Table	renameTable (TTableName oldTableName, TTableName newTableName) throws CatalogException
Table	reloadTable (TTableName tblName) throws CatalogException
Table	dropPartition (TableName tableName, List< TPartitionKeyValue > partitionSpec) throws CatalogException
Table	addPartition (HdfsPartition partition) throws CatalogException
boolean	invalidateTable (TTableName tableName, Pair< Db, Table > updatedObjects)
Role	addRole (String roleName, Set< String > grantGroups)
Role	removeRole (String roleName)
Role	addRoleGrantGroup (String roleName, String groupName) throws CatalogException
Role	removeRoleGrantGroup (String roleName, String groupName) throws CatalogException
RolePrivilege	addRolePrivilege (String roleName, TPrivilege thriftPriv) throws CatalogException
RolePrivilege	removeRolePrivilege (String roleName, TPrivilege thriftPriv) throws CatalogException
RolePrivilege	getRolePrivilege (String roleName, TPrivilege privSpec) throws CatalogException
long	incrementAndGetCatalogVersion ()
long	getCatalogVersion ()
TableId	getNextTableId ()
SentryProxy	getSentryProxy ()
AuthorizationPolicy	getAuthPolicy ()
Db	getBuiltinsDb ()
Db	addDb (Db db)
Db	getDb (String dbName)
List< String >	getDbNames (String dbPattern)
Table	getTable (String dbName, String tableName) throws CatalogException
Table	removeTable (TTableName tableName)
List< String >	getTableNames (String dbName, String tablePattern) throws DatabaseNotFoundException
boolean	containsTable (String dbName, String tableName)
DataSource	getDataSource (String dataSourceName)
List< DataSource >	getDataSources ()
List< DataSource >	getDataSources (String pattern)
List< String >	getDataSourceNames (String pattern)
Function	getFunction (Function desc, Function.CompareMode mode)
boolean	containsFunction (FunctionName name)
boolean	addHdfsCachePool (HdfsCachePool cachePool)
HdfsCachePool	getHdfsCachePool (String poolName)
void	close ()
MetaStoreClient	getMetaStoreClient ()
HdfsPartition	getHdfsPartition (String dbName, String tableName, List< TPartitionKeyValue > partitionSpec) throws CatalogException
boolean	containsHdfsPartition (String dbName, String tableName, List< TPartitionKeyValue > partitionSpec) throws CatalogException
TCatalogObject	getTCatalogObject (TCatalogObject objectDesc) throws CatalogException

Static Public Member Functions
static long	getLastDdlTime (org.apache.hadoop.hive.metastore.api.Table msTbl)
static Function	getBuiltin (Function desc, Function.CompareMode mode)

Static Public Attributes
static final long	INITIAL_CATALOG_VERSION = 0L
static final String	DEFAULT_DB = "default"
static final String	BUILTINS_DB = "_impala_builtins"

Protected Attributes
final AtomicInteger	nextTableId_ = new AtomicInteger(0)
final MetaStoreClientPool	metaStoreClientPool_ = new MetaStoreClientPool(0)
AuthorizationPolicy	authPolicy_ = new AuthorizationPolicy()
AtomicReference < ConcurrentHashMap< String, Db > >	dbCache_
final CatalogObjectCache < DataSource >	dataSources_
final CatalogObjectCache < HdfsCachePool >	hdfsCachePools_

Private Member Functions
Table	replaceTableIfUnchanged (Table updatedTbl, long expectedCatalogVersion) throws DatabaseNotFoundException

Private Attributes
final TUniqueId	catalogServiceId_
final ReentrantReadWriteLock	catalogLock_ = new ReentrantReadWriteLock(true)
long	catalogVersion_ = INITIAL_CATALOG_VERSION
final TableLoadingMgr	tableLoadingMgr_
final boolean	loadInBackground_
final ScheduledExecutorService	cachePoolReader_
final SentryProxy	sentryProxy_

Static Private Attributes
static final Logger	LOG = Logger.getLogger(CatalogServiceCatalog.class)

Detailed Description

Specialized Catalog that implements the CatalogService specific Catalog APIs. The CatalogServiceCatalog manages loading of all the catalog metadata and processing of DDL requests. For each DDL request, the CatalogServiceCatalog will return the catalog version that the update will show up in. The client can then wait until the statestore sends an update that contains that catalog version. The CatalogServiceCatalog also manages a global "catalog version". The version is incremented and assigned to a CatalogObject whenever it is added/modified/removed from the catalog. This means each CatalogObject will have a unique version and assigned versions are strictly increasing. Table metadata is loaded in the background by the TableLoadingMgr; tables can be prioritized for loading by calling prioritizeLoad(). Background loading can also be enabled for the catalog, in which case missing tables (tables that are not yet loaded) are submitted to the TableLoadingMgr any table metadata is invalidated and on startup. Accessing a table that is not yet loaded (via getTable()), will load the table's metadata on-demand, out-of-band of the table loading thread pool. TODO: Consider removing on-demand loading and have everything go through the table loading thread pool.

Definition at line 81 of file CatalogServiceCatalog.java.

Constructor & Destructor Documentation

com.cloudera.impala.catalog.CatalogServiceCatalog.CatalogServiceCatalog ( boolean  loadInBackground, int  numLoadingThreads, SentryConfig  sentryConfig, TUniqueId  catalogServiceId  )

inline

Initialize the CatalogServiceCatalog. If loadInBackground is true, table metadata will be loaded in the background

Definition at line 124 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.catalogServiceId_, com.cloudera.impala.catalog.CatalogServiceCatalog.loadInBackground_, com.cloudera.impala.catalog.CatalogServiceCatalog.sentryProxy_, and com.cloudera.impala.catalog.CatalogServiceCatalog.tableLoadingMgr_.

Member Function Documentation

boolean com.cloudera.impala.catalog.CatalogServiceCatalog.addDataSource ( DataSource  dataSource )

inline

Adds a data source to the catalog, incrementing the catalog version. Returns true if the add was successful, false otherwise.

Definition at line 582 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.dataSources_, and com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Db com.cloudera.impala.catalog.Catalog.addDb ( Db  db )

inlineinherited

Adds a new database to the catalog, replacing any existing database with the same name. Returns the previous database with this name, or null if there was no previous database.

Definition at line 107 of file Catalog.java.

References com.cloudera.impala.catalog.Db.getName().

Referenced by com.cloudera.impala.catalog.Catalog.Catalog().

Db com.cloudera.impala.catalog.CatalogServiceCatalog.addDb ( String  dbName ) throws ImpalaException

inline

Adds a database name to the metadata cache and returns the database's new Db object. Used by CREATE DATABASE statements.

Definition at line 435 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.invalidateTable().

boolean com.cloudera.impala.catalog.CatalogServiceCatalog.addFunction ( Function  fn )

inline

Adds a function from the catalog, incrementing the catalog version. Returns true if the add was successful, false otherwise.

Definition at line 567 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Db.addFunction(), com.cloudera.impala.catalog.Catalog.getDb(), com.cloudera.impala.catalog.Function.getFunctionName(), and com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

boolean com.cloudera.impala.catalog.Catalog.addHdfsCachePool ( HdfsCachePool  cachePool )

inlineinherited

Adds a new HdfsCachePool to the catalog.

Definition at line 304 of file Catalog.java.

Table com.cloudera.impala.catalog.CatalogServiceCatalog.addPartition ( HdfsPartition  partition ) throws CatalogException

inline

Adds the partition to its HdfsTable. Returns the table with an updated catalog version.

Definition at line 714 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.getDb(), and com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Role com.cloudera.impala.catalog.CatalogServiceCatalog.addRole ( String  roleName, Set< String >  grantGroups  )

inline

Adds a new role with the given name and grant groups to the AuthorizationPolicy. If a role with the same name already exists it will be overwritten.

Definition at line 817 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Role com.cloudera.impala.catalog.CatalogServiceCatalog.addRoleGrantGroup ( String  roleName, String  groupName  ) throws CatalogException

inline

Adds a grant group to the given role name and returns the modified Role with an updated catalog version. If the role does not exist a CatalogException is thrown.

Definition at line 850 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

RolePrivilege com.cloudera.impala.catalog.CatalogServiceCatalog.addRolePrivilege ( String  roleName, TPrivilege  thriftPriv  ) throws CatalogException

inline

Adds a privilege to the given role name. Returns the new RolePrivilege and increments the catalog version. If the parent role does not exist a CatalogException is thrown.

Definition at line 885 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Table com.cloudera.impala.catalog.CatalogServiceCatalog.addTable ( String  dbName, String  tblName  ) throws TableNotFoundException

inline

Adds a table with the given name to the catalog and returns the new table, loading the metadata if needed.

Definition at line 460 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.getDb(), com.cloudera.impala.catalog.CatalogServiceCatalog.getNextTableId(), and com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.renameTable().

void com.cloudera.impala.catalog.Catalog.close ( )

inlineinherited

Release the Hive Meta Store Client resources. Can be called multiple times (additional calls will be no-ops).

Definition at line 320 of file Catalog.java.

boolean com.cloudera.impala.catalog.Catalog.containsFunction ( FunctionName  name )

inlineinherited

Returns true if there is a function with this function name. Parameters are ignored.

Definition at line 295 of file Catalog.java.

References com.cloudera.impala.analysis.FunctionName.getDb(), and com.cloudera.impala.catalog.Catalog.getDb().

boolean com.cloudera.impala.catalog.Catalog.containsHdfsPartition ( String  dbName, String  tableName, List< TPartitionKeyValue >  partitionSpec  ) throws CatalogException

inlineinherited

Returns true if the table contains the given partition spec, otherwise false. This may trigger a metadata load if the table metadata is not yet cached.

Exceptions

DatabaseNotFoundException	- If the database does not exist.
TableNotFoundException	- If the table does not exist.
TableLoadingException	- If there is an error loading the table metadata.

Definition at line 384 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.getHdfsPartition().

Referenced by com.cloudera.impala.service.CatalogOpExecutor.alterTableAddPartition(), and com.cloudera.impala.service.CatalogOpExecutor.alterTableDropPartition().

boolean com.cloudera.impala.catalog.Catalog.containsTable ( String  dbName, String  tableName  )

inlineinherited

Returns true if the table and the database exist in the Impala Catalog. Returns false if either the table or the database do not exist.

Definition at line 187 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.getDb().

Referenced by com.cloudera.impala.service.CatalogOpExecutor.createTable(), com.cloudera.impala.service.CatalogOpExecutor.createTableLike(), and com.cloudera.impala.service.CatalogOpExecutor.createView().

Table com.cloudera.impala.catalog.CatalogServiceCatalog.dropPartition ( TableName  tableName, List< TPartitionKeyValue >  partitionSpec  ) throws CatalogException

inline

Drops the partition from its HdfsTable. If the HdfsTable does not exist, an exception is thrown. If the partition having the given partition spec does not exist, null is returned. Otherwise, the table with an updated catalog version is returned.

Definition at line 686 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Table.getCatalogVersion(), com.cloudera.impala.catalog.CatalogServiceCatalog.getOrLoadTable(), and com.cloudera.impala.catalog.CatalogServiceCatalog.replaceTableIfUnchanged().

AuthorizationPolicy com.cloudera.impala.catalog.CatalogServiceCatalog.getAuthPolicy ( )

inline

Definition at line 967 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.authPolicy_.

Referenced by com.cloudera.impala.util.SentryProxy.createRole().

static Function com.cloudera.impala.catalog.Catalog.getBuiltin ( Function  desc, Function.CompareMode  mode  )

inlinestaticinherited

Definition at line 276 of file Catalog.java.

Db com.cloudera.impala.catalog.Catalog.getBuiltinsDb ( )

inlineinherited

Definition at line 100 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.builtinsDb_.

TGetAllCatalogObjectsResponse com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogObjects ( long  fromVersion )

inline

Returns all known objects in the Catalog (Tables, Views, Databases, and Functions). Some metadata may be skipped for objects that have a catalog version < the specified "fromVersion".

Definition at line 212 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.authPolicy_, com.cloudera.impala.catalog.CatalogServiceCatalog.catalogServiceId_, com.cloudera.impala.catalog.AuthorizationPolicy.getAllRoles(), com.cloudera.impala.catalog.Db.getAllTableNames(), com.cloudera.impala.catalog.Db.getCatalogVersion(), com.cloudera.impala.catalog.Table.getCatalogVersion(), com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogVersion(), com.cloudera.impala.catalog.Catalog.getDataSources(), com.cloudera.impala.catalog.Catalog.getDb(), com.cloudera.impala.catalog.Catalog.getDbNames(), com.cloudera.impala.catalog.Db.getFunctions(), com.cloudera.impala.catalog.Catalog.hdfsCachePools_, com.cloudera.impala.catalog.Catalog.INITIAL_CATALOG_VERSION, and pool.

long com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogVersion ( )

inline

Returns the current Catalog version.

Definition at line 953 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.catalogVersion_.

Referenced by com.cloudera.impala.service.CatalogOpExecutor.createDatabase(), com.cloudera.impala.service.CatalogOpExecutor.createTable(), com.cloudera.impala.service.CatalogOpExecutor.createTableLike(), and com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogObjects().

DataSource com.cloudera.impala.catalog.Catalog.getDataSource ( String  dataSourceName )

inlineinherited

Gets the specified data source.

Definition at line 213 of file Catalog.java.

Referenced by com.cloudera.impala.service.CatalogOpExecutor.createDataSource(), and com.cloudera.impala.catalog.Catalog.getTCatalogObject().

List<String> com.cloudera.impala.catalog.Catalog.getDataSourceNames ( String  pattern )

inlineinherited

Returns a list of data sources names that match pattern. See filterStringsByPattern for details of the pattern match semantics.

pattern may be null (and thus matches everything).

Definition at line 231 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.dataSources_, and com.cloudera.impala.catalog.Catalog.filterStringsByPattern().

List<DataSource> com.cloudera.impala.catalog.Catalog.getDataSources ( )

inlineinherited

Gets a list of all data sources.

Definition at line 221 of file Catalog.java.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogObjects().

List<DataSource> com.cloudera.impala.catalog.Catalog.getDataSources ( String  pattern )

inlineinherited

Returns a list of data sources that match pattern. See filterStringsByPattern for details of the pattern match semantics.

pattern may be null (and thus matches everything).

Definition at line 241 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.dataSources_, com.cloudera.impala.catalog.Catalog.filterStringsByPattern(), and impala.name.

Db com.cloudera.impala.catalog.Catalog.getDb ( String  dbName )

inlineinherited

Gets the Db object from the Catalog using a case-insensitive lookup on the name. Returns null if no matching database is found.

Definition at line 115 of file Catalog.java.

Referenced by com.cloudera.impala.catalog.ImpaladCatalog.addDb(), com.cloudera.impala.catalog.Catalog.addFunction(), com.cloudera.impala.catalog.ImpaladCatalog.addFunction(), com.cloudera.impala.catalog.CatalogServiceCatalog.addFunction(), com.cloudera.impala.catalog.CatalogServiceCatalog.addPartition(), com.cloudera.impala.catalog.ImpaladCatalog.addTable(), com.cloudera.impala.catalog.CatalogServiceCatalog.addTable(), com.cloudera.impala.catalog.Catalog.containsFunction(), com.cloudera.impala.catalog.Catalog.containsTable(), com.cloudera.impala.service.CatalogOpExecutor.createDatabase(), com.cloudera.impala.catalog.TableLoadingMgr.LoadRequest.get(), com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogObjects(), com.cloudera.impala.catalog.Catalog.getFunction(), com.cloudera.impala.catalog.CatalogServiceCatalog.getFunctions(), com.cloudera.impala.testutil.ImpaladTestCatalog.getTable(), com.cloudera.impala.catalog.Catalog.getTable(), com.cloudera.impala.catalog.Catalog.getTableNames(), com.cloudera.impala.catalog.Catalog.getTCatalogObject(), com.cloudera.impala.testutil.ImpaladTestCatalog.ImpaladTestCatalog(), com.cloudera.impala.catalog.CatalogServiceCatalog.invalidateTable(), com.cloudera.impala.catalog.ImpaladCatalog.removeDb(), com.cloudera.impala.catalog.Catalog.removeFunction(), com.cloudera.impala.catalog.ImpaladCatalog.removeFunction(), com.cloudera.impala.catalog.Catalog.removeTable(), com.cloudera.impala.catalog.ImpaladCatalog.removeTable(), com.cloudera.impala.catalog.CatalogServiceCatalog.removeTable(), com.cloudera.impala.catalog.CatalogServiceCatalog.renameTable(), com.cloudera.impala.catalog.CatalogServiceCatalog.replaceTableIfUnchanged(), com.cloudera.impala.catalog.CatalogTest.testStats(), com.cloudera.impala.analysis.AuthorizationTest.TestTPCHCleanup(), and com.cloudera.impala.catalog.CatalogServiceCatalog.updateLastDdlTime().

List<String> com.cloudera.impala.catalog.Catalog.getDbNames ( String  dbPattern )

inlineinherited

Returns a list of databases that match dbPattern. See filterStringsByPattern for details of the pattern match semantics.

dbPattern may be null (and thus matches everything).

Definition at line 136 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.dbCache_, and com.cloudera.impala.catalog.Catalog.filterStringsByPattern().

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogObjects(), com.cloudera.impala.testutil.ImpaladTestCatalog.ImpaladTestCatalog(), and com.cloudera.impala.testutil.BlockIdGenerator.main().

Function com.cloudera.impala.catalog.Catalog.getFunction ( Function  desc, Function.CompareMode  mode  )

inlineinherited

Returns the function that best matches 'desc' that is registered with the catalog using 'mode' to check for matching. If desc matches multiple functions in the catalog, it will return the function with the strictest matching mode.

Definition at line 270 of file Catalog.java.

References com.cloudera.impala.catalog.Function.dbName(), and com.cloudera.impala.catalog.Catalog.getDb().

Referenced by com.cloudera.impala.catalog.Catalog.getTCatalogObject().

List<Function> com.cloudera.impala.catalog.CatalogServiceCatalog.getFunctions ( String  dbName ) throws DatabaseNotFoundException

inline

Returns all user defined functions (aggregate and scalar) in the specified database. Functions are not returned in a defined order.

Definition at line 323 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.getDb().

HdfsCachePool com.cloudera.impala.catalog.Catalog.getHdfsCachePool ( String  poolName )

inlineinherited

Gets a HdfsCachePool given a cache pool name. Returns null if the cache pool does not exist.

Definition at line 312 of file Catalog.java.

Referenced by com.cloudera.impala.catalog.Catalog.getTCatalogObject().

HdfsPartition com.cloudera.impala.catalog.Catalog.getHdfsPartition ( String  dbName, String  tableName, List< TPartitionKeyValue >  partitionSpec  ) throws CatalogException

inlineinherited

Returns the HdfsPartition object for the given dbName/tableName and partition spec. This will trigger a metadata load if the table metadata is not yet cached.

Exceptions

DatabaseNotFoundException	- If the database does not exist.
TableNotFoundException	- If the table does not exist.
PartitionNotFoundException	- If the partition does not exist.
TableLoadingException	- If there is an error loading the table metadata.

Definition at line 361 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.getTable().

Referenced by com.cloudera.impala.catalog.Catalog.containsHdfsPartition().

static long com.cloudera.impala.catalog.CatalogServiceCatalog.getLastDdlTime ( org.apache.hadoop.hive.metastore.api.Table  msTbl )

inlinestatic

Returns the table parameter 'transient_lastDdlTime', or -1 if it's not set. TODO: move this to a metastore helper class.

Definition at line 603 of file CatalogServiceCatalog.java.

Referenced by com.cloudera.impala.catalog.Table.Table().

MetaStoreClient com.cloudera.impala.catalog.Catalog.getMetaStoreClient ( )

inlineinherited

Returns a managed meta store client from the client connection pool.

Definition at line 326 of file Catalog.java.

Referenced by com.cloudera.impala.catalog.ImpaladCatalog.getTablePath(), and com.cloudera.impala.catalog.CatalogServiceCatalog.invalidateTable().

TableId com.cloudera.impala.catalog.CatalogServiceCatalog.getNextTableId ( )

inline

Gets the next table ID and increments the table ID counter.

Definition at line 965 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.nextTableId_.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.addTable(), com.cloudera.impala.catalog.CatalogServiceCatalog.invalidateTable(), and com.cloudera.impala.catalog.CatalogServiceCatalog.reset().

Table com.cloudera.impala.catalog.CatalogServiceCatalog.getOrLoadTable ( String  dbName, String  tblName  ) throws CatalogException

inline

Gets the table with the given name, loading it if needed (if the existing catalog object is not yet loaded). Returns the matching Table or null if no table with this name exists in the catalog. If the existing table is dropped or modified (indicated by the catalog version changing) while the load is in progress, the loaded value will be discarded and the current cached value will be returned. This may mean that a missing table (not yet loaded table) will be returned.

Definition at line 479 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.getTable(), com.cloudera.impala.catalog.Table.isLoaded(), and com.cloudera.impala.catalog.CatalogServiceCatalog.replaceTableIfUnchanged().

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.dropPartition(), com.cloudera.impala.catalog.CatalogTest.TestColSchema(), com.cloudera.impala.catalog.CatalogTest.testColStatsColTypeMismatch(), com.cloudera.impala.catalog.CatalogTest.testInternalHBaseTable(), com.cloudera.impala.catalog.CatalogTest.TestPartitions(), and com.cloudera.impala.catalog.CatalogTest.TestTableWithBadEscapeChar().

RolePrivilege com.cloudera.impala.catalog.CatalogServiceCatalog.getRolePrivilege ( String  roleName, TPrivilege  privSpec  ) throws CatalogException

inline

Gets a RolePrivilege from the given role name. Returns the privilege if it exists, or null if no privilege matching the privilege spec exist. Throws a CatalogException if the role does not exist.

Definition at line 926 of file CatalogServiceCatalog.java.

SentryProxy com.cloudera.impala.catalog.CatalogServiceCatalog.getSentryProxy ( )

inline

Definition at line 966 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.sentryProxy_.

Referenced by com.cloudera.impala.service.CatalogOpExecutor.verifySentryServiceEnabled().

Table com.cloudera.impala.catalog.Catalog.getTable ( String  dbName, String  tableName  ) throws CatalogException

inlineinherited

Returns the Table object for the given dbName/tableName. This will trigger a metadata load if the table metadata is not yet cached.

Definition at line 144 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.getDb().

Referenced by com.cloudera.impala.catalog.Catalog.getHdfsPartition(), com.cloudera.impala.catalog.CatalogServiceCatalog.getOrLoadTable(), com.cloudera.impala.catalog.Catalog.getTCatalogObject(), and com.cloudera.impala.catalog.CatalogServiceCatalog.reloadTable().

List<String> com.cloudera.impala.catalog.Catalog.getTableNames ( String  dbName, String  tablePattern  ) throws DatabaseNotFoundException

inlineinherited

Returns a list of tables in the supplied database that match tablePattern. See filterStringsByPattern for details of the pattern match semantics.

dbName must not be null, but tablePattern may be null (and thus matches everything).

Table names are returned unqualified.

Definition at line 173 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.filterStringsByPattern(), and com.cloudera.impala.catalog.Catalog.getDb().

TCatalogObject com.cloudera.impala.catalog.Catalog.getTCatalogObject ( TCatalogObject  objectDesc ) throws CatalogException

inlineinherited

Gets the thrift representation of a catalog object, given the "object description". The object description is just a TCatalogObject with only the catalog object type and object name set. If the object is not found, a CatalogException is thrown.

Definition at line 399 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.getDataSource(), com.cloudera.impala.catalog.Catalog.getDb(), com.cloudera.impala.catalog.Catalog.getFunction(), com.cloudera.impala.catalog.Catalog.getHdfsCachePool(), com.cloudera.impala.catalog.Role.getName(), com.cloudera.impala.catalog.Role.getPrivileges(), com.cloudera.impala.catalog.Catalog.getTable(), and pool.

long com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion ( )

inline

Increments the current Catalog version and returns the new value.

Definition at line 941 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.catalogVersion_.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.addDataSource(), com.cloudera.impala.catalog.CatalogServiceCatalog.addDb(), com.cloudera.impala.catalog.CatalogServiceCatalog.addFunction(), com.cloudera.impala.catalog.CatalogServiceCatalog.addPartition(), com.cloudera.impala.catalog.CatalogServiceCatalog.addRole(), com.cloudera.impala.catalog.CatalogServiceCatalog.addRoleGrantGroup(), com.cloudera.impala.catalog.CatalogServiceCatalog.addRolePrivilege(), com.cloudera.impala.catalog.CatalogServiceCatalog.addTable(), com.cloudera.impala.catalog.CatalogServiceCatalog.invalidateTable(), com.cloudera.impala.catalog.CatalogServiceCatalog.removeDataSource(), com.cloudera.impala.catalog.CatalogServiceCatalog.removeDb(), com.cloudera.impala.catalog.CatalogServiceCatalog.removeFunction(), com.cloudera.impala.catalog.CatalogServiceCatalog.removeRole(), com.cloudera.impala.catalog.CatalogServiceCatalog.removeRoleGrantGroup(), com.cloudera.impala.catalog.CatalogServiceCatalog.removeRolePrivilege(), com.cloudera.impala.catalog.CatalogServiceCatalog.removeTable(), com.cloudera.impala.catalog.CatalogServiceCatalog.replaceTableIfUnchanged(), and com.cloudera.impala.catalog.CatalogServiceCatalog.reset().

boolean com.cloudera.impala.catalog.CatalogServiceCatalog.invalidateTable ( TTableName  tableName, Pair< Db, Table >  updatedObjects  )

inline

Invalidates the table in the catalog cache, potentially adding/removing the table from the cache based on whether it exists in the Hive Metastore. The invalidation logic is:

• If the table exists in the metastore, add it to the catalog as an uninitialized IncompleteTable (replacing any existing entry). The table metadata will be loaded lazily, on the next access. If the parent database for this table does not yet exist in Impala's cache it will also be added.
• If the table does not exist in the metastore, remove it from the catalog cache.
• If we are unable to determine whether the table exists in the metastore (there was an exception thrown making the RPC), invalidate any existing Table by replacing it with an uninitialized IncompleteTable.

The parameter updatedObjects is a Pair that contains details on what catalog objects were modified as a result of the invalidateTable() call. The first item in the Pair is a Db which will only be set if a new database was added as a result of this call, otherwise it will be null. The second item in the Pair is the Table that was modified/added/removed. Returns a flag that indicates whether the items in updatedObjects were removed (returns true) or added/modified (return false). Only Tables should ever be removed.

Definition at line 753 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.addDb(), com.cloudera.impala.catalog.Db.containsTable(), com.cloudera.impala.catalog.Catalog.getDb(), com.cloudera.impala.catalog.Catalog.getMetaStoreClient(), com.cloudera.impala.catalog.CatalogServiceCatalog.getNextTableId(), com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion(), com.cloudera.impala.catalog.CatalogServiceCatalog.loadInBackground_, and com.cloudera.impala.catalog.CatalogServiceCatalog.removeTable().

void com.cloudera.impala.catalog.CatalogServiceCatalog.prioritizeLoad ( List< TCatalogObject >  objectDescs )

inline

Prioritizes the loading of the given list TCatalogObjects. Currently only support loading Table/View metadata since Db and Function metadata is not loaded lazily.

Definition at line 198 of file CatalogServiceCatalog.java.

Table com.cloudera.impala.catalog.CatalogServiceCatalog.reloadTable ( TTableName  tblName ) throws CatalogException

inline

Reloads a table's metadata, reusing any existing cached metadata to speed up the operation. Returns the updated Table object or null if no table with this name exists in the catalog. If the existing table is dropped or modified (indicated by the catalog version changing) while the reload is in progress, the loaded value will be discarded and the current cached value will be returned. This may mean that a missing table (not yet loaded table) will be returned.

Definition at line 657 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.getTable(), and com.cloudera.impala.catalog.CatalogServiceCatalog.replaceTableIfUnchanged().

DataSource com.cloudera.impala.catalog.CatalogServiceCatalog.removeDataSource ( String  dataSourceName )

inline

Definition at line 591 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Db com.cloudera.impala.catalog.CatalogServiceCatalog.removeDb ( String  dbName )

inline

Removes a database from the metadata cache and returns the removed database, or null if the database did not exist in the cache. Used by DROP DATABASE statements.

Definition at line 448 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Function com.cloudera.impala.catalog.CatalogServiceCatalog.removeFunction ( Function  desc )

inline

Removes a function from the catalog. Increments the catalog version and returns the Function object that was removed. If the function did not exist, null will be returned.

Definition at line 554 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Role com.cloudera.impala.catalog.CatalogServiceCatalog.removeRole ( String  roleName )

inline

Removes the role with the given name from the AuthorizationPolicy. Returns the removed role with an incremented catalog version, or null if no role with this name exists.

Definition at line 834 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Role com.cloudera.impala.catalog.CatalogServiceCatalog.removeRoleGrantGroup ( String  roleName, String  groupName  ) throws CatalogException

inline

Removes a grant group from the given role name and returns the modified Role with an updated catalog version. If the role does not exist a CatalogException is thrown.

Definition at line 867 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

RolePrivilege com.cloudera.impala.catalog.CatalogServiceCatalog.removeRolePrivilege ( String  roleName, TPrivilege  thriftPriv  ) throws CatalogException

inline

Removes a RolePrivilege from the given role name. Returns the removed RolePrivilege with an incremented catalog version or null if no matching privilege was found. Throws a CatalogException if no role exists with this name.

Definition at line 905 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Table com.cloudera.impala.catalog.Catalog.removeTable ( TTableName  tableName )

inlineinherited

Removes a table from the catalog and returns the table that was removed, or null if the table/database does not exist.

Definition at line 157 of file Catalog.java.

References com.cloudera.impala.catalog.Catalog.getDb().

Table com.cloudera.impala.catalog.CatalogServiceCatalog.removeTable ( String  dbName, String  tblName  )

inline

Removes a table from the catalog and increments the catalog version. Returns the removed Table, or null if the table or db does not exist.

Definition at line 537 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.getDb(), and com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.invalidateTable().

Table com.cloudera.impala.catalog.CatalogServiceCatalog.renameTable ( TTableName  oldTableName, TTableName  newTableName  ) throws CatalogException

inline

Renames a table. Equivalent to an atomic drop + add of the table. Returns the new Table object with an incremented catalog version or null if operation was not successful.

Definition at line 633 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.CatalogServiceCatalog.addTable(), and com.cloudera.impala.catalog.Catalog.getDb().

Referenced by com.cloudera.impala.service.CatalogOpExecutor.alterTableOrViewRename().

Table com.cloudera.impala.catalog.CatalogServiceCatalog.replaceTableIfUnchanged ( Table  updatedTbl, long  expectedCatalogVersion  ) throws DatabaseNotFoundException

inlineprivate

Replaces an existing Table with a new value if it exists and has not changed (has the same catalog version as 'expectedCatalogVersion').

Definition at line 509 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Table.getCatalogVersion(), com.cloudera.impala.catalog.Catalog.getDb(), and com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.dropPartition(), com.cloudera.impala.catalog.CatalogServiceCatalog.getOrLoadTable(), and com.cloudera.impala.catalog.CatalogServiceCatalog.reloadTable().

void com.cloudera.impala.catalog.CatalogServiceCatalog.reset ( ) throws CatalogException

inline

Resets this catalog instance by clearing all cached table and database metadata.

Definition at line 343 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.dbCache_, com.cloudera.impala.catalog.CatalogServiceCatalog.getNextTableId(), com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion(), com.cloudera.impala.catalog.CatalogServiceCatalog.loadInBackground_, and com.cloudera.impala.catalog.CatalogServiceCatalog.sentryProxy_.

void com.cloudera.impala.catalog.CatalogServiceCatalog.updateLastDdlTime ( TTableName  tblName, long  ddlTime  )

inline

Updates the cached lastDdlTime for the given table. The lastDdlTime is used during the metadata refresh() operations to determine if there have been any external (outside of Impala) modifications to the table.

Definition at line 620 of file CatalogServiceCatalog.java.

References com.cloudera.impala.catalog.Catalog.getDb().

void com.cloudera.impala.catalog.CatalogServiceCatalog.watchCacheDirs ( List< Long >  dirIds, TTableName  tblName  )

inline

Adds a list of cache directive IDs for the given table name. Asynchronously refreshes the table metadata once all cache directives complete.

Definition at line 190 of file CatalogServiceCatalog.java.

Referenced by com.cloudera.impala.service.CatalogOpExecutor.updateCatalog().

Member Data Documentation

AuthorizationPolicy com.cloudera.impala.catalog.Catalog.authPolicy_ = new AuthorizationPolicy()

protectedinherited

Definition at line 67 of file Catalog.java.

Referenced by com.cloudera.impala.testutil.CatalogServiceTestCatalog.getAuthPolicy(), com.cloudera.impala.catalog.ImpaladCatalog.getAuthPolicy(), com.cloudera.impala.catalog.CatalogServiceCatalog.getAuthPolicy(), com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogObjects(), and com.cloudera.impala.testutil.ImpaladTestCatalog.ImpaladTestCatalog().

final String com.cloudera.impala.catalog.Catalog.BUILTINS_DB = "_impala_builtins"

staticinherited

Definition at line 61 of file Catalog.java.

Referenced by com.cloudera.impala.analysis.ExtractFromExpr.analyze(), com.cloudera.impala.analysis.FunctionName.analyze(), com.cloudera.impala.analysis.CreateFunctionStmtBase.analyze(), com.cloudera.impala.analysis.CastExpr.analyze(), com.cloudera.impala.catalog.Catalog.Catalog(), com.cloudera.impala.catalog.ScalarFunction.createBuiltin(), com.cloudera.impala.catalog.ScalarFunction.createBuiltinSearchDesc(), and com.cloudera.impala.analysis.Expr.getBuiltinFunction().

final ScheduledExecutorService com.cloudera.impala.catalog.CatalogServiceCatalog.cachePoolReader_

private

Initial value:

=
      Executors.newScheduledThreadPool(1)

Definition at line 113 of file CatalogServiceCatalog.java.

final ReentrantReadWriteLock com.cloudera.impala.catalog.CatalogServiceCatalog.catalogLock_ = new ReentrantReadWriteLock(true)

private

Definition at line 97 of file CatalogServiceCatalog.java.

final TUniqueId com.cloudera.impala.catalog.CatalogServiceCatalog.catalogServiceId_

private

Definition at line 84 of file CatalogServiceCatalog.java.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.CatalogServiceCatalog(), and com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogObjects().

long com.cloudera.impala.catalog.CatalogServiceCatalog.catalogVersion_ = INITIAL_CATALOG_VERSION

private

Definition at line 103 of file CatalogServiceCatalog.java.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogVersion(), and com.cloudera.impala.catalog.CatalogServiceCatalog.incrementAndGetCatalogVersion().

final CatalogObjectCache<DataSource> com.cloudera.impala.catalog.Catalog.dataSources_

protectedinherited

Definition at line 80 of file Catalog.java.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.addDataSource(), com.cloudera.impala.catalog.Catalog.Catalog(), com.cloudera.impala.catalog.Catalog.getDataSourceNames(), and com.cloudera.impala.catalog.Catalog.getDataSources().

AtomicReference<ConcurrentHashMap<String, Db> > com.cloudera.impala.catalog.Catalog.dbCache_

protectedinherited

Initial value:

=
      new AtomicReference<ConcurrentHashMap<String, Db>>(
          new ConcurrentHashMap<String, Db>())

Definition at line 72 of file Catalog.java.

Referenced by com.cloudera.impala.catalog.Catalog.getDbNames(), and com.cloudera.impala.catalog.CatalogServiceCatalog.reset().

final String com.cloudera.impala.catalog.Catalog.DEFAULT_DB = "default"

staticinherited

Definition at line 58 of file Catalog.java.

Referenced by com.cloudera.impala.analysis.AnalyzerTest.AnalysisError(), com.cloudera.impala.analysis.UseStmt.analyze(), com.cloudera.impala.analysis.AuditingTest.AnalyzeAccessEvents(), com.cloudera.impala.analysis.AnalyzeAuthStmtsTest.AnalyzeAuthStmtsTest(), com.cloudera.impala.analysis.AnalyzeAuthStmtsTest.AnalyzeCreateDropRole(), com.cloudera.impala.analysis.AnalyzeAuthStmtsTest.AnalyzeGrantRevokePriv(), com.cloudera.impala.analysis.AnalyzeAuthStmtsTest.AnalyzeGrantRevokeRole(), com.cloudera.impala.analysis.AnalyzeAuthStmtsTest.AnalyzeShowGrantRole(), com.cloudera.impala.analysis.AnalyzeAuthStmtsTest.AnalyzeShowRoles(), com.cloudera.impala.analysis.AnalyzerTest.AnalyzesOk(), com.cloudera.impala.analysis.AuthorizationTest.AuthorizationTest(), com.cloudera.impala.analysis.AnalyzerTest.createAnalyzerUsingHiveColLabels(), com.cloudera.impala.testutil.TestUtils.createQueryContext(), com.cloudera.impala.analysis.AuthorizationTest.TestFunction(), com.cloudera.impala.analysis.AuthorizationTest.TestLocalGroupPolicyProvider(), com.cloudera.impala.analysis.AuthorizationTest.TestShortUsernameUsed(), and com.cloudera.impala.analysis.AuthorizationTest.TestWithIncorrectConfig().

final CatalogObjectCache<HdfsCachePool> com.cloudera.impala.catalog.Catalog.hdfsCachePools_

protectedinherited

Initial value:

=
      new CatalogObjectCache<HdfsCachePool>(false)

Definition at line 84 of file Catalog.java.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogObjects(), and com.cloudera.impala.catalog.CatalogServiceCatalog.CachePoolReader.run().

final long com.cloudera.impala.catalog.Catalog.INITIAL_CATALOG_VERSION = 0L

staticinherited

Definition at line 57 of file Catalog.java.

Referenced by com.cloudera.impala.service.CatalogOpExecutor.createDatabase(), com.cloudera.impala.catalog.CatalogServiceCatalog.getCatalogObjects(), and com.cloudera.impala.service.CatalogOpExecutor.TableToTCatalogObject().

final boolean com.cloudera.impala.catalog.CatalogServiceCatalog.loadInBackground_

private

Definition at line 110 of file CatalogServiceCatalog.java.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.CatalogServiceCatalog(), com.cloudera.impala.catalog.CatalogServiceCatalog.invalidateTable(), and com.cloudera.impala.catalog.CatalogServiceCatalog.reset().

final Logger com.cloudera.impala.catalog.CatalogServiceCatalog.LOG = Logger.getLogger(CatalogServiceCatalog.class)

staticprivate

Definition at line 82 of file CatalogServiceCatalog.java.

final MetaStoreClientPool com.cloudera.impala.catalog.Catalog.metaStoreClientPool_ = new MetaStoreClientPool(0)

protectedinherited

Definition at line 63 of file Catalog.java.

final AtomicInteger com.cloudera.impala.catalog.CatalogServiceCatalog.nextTableId_ = new AtomicInteger(0)

protected

Definition at line 105 of file CatalogServiceCatalog.java.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.getNextTableId().

final SentryProxy com.cloudera.impala.catalog.CatalogServiceCatalog.sentryProxy_

private

Definition at line 118 of file CatalogServiceCatalog.java.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.CatalogServiceCatalog(), com.cloudera.impala.catalog.CatalogServiceCatalog.getSentryProxy(), and com.cloudera.impala.catalog.CatalogServiceCatalog.reset().

final TableLoadingMgr com.cloudera.impala.catalog.CatalogServiceCatalog.tableLoadingMgr_

private

Definition at line 108 of file CatalogServiceCatalog.java.

Referenced by com.cloudera.impala.catalog.CatalogServiceCatalog.CatalogServiceCatalog().

---

The documentation for this class was generated from the following file:

• fe/src/main/java/com/cloudera/impala/catalog/CatalogServiceCatalog.java