com.cloudera.impala.service.Frontend Class Reference

Collaboration diagram for com.cloudera.impala.service.Frontend:

Classes
class	AuthorizationPolicyReader

Public Member Functions
	Frontend (AuthorizationConfig authorizationConfig)
	Frontend (AuthorizationConfig authorizationConfig, ImpaladCatalog catalog)
ImpaladCatalog	getCatalog ()
AuthorizationChecker	getAuthzChecker ()
TUpdateCatalogCacheResponse	updateCatalogCache (TUpdateCatalogCacheRequest req) throws CatalogException
TLoadDataResp	loadTableData (TLoadDataReq request) throws ImpalaException, IOException
String	getExplainString (TQueryCtx queryCtx) throws ImpalaException
List< String >	getTableNames (String dbName, String tablePattern, User user) throws ImpalaException
List< String >	getDbNames (String dbPattern, User user)
List< DataSource >	getDataSrcs (String pattern)
TResultSet	getColumnStats (String dbName, String tableName) throws ImpalaException
TResultSet	getTableStats (String dbName, String tableName) throws ImpalaException
List< Function >	getFunctions (TFunctionCategory category, String dbName, String fnPattern) throws DatabaseNotFoundException
TDescribeTableResult	describeTable (String dbName, String tableName, TDescribeTableOutputStyle outputStyle) throws ImpalaException
boolean	requestTblLoadAndWait (Set< TableName > requestedTbls) throws InternalException
TExecRequest	createExecRequest (TQueryCtx queryCtx, StringBuilder explainString) throws ImpalaException
TResultSet	execHiveServer2MetadataOp (TMetadataOpRequest request) throws ImpalaException
TResultSet	getTableFiles (TShowFilesParams request) throws ImpalaException

Private Member Functions
void	createCatalogOpRequest (AnalysisContext.AnalysisResult analysis, TExecRequest result)
Set< TableName >	getMissingTbls (Set< TableName > tableNames)
boolean	requestTblLoadAndWait (Set< TableName > requestedTbls, long timeoutMs) throws InternalException
AnalysisContext.AnalysisResult	analyzeStmt (TQueryCtx queryCtx) throws AnalysisException, InternalException, AuthorizationException
void	createExplainRequest (String explainString, TExecRequest result)

Private Attributes
final long	MISSING_TBL_LOAD_WAIT_TIMEOUT_MS = 2 * 60 * 1000
final long	MAX_CATALOG_UPDATE_WAIT_TIME_MS = 2 * 1000
ImpaladCatalog	impaladCatalog_
final AuthorizationConfig	authzConfig_
final AtomicReference < AuthorizationChecker >	authzChecker_
final ScheduledExecutorService	policyReader_

Static Private Attributes
static final Logger	LOG = LoggerFactory.getLogger(Frontend.class)
static final int	AUTHORIZATION_POLICY_RELOAD_INTERVAL_SECS = 5 * 60

Detailed Description

Frontend API for the impalad process. This class allows the impala daemon to create TQueryExecRequest in response to TClientRequests. Also handles management of the authorization policy.

Definition at line 142 of file Frontend.java.

Constructor & Destructor Documentation

com.cloudera.impala.service.Frontend.Frontend ( AuthorizationConfig  authorizationConfig )

inline

Definition at line 159 of file Frontend.java.

com.cloudera.impala.service.Frontend.Frontend ( AuthorizationConfig  authorizationConfig, ImpaladCatalog  catalog  )

inline

C'tor used by tests to pass in a custom ImpaladCatalog.

Definition at line 166 of file Frontend.java.

References com.cloudera.impala.service.Frontend.AUTHORIZATION_POLICY_RELOAD_INTERVAL_SECS, com.cloudera.impala.service.Frontend.authzChecker_, com.cloudera.impala.service.Frontend.authzConfig_, com.cloudera.impala.catalog.ImpaladCatalog.getAuthPolicy(), com.cloudera.impala.service.Frontend.impaladCatalog_, and com.cloudera.impala.authorization.AuthorizationConfig.isEnabled().

Member Function Documentation

AnalysisContext.AnalysisResult com.cloudera.impala.service.Frontend.analyzeStmt ( TQueryCtx  queryCtx ) throws AnalysisException, InternalException, AuthorizationException

inlineprivate

Analyzes the SQL statement included in queryCtx and returns the AnalysisResult. Authorizes all catalog object accesses and throws an AuthorizationException if the user does not have privileges to access one or more objects. If a statement fails analysis because table/view metadata was not loaded, an RPC to the CatalogServer will be executed to request loading the missing metadata and analysis will be restarted once the required tables have been loaded in the local Impalad Catalog or the MISSING_TBL_LOAD_WAIT_TIMEOUT_MS timeout is reached. The goal of this timeout is not to analysis, but to restart the analysis/missing table collection process. This helps ensure a statement never waits indefinitely for a table to be loaded in event the table metadata was invalidated. TODO: Also consider adding an overall timeout that fails analysis.

Definition at line 762 of file Frontend.java.

References com.cloudera.impala.service.Frontend.authzConfig_, com.cloudera.impala.service.Frontend.getAuthzChecker(), com.cloudera.impala.service.Frontend.getMissingTbls(), com.cloudera.impala.service.Frontend.impaladCatalog_, com.cloudera.impala.service.Frontend.MISSING_TBL_LOAD_WAIT_TIMEOUT_MS, and com.cloudera.impala.service.Frontend.requestTblLoadAndWait().

Referenced by com.cloudera.impala.service.Frontend.createExecRequest().

void com.cloudera.impala.service.Frontend.createCatalogOpRequest ( AnalysisContext.AnalysisResult  analysis, TExecRequest  result  )

inlineprivate

Constructs a TCatalogOpRequest and attaches it, plus any metadata, to the result argument.

Definition at line 232 of file Frontend.java.

References com.cloudera.impala.service.Frontend.getAuthzChecker(), com.cloudera.impala.analysis.ShowGrantRoleStmt.getRole(), com.cloudera.impala.catalog.Type.STRING, and com.cloudera.impala.catalog.ScalarType.toThrift().

Referenced by com.cloudera.impala.service.Frontend.createExecRequest().

TExecRequest com.cloudera.impala.service.Frontend.createExecRequest ( TQueryCtx  queryCtx, StringBuilder  explainString  ) throws ImpalaException

inline

Create a populated TExecRequest corresponding to the supplied TQueryCtx.

Definition at line 801 of file Frontend.java.

References com.cloudera.impala.service.Frontend.analyzeStmt(), com.cloudera.impala.service.Frontend.createCatalogOpRequest(), com.cloudera.impala.service.Frontend.createExplainRequest(), gen_ir_descriptions.idx, planner, com.cloudera.impala.catalog.Type.STRING, and com.cloudera.impala.catalog.ScalarType.toThrift().

Referenced by com.cloudera.impala.service.Frontend.getExplainString().

void com.cloudera.impala.service.Frontend.createExplainRequest ( String  explainString, TExecRequest  result  )

inlineprivate

Attaches the explain result to the TExecRequest.

Definition at line 986 of file Frontend.java.

References com.cloudera.impala.catalog.Type.STRING, and com.cloudera.impala.catalog.ScalarType.toThrift().

Referenced by com.cloudera.impala.service.Frontend.createExecRequest().

TDescribeTableResult com.cloudera.impala.service.Frontend.describeTable ( String  dbName, String  tableName, TDescribeTableOutputStyle  outputStyle  ) throws ImpalaException

inline

Returns table metadata, such as the column descriptors, in the specified table. Throws an exception if the table or db is not found or if there is an error loading the table metadata.

Definition at line 673 of file Frontend.java.

TResultSet com.cloudera.impala.service.Frontend.execHiveServer2MetadataOp ( TMetadataOpRequest  request ) throws ImpalaException

inline

Executes a HiveServer2 metadata operation and returns a TResultSet

Definition at line 1009 of file Frontend.java.

References com.cloudera.impala.util.TSessionStateUtil.getEffectiveUser(), and com.cloudera.impala.authorization.ImpalaInternalAdminUser.getInstance().

AuthorizationChecker com.cloudera.impala.service.Frontend.getAuthzChecker ( )

inline

Definition at line 205 of file Frontend.java.

Referenced by com.cloudera.impala.service.Frontend.analyzeStmt(), and com.cloudera.impala.service.Frontend.createCatalogOpRequest().

ImpaladCatalog com.cloudera.impala.service.Frontend.getCatalog ( )

inline

Definition at line 204 of file Frontend.java.

References com.cloudera.impala.service.Frontend.impaladCatalog_.

Referenced by com.cloudera.impala.service.Frontend.getMissingTbls(), com.cloudera.impala.service.Frontend.requestTblLoadAndWait(), and com.cloudera.impala.service.Frontend.AuthorizationPolicyReader.run().

TResultSet com.cloudera.impala.service.Frontend.getColumnStats ( String  dbName, String  tableName  ) throws ImpalaException

inline

Generate result set and schema for a SHOW COLUMN STATS command.

Definition at line 604 of file Frontend.java.

References com.cloudera.impala.catalog.Type.BIGINT, com.cloudera.impala.catalog.Type.DOUBLE, com.cloudera.impala.catalog.Table.getColumnsInHiveOrder(), com.cloudera.impala.catalog.Type.INT, com.cloudera.impala.catalog.Type.STRING, and com.cloudera.impala.catalog.ScalarType.toThrift().

List<DataSource> com.cloudera.impala.service.Frontend.getDataSrcs ( String  pattern )

inline

Returns all data sources that match the pattern. If pattern is null, matches all data sources.

Definition at line 597 of file Frontend.java.

List<String> com.cloudera.impala.service.Frontend.getDbNames ( String  dbPattern, User  user  )

inline

Returns all database names that match the pattern and are accessible to the given user. If pattern is null, matches all dbs.

Definition at line 573 of file Frontend.java.

References com.cloudera.impala.service.Frontend.authzChecker_, com.cloudera.impala.service.Frontend.authzConfig_, and com.cloudera.impala.authorization.AuthorizationConfig.isEnabled().

String com.cloudera.impala.service.Frontend.getExplainString ( TQueryCtx  queryCtx ) throws ImpalaException

inline

Parses and plans a query in order to generate its explain string. This method does not increase the query id counter.

Definition at line 542 of file Frontend.java.

References com.cloudera.impala.service.Frontend.createExecRequest().

List<Function> com.cloudera.impala.service.Frontend.getFunctions ( TFunctionCategory  category, String  dbName, String  fnPattern  ) throws DatabaseNotFoundException

inline

Returns all function signatures that match the pattern. If pattern is null, matches all functions.

Definition at line 650 of file Frontend.java.

References com.cloudera.impala.catalog.Function.signatureString().

Set<TableName> com.cloudera.impala.service.Frontend.getMissingTbls ( Set< TableName >  tableNames )

inlineprivate

Given a set of table names, returns the set of table names that are missing metadata (are not yet loaded).

Definition at line 683 of file Frontend.java.

References com.cloudera.impala.service.Frontend.getCatalog(), and com.cloudera.impala.catalog.Table.isLoaded().

Referenced by com.cloudera.impala.service.Frontend.analyzeStmt(), and com.cloudera.impala.service.Frontend.requestTblLoadAndWait().

TResultSet com.cloudera.impala.service.Frontend.getTableFiles ( TShowFilesParams  request ) throws ImpalaException

inline

Returns all files info of a table or partition.

Definition at line 1050 of file Frontend.java.

List<String> com.cloudera.impala.service.Frontend.getTableNames ( String  dbName, String  tablePattern, User  user  ) throws ImpalaException

inline

Returns all tables that match the specified database and pattern that are accessible to the given user. If pattern is null, matches all tables. If db is null, all databases are searched for matches.

Definition at line 553 of file Frontend.java.

References com.cloudera.impala.authorization.Privilege.ANY, com.cloudera.impala.service.Frontend.authzChecker_, com.cloudera.impala.service.Frontend.authzConfig_, and com.cloudera.impala.authorization.AuthorizationConfig.isEnabled().

TResultSet com.cloudera.impala.service.Frontend.getTableStats ( String  dbName, String  tableName  ) throws ImpalaException

inline

Generate result set and schema for a SHOW TABLE STATS command.

Definition at line 632 of file Frontend.java.

TLoadDataResp com.cloudera.impala.service.Frontend.loadTableData ( TLoadDataReq  request ) throws ImpalaException, IOException

inline

Loads a table or partition with one or more data files. If the "overwrite" flag in the request is true, all existing data in the table/partition will be replaced. If the "overwrite" flag is false, the files will be added alongside any existing data files.

Definition at line 488 of file Frontend.java.

References com.cloudera.impala.analysis.TableName.getTbl().

boolean com.cloudera.impala.service.Frontend.requestTblLoadAndWait ( Set< TableName >  requestedTbls, long  timeoutMs  ) throws InternalException

inlineprivate

Requests the catalog server load the given set of tables and waits until these tables show up in the local catalog, or the given timeout has been reached. The timeout is specified in milliseconds, with a value <= 0 indicating no timeout. The exact steps taken are: 1) Collect the tables that are missing (not yet loaded locally). 2) Make an RPC to the CatalogServer to prioritize the loading of these tables. 3) Wait until the local catalog contains all missing tables by (re)checking the catalog each time a new catalog update is received.

Returns true if all missing tables were received before timing out and false if the timeout was reached before all tables were received.

Definition at line 708 of file Frontend.java.

References com.cloudera.impala.service.Frontend.getCatalog(), com.cloudera.impala.service.Frontend.getMissingTbls(), com.cloudera.impala.service.Frontend.MAX_CATALOG_UPDATE_WAIT_TIME_MS, and impala.OK.

Referenced by com.cloudera.impala.service.Frontend.analyzeStmt(), and com.cloudera.impala.service.Frontend.requestTblLoadAndWait().

boolean com.cloudera.impala.service.Frontend.requestTblLoadAndWait ( Set< TableName >  requestedTbls ) throws InternalException

inline

Overload of requestTblLoadAndWait that uses the default timeout.

Definition at line 743 of file Frontend.java.

References com.cloudera.impala.service.Frontend.MISSING_TBL_LOAD_WAIT_TIMEOUT_MS, and com.cloudera.impala.service.Frontend.requestTblLoadAndWait().

TUpdateCatalogCacheResponse com.cloudera.impala.service.Frontend.updateCatalogCache ( TUpdateCatalogCacheRequest  req ) throws CatalogException

inline

Definition at line 207 of file Frontend.java.

References com.cloudera.impala.service.Frontend.authzConfig_, com.cloudera.impala.catalog.ImpaladCatalog.getAuthPolicy(), and com.cloudera.impala.service.Frontend.impaladCatalog_.

Member Data Documentation

final int com.cloudera.impala.service.Frontend.AUTHORIZATION_POLICY_RELOAD_INTERVAL_SECS = 5 * 60

staticprivate

Definition at line 151 of file Frontend.java.

Referenced by com.cloudera.impala.service.Frontend.Frontend().

final AtomicReference<AuthorizationChecker> com.cloudera.impala.service.Frontend.authzChecker_

private

Definition at line 155 of file Frontend.java.

Referenced by com.cloudera.impala.service.Frontend.Frontend(), com.cloudera.impala.service.Frontend.getDbNames(), and com.cloudera.impala.service.Frontend.getTableNames().

final AuthorizationConfig com.cloudera.impala.service.Frontend.authzConfig_

private

Definition at line 154 of file Frontend.java.

Referenced by com.cloudera.impala.service.Frontend.analyzeStmt(), com.cloudera.impala.service.Frontend.Frontend(), com.cloudera.impala.service.Frontend.getDbNames(), com.cloudera.impala.service.Frontend.getTableNames(), and com.cloudera.impala.service.Frontend.updateCatalogCache().

ImpaladCatalog com.cloudera.impala.service.Frontend.impaladCatalog_

private

Definition at line 153 of file Frontend.java.

Referenced by com.cloudera.impala.service.Frontend.analyzeStmt(), com.cloudera.impala.service.Frontend.Frontend(), com.cloudera.impala.service.Frontend.getCatalog(), and com.cloudera.impala.service.Frontend.updateCatalogCache().

final Logger com.cloudera.impala.service.Frontend.LOG = LoggerFactory.getLogger(Frontend.class)

staticprivate

Definition at line 143 of file Frontend.java.

final long com.cloudera.impala.service.Frontend.MAX_CATALOG_UPDATE_WAIT_TIME_MS = 2 * 1000

private

Definition at line 148 of file Frontend.java.

Referenced by com.cloudera.impala.service.Frontend.requestTblLoadAndWait().

final long com.cloudera.impala.service.Frontend.MISSING_TBL_LOAD_WAIT_TIMEOUT_MS = 2 * 60 * 1000

private

Definition at line 145 of file Frontend.java.

Referenced by com.cloudera.impala.service.Frontend.analyzeStmt(), and com.cloudera.impala.service.Frontend.requestTblLoadAndWait().

final ScheduledExecutorService com.cloudera.impala.service.Frontend.policyReader_

private

Initial value:

=
      Executors.newScheduledThreadPool(1)

Definition at line 156 of file Frontend.java.

---

The documentation for this class was generated from the following file:

• fe/src/main/java/com/cloudera/impala/service/Frontend.java