Inheritance diagram for com.cloudera.impala.planner.HdfsScanNode:

Collaboration diagram for com.cloudera.impala.planner.HdfsScanNode:

Public Member Functions
	HdfsScanNode (PlanNodeId id, TupleDescriptor desc, HdfsTable tbl)

void	init (Analyzer analyzer) throws InternalException

void	computeStats (Analyzer analyzer)

void	computeCosts (TQueryOptions queryOptions)

TupleDescriptor	getTupleDesc ()

List< TScanRangeLocations >	getScanRangeLocations ()

boolean	isTableMissingStats ()

boolean	isTableMissingTableStats ()

boolean	isTableMissingColumnStats ()

long	getInputCardinality ()

PlanNodeId	getId ()

void	setId (PlanNodeId id)

long	getLimit ()

boolean	hasLimit ()

long	getPerHostMemCost ()

long	getCardinality ()

int	getNumNodes ()

float	getAvgRowSize ()

void	setFragment (PlanFragment fragment)

PlanFragment	getFragment ()

List< Expr >	getConjuncts ()

ExprSubstitutionMap	getOutputSmap ()

void	setOutputSmap (ExprSubstitutionMap smap)

Set< ExprId >	getAssignedConjuncts ()

void	setAssignedConjuncts (Set< ExprId > conjuncts)

void	setLimit (long limit)

void	unsetLimit ()

ArrayList< TupleId >	getTupleIds ()

ArrayList< TupleId >	getTblRefIds ()

void	setTblRefIds (ArrayList< TupleId > ids)

Set< TupleId >	getNullableTupleIds ()

void	addConjuncts (List< Expr > conjuncts)

void	transferConjuncts (PlanNode recipient)

String	getExplainString ()

TPlan	treeToThrift ()

boolean	isBlockingNode ()

Static Public Member Functions
static long	getPerHostMemUpperBound ()

static long	addCardinalities (long a, long b)

static long	multiplyCardinalities (long a, long b)

Protected Member Functions
String	debugString ()

void	toThrift (TPlanNode msg)

String	getDisplayLabelDetail ()

String	getNodeExplainString (String prefix, String detailPrefix, TExplainLevel detailLevel)

String	getStatsExplainString (String prefix, TExplainLevel detailLevel)

final String	getExplainString (String rootPrefix, String prefix, TExplainLevel detailLevel)

String	getExplainString (List<?extends Expr > exprs)

void	setDisplayName (String s)

final String	getDisplayLabel ()

String	getOffsetExplainString (String prefix)

void	assignConjuncts (Analyzer analyzer)

ExprSubstitutionMap	getCombinedChildSmap ()

void	createDefaultSmap (Analyzer analyzer)

long	capAtLimit (long cardinality)

void	markSlotsMaterialized (Analyzer analyzer, List< Expr > exprs)

void	computeMemLayout (Analyzer analyzer)

double	computeSelectivity ()

boolean	hasValidStats ()

Static Protected Member Functions
static TNetworkAddress	addressToTNetworkAddress (String address)

Protected Attributes
final TupleDescriptor	desc_

long	inputCardinality_ = -1

int	numPartitionsMissingStats_ = 0

List< TScanRangeLocations >	scanRanges_

String	displayName_

PlanNodeId	id_

long	limit_

ArrayList< TupleId >	tupleIds_

ArrayList< TupleId >	tblRefIds_

Set< TupleId >	nullableTupleIds_ = Sets.newHashSet()

List< Expr >	conjuncts_ = Lists.newArrayList()

PlanFragment	fragment_

ExprSubstitutionMap	outputSmap_

Set< ExprId >	assignedConjuncts_

long	cardinality_

int	numNodes_

float	avgRowSize_

long	perHostMemCost_ = -1

Static Protected Attributes
static final int	DEFAULT_BATCH_SIZE = 1024

Private Member Functions
void	computeScanRangeLocations (Analyzer analyzer)

boolean	canEvalUsingPartitionMd (Expr expr, Analyzer analyzer)

HashSet< Long >	evalBinaryPredicate (Expr expr)

HashSet< Long >	evalInPredicate (Expr expr)

HashSet< Long >	evalIsNullPredicate (Expr expr)

HashSet< Long >	evalSlotBindingFilter (Expr expr)

void	prunePartitions (Analyzer analyzer) throws InternalException

void	evalPartitionFiltersInBe (List< HdfsPartitionFilter > filters, HashSet< Long > matchingPartitionIds, Analyzer analyzer) throws InternalException

Private Attributes
final HdfsTable	tbl_

final ArrayList< HdfsPartition >	partitions_ = Lists.newArrayList()

long	totalFiles_ = 0

long	totalBytes_ = 0

Static Private Attributes
static final Logger	LOG = LoggerFactory.getLogger(HdfsScanNode.class)

static final long	IO_MGR_BUFFER_SIZE = 8L * 1024L * 1024L

static final long	MAX_IO_BUFFERS_PER_THREAD = 10

static final int	THREADS_PER_CORE = 3

static final double	SCAN_RANGE_SKEW_FACTOR = 1.2

static final int	PARTITION_PRUNING_BATCH_SIZE = 1024

Detailed Description

Scan of a single single table. Currently limited to full-table scans. TODO: pass in range restrictions.

Definition at line 72 of file HdfsScanNode.java.

Constructor & Destructor Documentation

com.cloudera.impala.planner.HdfsScanNode.HdfsScanNode	(	PlanNodeId	id,
		TupleDescriptor	desc,
		HdfsTable	tbl
	)

inline

Constructs node to scan given data files of table 'tbl_'.

Definition at line 106 of file HdfsScanNode.java.

References com.cloudera.impala.planner.HdfsScanNode.tbl_.

Member Function Documentation

static long com.cloudera.impala.planner.PlanNode.addCardinalities	(	long	a,
		long	b
	)

inlinestaticinherited

Computes and returns the sum of two cardinalities. If an overflow occurs, the maximum Long value is returned (Long.MAX_VALUE).

Definition at line 528 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.ExchangeNode.computeStats(), com.cloudera.impala.planner.UnionNode.computeStats(), com.cloudera.impala.planner.HashJoinNode.computeStats(), com.cloudera.impala.planner.HdfsScanNode.computeStats(), and com.cloudera.impala.planner.PlanNode.getInputCardinality().

void com.cloudera.impala.planner.PlanNode.addConjuncts ( List< Expr > conjuncts )

inlineinherited

Definition at line 209 of file PlanNode.java.

static TNetworkAddress com.cloudera.impala.planner.ScanNode.addressToTNetworkAddress ( String address )

inlinestaticprotectedinherited

Helper function to parse a "host:port" address string into TNetworkAddress This is called with ipaddress:port when doing scan range assignment.

Definition at line 132 of file ScanNode.java.

Referenced by com.cloudera.impala.planner.HBaseScanNode.computeScanRangeLocations(), and com.cloudera.impala.planner.DataSourceScanNode.computeScanRangeLocations().

void com.cloudera.impala.planner.PlanNode.assignConjuncts ( Analyzer analyzer )

inlineprotectedinherited

Assign remaining unassigned conjuncts.

Definition at line 401 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.DataSourceScanNode.init(), com.cloudera.impala.planner.SortNode.init(), com.cloudera.impala.planner.AggregationNode.init(), com.cloudera.impala.planner.HBaseScanNode.init(), com.cloudera.impala.planner.HdfsScanNode.init(), com.cloudera.impala.planner.HashJoinNode.init(), and com.cloudera.impala.planner.PlanNode.init().

boolean com.cloudera.impala.planner.HdfsScanNode.canEvalUsingPartitionMd	(	Expr	expr,
		Analyzer	analyzer
	)

inlineprivate

Recursive function that checks if a given partition expr can be evaluated directly from the partition key values. If 'expr' contains any constant expressions, they are evaluated in the BE and are replaced by their corresponding results, as LiteralExprs.

Definition at line 217 of file HdfsScanNode.java.

References com.cloudera.impala.analysis.Expr.isLiteral().

Referenced by com.cloudera.impala.planner.HdfsScanNode.prunePartitions().

long com.cloudera.impala.planner.PlanNode.capAtLimit ( long cardinality )

inlineprotectedinherited

Definition at line 450 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.hasLimit(), and com.cloudera.impala.planner.PlanNode.limit_.

Referenced by com.cloudera.impala.planner.SortNode.computeStats(), com.cloudera.impala.planner.HBaseScanNode.computeStats(), com.cloudera.impala.planner.DataSourceScanNode.computeStats(), and com.cloudera.impala.planner.HdfsScanNode.computeStats().

void com.cloudera.impala.planner.HdfsScanNode.computeCosts ( TQueryOptions queryOptions )

inline

Definition at line 675 of file HdfsScanNode.java.

void com.cloudera.impala.planner.PlanNode.computeMemLayout ( Analyzer analyzer )

inlineprotectedinherited

Call computeMemLayout() for all materialized tuples.

Definition at line 475 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.tupleIds_.

Referenced by com.cloudera.impala.planner.EmptySetNode.init(), com.cloudera.impala.planner.DataSourceScanNode.init(), com.cloudera.impala.planner.SortNode.init(), com.cloudera.impala.planner.AggregationNode.init(), com.cloudera.impala.planner.AnalyticEvalNode.init(), com.cloudera.impala.planner.HBaseScanNode.init(), com.cloudera.impala.planner.HdfsScanNode.init(), and com.cloudera.impala.planner.UnionNode.init().

void com.cloudera.impala.planner.HdfsScanNode.computeScanRangeLocations ( Analyzer analyzer )

inlineprivate

Computes scan ranges (hdfs splits) plus their storage locations, including volume ids, based on the given maximum number of bytes each scan range should scan.

Definition at line 155 of file HdfsScanNode.java.

References com.cloudera.impala.catalog.HdfsPartition.getFileDescriptors(), com.cloudera.impala.planner.HdfsScanNode.partitions_, and com.cloudera.impala.planner.ScanNode.scanRanges_.

Referenced by com.cloudera.impala.planner.HdfsScanNode.init().

double com.cloudera.impala.planner.PlanNode.computeSelectivity ( )

inlineprotectedinherited

Compute the product of the selectivies of all conjuncts.

Definition at line 484 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.conjuncts_.

Referenced by com.cloudera.impala.planner.SelectNode.computeStats(), com.cloudera.impala.planner.CrossJoinNode.computeStats(), com.cloudera.impala.planner.AggregationNode.computeStats(), com.cloudera.impala.planner.HBaseScanNode.computeStats(), com.cloudera.impala.planner.DataSourceScanNode.computeStats(), and com.cloudera.impala.planner.HdfsScanNode.computeStats().

void com.cloudera.impala.planner.HdfsScanNode.computeStats ( Analyzer analyzer )

inline

Also computes totalBytes_

Definition at line 574 of file HdfsScanNode.java.

References com.cloudera.impala.planner.PlanNode.addCardinalities(), com.cloudera.impala.planner.PlanNode.capAtLimit(), com.cloudera.impala.planner.PlanNode.cardinality_, com.cloudera.impala.planner.PlanNode.computeSelectivity(), com.cloudera.impala.catalog.HdfsTable.getPartitions(), com.cloudera.impala.planner.ScanNode.inputCardinality_, com.cloudera.impala.planner.PlanNode.numNodes_, com.cloudera.impala.planner.ScanNode.numPartitionsMissingStats_, com.cloudera.impala.planner.HdfsScanNode.partitions_, com.cloudera.impala.planner.HdfsScanNode.tbl_, com.cloudera.impala.planner.HdfsScanNode.totalBytes_, and com.cloudera.impala.planner.HdfsScanNode.totalFiles_.

Referenced by com.cloudera.impala.planner.HdfsScanNode.init().

void com.cloudera.impala.planner.PlanNode.createDefaultSmap ( Analyzer analyzer )

inlineprotectedinherited

Sets outputSmap_ to compose(existing smap, combined child smap). Also substitutes conjuncts_ using the combined child smap.

Definition at line 425 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.conjuncts_, com.cloudera.impala.planner.PlanNode.getCombinedChildSmap(), and com.cloudera.impala.planner.PlanNode.outputSmap_.

Referenced by com.cloudera.impala.planner.SelectNode.init(), com.cloudera.impala.planner.AnalyticEvalNode.init(), com.cloudera.impala.planner.HashJoinNode.init(), and com.cloudera.impala.planner.PlanNode.init().

String com.cloudera.impala.planner.HdfsScanNode.debugString ( )

inlineprotected

Definition at line 112 of file HdfsScanNode.java.

References com.cloudera.impala.planner.HdfsScanNode.partitions_.

HashSet<Long> com.cloudera.impala.planner.HdfsScanNode.evalBinaryPredicate ( Expr expr )

inlineprivate

Evaluate a BinaryPredicate filter on a partition column and return the ids of the matching partitions. An empty set is returned if there are no matching partitions.

Definition at line 267 of file HdfsScanNode.java.

References com.cloudera.impala.analysis.LiteralExpr.compareTo(), com.cloudera.impala.analysis.BinaryPredicate.Operator.EQ, com.cloudera.impala.analysis.BinaryPredicate.Operator.GE, com.cloudera.impala.analysis.BinaryPredicate.Operator.GT, com.cloudera.impala.analysis.BinaryPredicate.Operator.LE, com.cloudera.impala.analysis.BinaryPredicate.Operator.LT, and com.cloudera.impala.analysis.BinaryPredicate.Operator.NE.

Referenced by com.cloudera.impala.planner.HdfsScanNode.evalSlotBindingFilter().

HashSet<Long> com.cloudera.impala.planner.HdfsScanNode.evalInPredicate ( Expr expr )

inlineprivate

Evaluate an InPredicate filter on a partition column and return the ids of the matching partitions.

Definition at line 362 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.evalSlotBindingFilter().

HashSet<Long> com.cloudera.impala.planner.HdfsScanNode.evalIsNullPredicate ( Expr expr )

inlineprivate

Evaluate an IsNullPredicate on a partition column and return the ids of the matching partitions.

Definition at line 403 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.evalSlotBindingFilter().

void com.cloudera.impala.planner.HdfsScanNode.evalPartitionFiltersInBe	(	List< HdfsPartitionFilter >	filters,
		HashSet< Long >	matchingPartitionIds,
		Analyzer	analyzer
	)		throws InternalException

inlineprivate

Evaluate a list of HdfsPartitionFilters in the BE. These are 'complex' filters that could not be evaluated from the partition key values.

Definition at line 537 of file HdfsScanNode.java.

References com.cloudera.impala.planner.HdfsScanNode.PARTITION_PRUNING_BATCH_SIZE.

Referenced by com.cloudera.impala.planner.HdfsScanNode.prunePartitions().

HashSet<Long> com.cloudera.impala.planner.HdfsScanNode.evalSlotBindingFilter ( Expr expr )

inlineprivate

Evaluate a slot binding predicate on a partition key using the partition key values; return the matching partition ids. An empty set is returned if there are no matching partitions. This function can evaluate the following types of predicates: BinaryPredicate, CompoundPredicate, IsNullPredicate, and InPredicate.

Definition at line 429 of file HdfsScanNode.java.

References com.cloudera.impala.planner.HdfsScanNode.evalBinaryPredicate(), com.cloudera.impala.planner.HdfsScanNode.evalInPredicate(), and com.cloudera.impala.planner.HdfsScanNode.evalIsNullPredicate().

Referenced by com.cloudera.impala.planner.HdfsScanNode.prunePartitions().

Set<ExprId> com.cloudera.impala.planner.PlanNode.getAssignedConjuncts ( )

inlineinherited

Definition at line 180 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.assignedConjuncts_.

float com.cloudera.impala.planner.PlanNode.getAvgRowSize ( )

inlineinherited

Definition at line 174 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.avgRowSize_.

Referenced by com.cloudera.impala.planner.CrossJoinNode.computeCosts(), com.cloudera.impala.planner.HashJoinNode.computeCosts(), and com.cloudera.impala.planner.SingleNodePlanner.createCheapestJoinPlan().

long com.cloudera.impala.planner.PlanNode.getCardinality ( )

inlineinherited

Definition at line 172 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.cardinality_.

Referenced by com.cloudera.impala.planner.CrossJoinNode.computeCosts(), com.cloudera.impala.planner.HashJoinNode.computeCosts(), com.cloudera.impala.planner.AggregationNode.computeStats(), com.cloudera.impala.planner.SingleNodePlanner.createCheapestJoinPlan(), and com.cloudera.impala.planner.SingleNodePlanner.createJoinPlan().

ExprSubstitutionMap com.cloudera.impala.planner.PlanNode.getCombinedChildSmap ( )

inlineprotectedinherited

Returns an smap that combines the childrens' smaps.

Definition at line 410 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.PlanNode.createDefaultSmap(), com.cloudera.impala.planner.SortNode.init(), com.cloudera.impala.planner.AggregationNode.init(), com.cloudera.impala.planner.AnalyticEvalNode.init(), and com.cloudera.impala.planner.HashJoinNode.init().

List<Expr> com.cloudera.impala.planner.PlanNode.getConjuncts ( )

inlineinherited

Definition at line 177 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.conjuncts_.

Referenced by com.cloudera.impala.planner.ScanNode.getInputCardinality().

final String com.cloudera.impala.planner.PlanNode.getDisplayLabel ( )

inlineprotectedinherited

Definition at line 225 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.displayName_.

Referenced by com.cloudera.impala.planner.ExchangeNode.getNodeExplainString(), com.cloudera.impala.planner.AnalyticEvalNode.getNodeExplainString(), com.cloudera.impala.planner.AggregationNode.getNodeExplainString(), com.cloudera.impala.planner.HashJoinNode.getNodeExplainString(), com.cloudera.impala.planner.HdfsScanNode.getNodeExplainString(), and com.cloudera.impala.planner.PlanNode.treeToThriftHelper().

String com.cloudera.impala.planner.HdfsScanNode.getDisplayLabelDetail ( )

inlineprotected

Definition at line 630 of file HdfsScanNode.java.

References com.cloudera.impala.planner.ScanNode.desc_, com.cloudera.impala.analysis.TupleDescriptor.getTable(), com.cloudera.impala.analysis.TupleDescriptor.hasExplicitAlias(), and path().

Referenced by com.cloudera.impala.planner.HdfsScanNode.getNodeExplainString().

String com.cloudera.impala.planner.PlanNode.getExplainString ( )

inlineinherited

Definition at line 219 of file PlanNode.java.

final String com.cloudera.impala.planner.PlanNode.getExplainString	(	String	rootPrefix,
		String	prefix,
		TExplainLevel	detailLevel
	)

inlineprotectedinherited

Generate the explain plan tree. The plan will be in the form of:

The root node header line will be prefixed by rootPrefix and the remaining plan output will be prefixed by prefix.

Definition at line 252 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.avgRowSize_, com.cloudera.impala.planner.PlanNode.cardinality_, com.cloudera.impala.planner.PlanNode.getExplainString(), com.cloudera.impala.planner.PlanNode.getNodeExplainString(), com.cloudera.impala.planner.PlanNode.getOffsetExplainString(), com.cloudera.impala.planner.PlanNode.limit_, com.cloudera.impala.planner.PlanNode.numNodes_, com.cloudera.impala.planner.PlanNode.perHostMemCost_, and com.cloudera.impala.planner.PlanNode.tupleIds_.

String com.cloudera.impala.planner.PlanNode.getExplainString ( List<?extends Expr > exprs )

inlineprotectedinherited

Definition at line 506 of file PlanNode.java.

PlanFragment com.cloudera.impala.planner.PlanNode.getFragment ( )

inlineinherited

Definition at line 176 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.fragment_.

Referenced by com.cloudera.impala.planner.ExchangeNode.addChild(), and com.cloudera.impala.planner.PipelinedPlanNodeSet.computePlanNodeSets().

PlanNodeId com.cloudera.impala.planner.PlanNode.getId ( )

inlineinherited

Definition at line 164 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.id_.

Referenced by com.cloudera.impala.planner.DataStreamSink.getExplainString(), com.cloudera.impala.planner.AggregationNode.init(), and com.cloudera.impala.planner.DataStreamSink.toThrift().

long com.cloudera.impala.planner.ScanNode.getInputCardinality ( )

inlineinherited

Definition at line 141 of file ScanNode.java.

References com.cloudera.impala.planner.PlanNode.getConjuncts(), com.cloudera.impala.planner.PlanNode.getLimit(), com.cloudera.impala.planner.PlanNode.hasLimit(), and com.cloudera.impala.planner.ScanNode.inputCardinality_.

long com.cloudera.impala.planner.PlanNode.getLimit ( )

inlineinherited

Definition at line 169 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.limit_.

Referenced by com.cloudera.impala.planner.ScanNode.getInputCardinality().

String com.cloudera.impala.planner.HdfsScanNode.getNodeExplainString	(	String	prefix,
		String	detailPrefix,
		TExplainLevel	detailLevel
	)

inlineprotected

Definition at line 644 of file HdfsScanNode.java.

Set<TupleId> com.cloudera.impala.planner.PlanNode.getNullableTupleIds ( )

inlineinherited

Definition at line 204 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.nullableTupleIds_.

int com.cloudera.impala.planner.PlanNode.getNumNodes ( )

inlineinherited

Definition at line 173 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.numNodes_.

Referenced by com.cloudera.impala.planner.PipelinedPlanNodeSet.computeResourceEstimates(), com.cloudera.impala.planner.HBaseScanNode.computeStats(), and com.cloudera.impala.planner.DataSourceScanNode.computeStats().

String com.cloudera.impala.planner.PlanNode.getOffsetExplainString ( String prefix )

inlineprotectedinherited

Return the offset_ details, if applicable. This is available separately from 'getNodeExplainString' because we want to output 'limit: ...' (which can be printed from PlanNode) before 'offset: ...', which is only printed from SortNodes right now.

Definition at line 336 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.PlanNode.getExplainString().

ExprSubstitutionMap com.cloudera.impala.planner.PlanNode.getOutputSmap ( )

inlineinherited

Definition at line 178 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.outputSmap_.

long com.cloudera.impala.planner.PlanNode.getPerHostMemCost ( )

inlineinherited

Definition at line 171 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.perHostMemCost_.

Referenced by com.cloudera.impala.planner.PipelinedPlanNodeSet.computeResourceEstimates().

static long com.cloudera.impala.planner.HdfsScanNode.getPerHostMemUpperBound ( )

inlinestatic

Hdfs scans use a shared pool of buffers managed by the I/O manager. Intuitively, the maximum number of I/O buffers is limited by the total disk bandwidth of a node. Therefore, this upper bound is independent of the number of concurrent scans and queries and helps to derive a tighter per-host memory estimate for queries with multiple concurrent scans.

Definition at line 749 of file HdfsScanNode.java.

References com.cloudera.impala.planner.HdfsScanNode.IO_MGR_BUFFER_SIZE, com.cloudera.impala.planner.HdfsScanNode.MAX_IO_BUFFERS_PER_THREAD, and com.cloudera.impala.planner.HdfsScanNode.THREADS_PER_CORE.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeCosts().

List<TScanRangeLocations> com.cloudera.impala.planner.ScanNode.getScanRangeLocations ( )

inlineinherited

Returns all scan ranges plus their locations.

Definition at line 54 of file ScanNode.java.

References com.cloudera.impala.planner.ScanNode.scanRanges_.

String com.cloudera.impala.planner.ScanNode.getStatsExplainString	(	String	prefix,
		TExplainLevel	detailLevel
	)

inlineprotectedinherited

Returns the explain string for table and columns stats to be included into the a ScanNode's explain string. The given prefix is prepended to each of the lines. The prefix is used for proper formatting when the string returned by this method is embedded in a query's explain plan.

Definition at line 75 of file ScanNode.java.

References com.cloudera.impala.planner.ScanNode.desc_, com.cloudera.impala.analysis.TupleDescriptor.getSlots(), com.cloudera.impala.analysis.TupleDescriptor.getTable(), and com.cloudera.impala.planner.ScanNode.numPartitionsMissingStats_.

Referenced by com.cloudera.impala.planner.DataSourceScanNode.getNodeExplainString(), com.cloudera.impala.planner.HBaseScanNode.getNodeExplainString(), and com.cloudera.impala.planner.HdfsScanNode.getNodeExplainString().

ArrayList<TupleId> com.cloudera.impala.planner.PlanNode.getTblRefIds ( )

inlineinherited

Definition at line 201 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.tblRefIds_.

Referenced by com.cloudera.impala.analysis.Analyzer.canEvalPredicate().

TupleDescriptor com.cloudera.impala.planner.ScanNode.getTupleDesc ( )

inlineinherited

Definition at line 49 of file ScanNode.java.

References com.cloudera.impala.planner.ScanNode.desc_.

ArrayList<TupleId> com.cloudera.impala.planner.PlanNode.getTupleIds ( )

inlineinherited

Definition at line 196 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.tupleIds_.

Referenced by com.cloudera.impala.planner.AnalyticEvalNode.AnalyticEvalNode(), com.cloudera.impala.planner.SingleNodePlanner.createInlineViewPlan(), com.cloudera.impala.planner.AnalyticPlanner.createSortInfo(), and com.cloudera.impala.planner.SelectNode.SelectNode().

boolean com.cloudera.impala.planner.PlanNode.hasLimit ( )

inlineinherited

Definition at line 170 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.limit_.

Referenced by com.cloudera.impala.planner.PlanNode.capAtLimit(), com.cloudera.impala.planner.ExchangeNode.computeStats(), com.cloudera.impala.planner.ScanNode.getInputCardinality(), and com.cloudera.impala.planner.SortNode.getNodeExplainDetail().

boolean com.cloudera.impala.planner.PlanNode.hasValidStats ( )

inlineprotectedinherited

Returns true if stats-related variables are valid.

Definition at line 519 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.cardinality_, and com.cloudera.impala.planner.PlanNode.numNodes_.

Referenced by com.cloudera.impala.planner.SortNode.computeCosts(), and com.cloudera.impala.planner.HashJoinNode.computeStats().

void com.cloudera.impala.planner.HdfsScanNode.init ( Analyzer analyzer ) throws InternalException

inline

Populate conjuncts_, partitions_, and scanRanges_.

Definition at line 124 of file HdfsScanNode.java.

References com.cloudera.impala.planner.PlanNode.assignConjuncts(), com.cloudera.impala.planner.PlanNode.assignedConjuncts_, com.cloudera.impala.planner.PlanNode.computeMemLayout(), com.cloudera.impala.planner.HdfsScanNode.computeScanRangeLocations(), com.cloudera.impala.planner.HdfsScanNode.computeStats(), com.cloudera.impala.planner.PlanNode.conjuncts_, com.cloudera.impala.planner.PlanNode.markSlotsMaterialized(), and com.cloudera.impala.planner.HdfsScanNode.prunePartitions().

boolean com.cloudera.impala.planner.PlanNode.isBlockingNode ( )

inlineinherited

Returns true if this plan node can output its first row only after consuming all rows of all its children. This method is used to group plan nodes into pipelined units for resource estimation.

Definition at line 555 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.PipelinedPlanNodeSet.computePlanNodeSets().

boolean com.cloudera.impala.planner.ScanNode.isTableMissingColumnStats ( )

inlineinherited

Definition at line 120 of file ScanNode.java.

References com.cloudera.impala.planner.ScanNode.desc_, and com.cloudera.impala.analysis.TupleDescriptor.getSlots().

Referenced by com.cloudera.impala.planner.ScanNode.isTableMissingStats().

boolean com.cloudera.impala.planner.ScanNode.isTableMissingStats ( )

inlineinherited

Returns true if the table underlying this scan is missing table stats or column stats relevant to this scan node.

Definition at line 111 of file ScanNode.java.

References com.cloudera.impala.planner.ScanNode.isTableMissingColumnStats(), and com.cloudera.impala.planner.ScanNode.isTableMissingTableStats().

boolean com.cloudera.impala.planner.ScanNode.isTableMissingTableStats ( )

inlineinherited

Definition at line 115 of file ScanNode.java.

References com.cloudera.impala.planner.ScanNode.desc_, com.cloudera.impala.analysis.TupleDescriptor.getTable(), and com.cloudera.impala.planner.ScanNode.numPartitionsMissingStats_.

Referenced by com.cloudera.impala.planner.ScanNode.isTableMissingStats().

void com.cloudera.impala.planner.PlanNode.markSlotsMaterialized	(	Analyzer	analyzer,
		List< Expr >	exprs
	)

inlineprotectedinherited

Marks all slots referenced in exprs as materialized.

Definition at line 464 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.init().

static long com.cloudera.impala.planner.PlanNode.multiplyCardinalities	(	long	a,
		long	b
	)

inlinestaticinherited

Computes and returns the product of two cardinalities. If an overflow occurs, the maximum Long value is returned (Long.MAX_VALUE).

Definition at line 541 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.CrossJoinNode.computeStats(), and com.cloudera.impala.planner.HashJoinNode.getJoinCardinality().

void com.cloudera.impala.planner.HdfsScanNode.prunePartitions ( Analyzer analyzer ) throws InternalException

inlineprivate

Populate partitions_ based on all applicable conjuncts and remove conjuncts used for filtering from conjuncts_.

Definition at line 458 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.init().

void com.cloudera.impala.planner.PlanNode.setAssignedConjuncts ( Set< ExprId > conjuncts )

inlineinherited

Definition at line 181 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.assignedConjuncts_.

void com.cloudera.impala.planner.PlanNode.setDisplayName ( String s )

inlineprotectedinherited

Definition at line 223 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.displayName_.

void com.cloudera.impala.planner.PlanNode.setFragment ( PlanFragment fragment )

inlineinherited

Definition at line 175 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.fragment_.

void com.cloudera.impala.planner.PlanNode.setId ( PlanNodeId id )

inlineinherited

Definition at line 165 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.id_.

void com.cloudera.impala.planner.PlanNode.setLimit ( long limit )

inlineinherited

Set the limit_ to the given limit_ only if the limit_ hasn't been set, or the new limit_ is lower.

Parameters

limit_

Definition at line 190 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.limit_.

void com.cloudera.impala.planner.PlanNode.setOutputSmap ( ExprSubstitutionMap smap )

inlineinherited

Definition at line 179 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.outputSmap_.

void com.cloudera.impala.planner.PlanNode.setTblRefIds ( ArrayList< TupleId > ids )

inlineinherited

Definition at line 202 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.tblRefIds_.

void com.cloudera.impala.planner.HdfsScanNode.toThrift ( TPlanNode msg )

inlineprotected

Definition at line 623 of file HdfsScanNode.java.

References com.cloudera.impala.planner.ScanNode.desc_, and com.cloudera.impala.analysis.TupleDescriptor.getId().

void com.cloudera.impala.planner.PlanNode.transferConjuncts ( PlanNode recipient )

inlineinherited

Definition at line 214 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.conjuncts_.

TPlan com.cloudera.impala.planner.PlanNode.treeToThrift ( )

inlineinherited

Definition at line 341 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.treeToThriftHelper().

void com.cloudera.impala.planner.PlanNode.unsetLimit ( )

inlineinherited

Definition at line 194 of file PlanNode.java.

References com.cloudera.impala.planner.PlanNode.limit_.

Member Data Documentation

Set<ExprId> com.cloudera.impala.planner.PlanNode.assignedConjuncts_

protectedinherited

Definition at line 99 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.PlanNode.getAssignedConjuncts(), com.cloudera.impala.planner.CrossJoinNode.init(), com.cloudera.impala.planner.HdfsScanNode.init(), com.cloudera.impala.planner.HashJoinNode.init(), and com.cloudera.impala.planner.PlanNode.setAssignedConjuncts().

float com.cloudera.impala.planner.PlanNode.avgRowSize_

protectedinherited

Definition at line 110 of file PlanNode.java.

long com.cloudera.impala.planner.PlanNode.cardinality_

protectedinherited

Definition at line 103 of file PlanNode.java.

List<Expr> com.cloudera.impala.planner.PlanNode.conjuncts_ = Lists.newArrayList()

protectedinherited

Definition at line 86 of file PlanNode.java.

final int com.cloudera.impala.planner.PlanNode.DEFAULT_BATCH_SIZE = 1024

staticprotectedinherited

Definition at line 63 of file PlanNode.java.

final TupleDescriptor com.cloudera.impala.planner.ScanNode.desc_

protectedinherited

Definition at line 33 of file ScanNode.java.

String com.cloudera.impala.planner.PlanNode.displayName_

protectedinherited

Definition at line 66 of file PlanNode.java.

PlanFragment com.cloudera.impala.planner.PlanNode.fragment_

protectedinherited

Definition at line 90 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.AnalyticEvalNode.computeCosts(), com.cloudera.impala.planner.AggregationNode.computeCosts(), com.cloudera.impala.planner.ExchangeNode.getDisplayLabelDetail(), com.cloudera.impala.planner.PlanNode.getFragment(), com.cloudera.impala.planner.HdfsScanNode.getNodeExplainString(), com.cloudera.impala.planner.UnionNode.reorderOperands(), and com.cloudera.impala.planner.PlanNode.setFragment().

PlanNodeId com.cloudera.impala.planner.PlanNode.id_

protectedinherited

Definition at line 69 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.PlanNode.getId(), com.cloudera.impala.planner.PlanNode.PlanNode(), and com.cloudera.impala.planner.PlanNode.setId().

long com.cloudera.impala.planner.ScanNode.inputCardinality_ = -1

protectedinherited

Definition at line 36 of file ScanNode.java.

Referenced by com.cloudera.impala.planner.HBaseScanNode.computeStats(), com.cloudera.impala.planner.DataSourceScanNode.computeStats(), com.cloudera.impala.planner.HdfsScanNode.computeStats(), and com.cloudera.impala.planner.ScanNode.getInputCardinality().

final long com.cloudera.impala.planner.HdfsScanNode.IO_MGR_BUFFER_SIZE = 8L * 1024L * 1024L

staticprivate

Definition at line 76 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeCosts(), and com.cloudera.impala.planner.HdfsScanNode.getPerHostMemUpperBound().

long com.cloudera.impala.planner.PlanNode.limit_

protectedinherited

Definition at line 71 of file PlanNode.java.

final Logger com.cloudera.impala.planner.HdfsScanNode.LOG = LoggerFactory.getLogger(HdfsScanNode.class)

staticprivate

Definition at line 73 of file HdfsScanNode.java.

final long com.cloudera.impala.planner.HdfsScanNode.MAX_IO_BUFFERS_PER_THREAD = 10

staticprivate

Definition at line 79 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeCosts(), and com.cloudera.impala.planner.HdfsScanNode.getPerHostMemUpperBound().

Set<TupleId> com.cloudera.impala.planner.PlanNode.nullableTupleIds_ = Sets.newHashSet()

protectedinherited

Definition at line 84 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.ExchangeNode.addChild(), com.cloudera.impala.planner.PlanNode.getNullableTupleIds(), and com.cloudera.impala.planner.PlanNode.PlanNode().

int com.cloudera.impala.planner.PlanNode.numNodes_

protectedinherited

Definition at line 107 of file PlanNode.java.

int com.cloudera.impala.planner.ScanNode.numPartitionsMissingStats_ = 0

protectedinherited

Definition at line 39 of file ScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeStats(), com.cloudera.impala.planner.ScanNode.getStatsExplainString(), and com.cloudera.impala.planner.ScanNode.isTableMissingTableStats().

ExprSubstitutionMap com.cloudera.impala.planner.PlanNode.outputSmap_

protectedinherited

Definition at line 93 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.PlanNode.createDefaultSmap(), com.cloudera.impala.planner.PlanNode.getOutputSmap(), com.cloudera.impala.planner.SortNode.init(), com.cloudera.impala.planner.AggregationNode.init(), com.cloudera.impala.planner.AnalyticEvalNode.init(), and com.cloudera.impala.planner.PlanNode.setOutputSmap().

final int com.cloudera.impala.planner.HdfsScanNode.PARTITION_PRUNING_BATCH_SIZE = 1024

staticprivate

Definition at line 90 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.evalPartitionFiltersInBe().

final ArrayList<HdfsPartition> com.cloudera.impala.planner.HdfsScanNode.partitions_ = Lists.newArrayList()

private

Definition at line 95 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeScanRangeLocations(), com.cloudera.impala.planner.HdfsScanNode.computeStats(), and com.cloudera.impala.planner.HdfsScanNode.debugString().

long com.cloudera.impala.planner.PlanNode.perHostMemCost_ = -1

protectedinherited

Definition at line 114 of file PlanNode.java.

final double com.cloudera.impala.planner.HdfsScanNode.SCAN_RANGE_SKEW_FACTOR = 1.2

staticprivate

Definition at line 87 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeCosts().

List<TScanRangeLocations> com.cloudera.impala.planner.ScanNode.scanRanges_

protectedinherited

Definition at line 42 of file ScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeCosts(), com.cloudera.impala.planner.HdfsScanNode.computeScanRangeLocations(), com.cloudera.impala.planner.HBaseScanNode.computeScanRangeLocations(), com.cloudera.impala.planner.DataSourceScanNode.computeScanRangeLocations(), and com.cloudera.impala.planner.ScanNode.getScanRangeLocations().

final HdfsTable com.cloudera.impala.planner.HdfsScanNode.tbl_

private

Definition at line 92 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeStats(), com.cloudera.impala.planner.HdfsScanNode.getNodeExplainString(), com.cloudera.impala.planner.HdfsScanNode.HdfsScanNode(), and com.cloudera.impala.planner.HdfsScanNode.prunePartitions().

ArrayList<TupleId> com.cloudera.impala.planner.PlanNode.tblRefIds_

protectedinherited

Definition at line 79 of file PlanNode.java.

Referenced by com.cloudera.impala.planner.PlanNode.getTblRefIds(), com.cloudera.impala.planner.PlanNode.PlanNode(), and com.cloudera.impala.planner.PlanNode.setTblRefIds().

final int com.cloudera.impala.planner.HdfsScanNode.THREADS_PER_CORE = 3

staticprivate

Definition at line 82 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeCosts(), and com.cloudera.impala.planner.HdfsScanNode.getPerHostMemUpperBound().

long com.cloudera.impala.planner.HdfsScanNode.totalBytes_ = 0

private

Definition at line 101 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeCosts(), com.cloudera.impala.planner.HdfsScanNode.computeStats(), and com.cloudera.impala.planner.HdfsScanNode.getNodeExplainString().

long com.cloudera.impala.planner.HdfsScanNode.totalFiles_ = 0

private

Definition at line 98 of file HdfsScanNode.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeStats(), and com.cloudera.impala.planner.HdfsScanNode.getNodeExplainString().

ArrayList<TupleId> com.cloudera.impala.planner.PlanNode.tupleIds_

protectedinherited

Definition at line 74 of file PlanNode.java.

The documentation for this class was generated from the following file:

fe/src/main/java/com/cloudera/impala/planner/HdfsScanNode.java

Public Member Functions

Static Public Member Functions

Protected Member Functions

Static Protected Member Functions

Protected Attributes

Static Protected Attributes

Private Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation