Inheritance diagram for com.cloudera.impala.catalog.HdfsPartition:

Collaboration diagram for com.cloudera.impala.catalog.HdfsPartition:

Classes
class	BlockReplica

class	CachedHmsPartitionDescriptor

class	FileBlock

class	FileDescriptor

Public Member Functions
HdfsStorageDescriptor	getInputFormatDescriptor ()

boolean	isDefaultPartition ()

String	getPartitionName ()

List< String >	getPartitionValuesAsStrings (boolean mapNullsToHiveKey)

String	getConjunctSql ()

String	getValuesAsString ()

String	getLocation ()

long	getId ()

HdfsTable	getTable ()

void	setNumRows (long numRows)

long	getNumRows ()

boolean	isMarkedCached ()

void	setFileFormat (HdfsFileFormat fileFormat)

void	setLocation (String location)

org.apache.hadoop.hive.metastore.api.SerDeInfo	getSerdeInfo ()

TPartitionStats	getPartitionStats ()

boolean	hasIncrementalStats ()

TAccessLevel	getAccessLevel ()

String	getParameter (String key)

Map< String, String >	getParameters ()

void	putToParameters (String k, String v)

void	markDirty ()

boolean	isDirty ()

List< LiteralExpr >	getPartitionValues ()

List < HdfsPartition.FileDescriptor >	getFileDescriptors ()

boolean	hasFileDescriptors ()

org.apache.hadoop.hive.metastore.api.Partition	toHmsPartition ()

	HdfsPartition (HdfsTable table, org.apache.hadoop.hive.metastore.api.Partition msPartition, List< LiteralExpr > partitionKeyValues, HdfsStorageDescriptor fileFormatDescriptor, List< HdfsPartition.FileDescriptor > fileDescriptors, TAccessLevel accessLevel)

long	getSize ()

String	toString ()

void	checkWellFormed () throws CatalogException

THdfsPartition	toThrift (boolean includeFileDesc)

int	compareTo (HdfsPartition o)

Static Public Member Functions
static HdfsPartition	defaultPartition (HdfsTable table, HdfsStorageDescriptor storageDescriptor)

static HdfsPartition	fromThrift (HdfsTable table, long id, THdfsPartition thriftPartition)

static int	comparePartitionKeyValues (List< LiteralExpr > lhs, List< LiteralExpr > rhs)

Package Functions
void	markCached ()

Private Member Functions
	HdfsPartition (HdfsTable table, org.apache.hadoop.hive.metastore.api.Partition msPartition, List< LiteralExpr > partitionKeyValues, HdfsStorageDescriptor fileFormatDescriptor, List< HdfsPartition.FileDescriptor > fileDescriptors, long id, String location, TAccessLevel accessLevel)

Private Attributes
final HdfsTable	table_

final List< LiteralExpr >	partitionKeyValues_

long	numRows_ = -1

final long	id_

final HdfsStorageDescriptor	fileFormatDescriptor_

final List< FileDescriptor >	fileDescriptors_

String	location_

boolean	isDirty_ = false

boolean	isMarkedCached_ = false

final TAccessLevel	accessLevel_

Map< String, String >	hmsParameters_

final CachedHmsPartitionDescriptor	cachedMsPartitionDescriptor_

Static Private Attributes
static AtomicLong	partitionIdCounter_ = new AtomicLong()

static final Logger	LOG = LoggerFactory.getLogger(HdfsPartition.class)

Detailed Description

Query-relevant information for one table partition. Partitions are comparable based on their partition-key values. The comparison orders partitions in ascending order with NULLs sorting last. The ordering is useful for displaying partitions in SHOW statements.

Definition at line 60 of file HdfsPartition.java.

Constructor & Destructor Documentation

com.cloudera.impala.catalog.HdfsPartition.HdfsPartition	(	HdfsTable	table,
		org.apache.hadoop.hive.metastore.api.Partition	msPartition,
		List< LiteralExpr >	partitionKeyValues,
		HdfsStorageDescriptor	fileFormatDescriptor,
		List< HdfsPartition.FileDescriptor >	fileDescriptors,
		long	id,
		String	location,
		TAccessLevel	accessLevel
	)

inlineprivate

Definition at line 514 of file HdfsPartition.java.

Referenced by com.cloudera.impala.catalog.HdfsPartition.defaultPartition(), and com.cloudera.impala.catalog.HdfsPartition.fromThrift().

com.cloudera.impala.catalog.HdfsPartition.HdfsPartition	(	HdfsTable	table,
		org.apache.hadoop.hive.metastore.api.Partition	msPartition,
		List< LiteralExpr >	partitionKeyValues,
		HdfsStorageDescriptor	fileFormatDescriptor,
		List< HdfsPartition.FileDescriptor >	fileDescriptors,
		TAccessLevel	accessLevel
	)

inline

Definition at line 552 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsTable.getLocation().

Member Function Documentation

void com.cloudera.impala.catalog.HdfsPartition.checkWellFormed ( ) throws CatalogException

inline

Checks that this partition's metadata is well formed. This does not necessarily mean the partition is supported by Impala. Throws a CatalogException if there are any errors in the partition metadata.

Definition at line 658 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.getPartitionName(), and com.cloudera.impala.catalog.HdfsPartition.getPartitionValues().

static int com.cloudera.impala.catalog.HdfsPartition.comparePartitionKeyValues	(	List< LiteralExpr >	lhs,
		List< LiteralExpr >	rhs
	)

inlinestatic

Definition at line 704 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.compareTo().

Referenced by com.cloudera.impala.catalog.HdfsPartition.compareTo().

int com.cloudera.impala.catalog.HdfsPartition.compareTo ( HdfsPartition o )

inline

Comparison method to allow ordering of HdfsPartitions by their partition-key values.

Definition at line 699 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.comparePartitionKeyValues(), com.cloudera.impala.catalog.HdfsPartition.getPartitionValues(), and com.cloudera.impala.catalog.HdfsPartition.partitionKeyValues_.

Referenced by com.cloudera.impala.catalog.HdfsPartition.comparePartitionKeyValues().

static HdfsPartition com.cloudera.impala.catalog.HdfsPartition.defaultPartition	(	HdfsTable	table,
		HdfsStorageDescriptor	storageDescriptor
	)

inlinestatic

Definition at line 562 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.HdfsPartition().

static HdfsPartition com.cloudera.impala.catalog.HdfsPartition.fromThrift	(	HdfsTable	table,
		long	id,
		THdfsPartition	thriftPartition
	)

inlinestatic

Definition at line 590 of file HdfsPartition.java.

References com.cloudera.impala.catalog.Table.getName(), and com.cloudera.impala.catalog.HdfsPartition.HdfsPartition().

TAccessLevel com.cloudera.impala.catalog.HdfsPartition.getAccessLevel ( )

inline

Returns the HDFS permissions Impala has to this partition's directory - READ_ONLY, READ_WRITE, etc.

Definition at line 404 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.accessLevel_.

Referenced by com.cloudera.impala.analysis.LoadDataStmt.analyzePaths(), and com.cloudera.impala.catalog.HdfsTable.loadPartitions().

String com.cloudera.impala.catalog.HdfsPartition.getConjunctSql ( )

inline

Utility method which returns a string of conjuncts of equality exprs to exactly select this partition (e.g. ((month=2009) AND (year=2012)). TODO: Remove this when TODO elsewhere in this file to save and expose the list of TPartitionKeyValues has been resolved.

Definition at line 322 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.getPartitionValues(), and com.cloudera.impala.catalog.HdfsPartition.getTable().

List<HdfsPartition.FileDescriptor> com.cloudera.impala.catalog.HdfsPartition.getFileDescriptors ( )

inline

Definition at line 429 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.fileDescriptors_.

Referenced by com.cloudera.impala.planner.HdfsScanNode.computeScanRangeLocations().

long com.cloudera.impala.catalog.HdfsPartition.getId ( )

inline

Definition at line 365 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.id_.

Referenced by com.cloudera.impala.catalog.HdfsPartition.toThrift(), and com.cloudera.impala.catalog.HdfsTable.updatePartitionMdAndColStats().

HdfsStorageDescriptor com.cloudera.impala.catalog.HdfsPartition.getInputFormatDescriptor ( )

inline

Definition at line 270 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.fileFormatDescriptor_.

Referenced by com.cloudera.impala.catalog.HdfsPartition.HdfsPartition().

String com.cloudera.impala.catalog.HdfsPartition.getLocation ( )

inline

Returns the storage location (HDFS path) of this partition. Should only be called for partitioned tables.

Definition at line 364 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.location_.

Referenced by com.cloudera.impala.analysis.LoadDataStmt.analyzePaths().

long com.cloudera.impala.catalog.HdfsPartition.getNumRows ( )

inline

Definition at line 368 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.numRows_.

String com.cloudera.impala.catalog.HdfsPartition.getParameter ( String key )

inline

Returns the HMS parameter with key 'key' if it exists, otherwise returns null.

Definition at line 409 of file HdfsPartition.java.

Map<String, String> com.cloudera.impala.catalog.HdfsPartition.getParameters ( )

inline

Definition at line 413 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.hmsParameters_.

Referenced by com.cloudera.impala.service.CatalogOpExecutor.alterPartitionSetCached(), com.cloudera.impala.catalog.PartitionStatsUtil.deletePartStats(), and com.cloudera.impala.catalog.HdfsPartition.toHmsPartition().

String com.cloudera.impala.catalog.HdfsPartition.getPartitionName ( )

inline

Return a partition name formed by concatenating partition keys and their values, compatible with the way Hive names partitions. Reuses Hive's org.apache.hadoop.hive.common.FileUtils.makePartName() function to build the name string because there are a number of special cases for how partition names are URL escaped. TODO: Consider storing the PartitionKeyValue in HdfsPartition. It would simplify this code would be useful in other places, such as fromThrift().

Definition at line 287 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.getPartitionValuesAsStrings(), and com.cloudera.impala.catalog.HdfsPartition.getTable().

Referenced by com.cloudera.impala.catalog.HdfsPartition.checkWellFormed(), and com.cloudera.impala.catalog.HdfsPartition.getPartitionStats().

TPartitionStats com.cloudera.impala.catalog.HdfsPartition.getPartitionStats ( )

inline

Definition at line 384 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.getPartitionName(), and com.cloudera.impala.catalog.HdfsPartition.hmsParameters_.

Referenced by com.cloudera.impala.service.CatalogOpExecutor.dropStats(), and com.cloudera.impala.catalog.HdfsPartition.hasIncrementalStats().

List<LiteralExpr> com.cloudera.impala.catalog.HdfsPartition.getPartitionValues ( )

inline

Returns an immutable list of partition key expressions

Definition at line 428 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.partitionKeyValues_.

Referenced by com.cloudera.impala.catalog.HdfsPartition.checkWellFormed(), com.cloudera.impala.catalog.HdfsPartition.compareTo(), com.cloudera.impala.catalog.HdfsPartition.getConjunctSql(), com.cloudera.impala.catalog.HdfsPartition.getPartitionValuesAsStrings(), com.cloudera.impala.catalog.HdfsPartition.getValuesAsString(), com.cloudera.impala.catalog.HdfsPartition.toThrift(), and com.cloudera.impala.catalog.HdfsTable.updatePartitionMdAndColStats().

List<String> com.cloudera.impala.catalog.HdfsPartition.getPartitionValuesAsStrings ( boolean mapNullsToHiveKey )

inline

Returns a list of partition values as strings. If mapNullsToHiveKey is true, any NULL value is returned as the table's default null partition key string value, otherwise they are returned as 'NULL'.

Definition at line 303 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.getPartitionValues(), and com.cloudera.impala.catalog.HdfsPartition.getTable().

Referenced by com.cloudera.impala.catalog.HdfsPartition.getPartitionName(), and com.cloudera.impala.catalog.HdfsPartition.toHmsPartition().

org.apache.hadoop.hive.metastore.api.SerDeInfo com.cloudera.impala.catalog.HdfsPartition.getSerdeInfo ( )

inline

Definition at line 378 of file HdfsPartition.java.

long com.cloudera.impala.catalog.HdfsPartition.getSize ( )

inline

Return the size (in bytes) of all the files inside this partition

Definition at line 575 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.fileDescriptors_.

HdfsTable com.cloudera.impala.catalog.HdfsPartition.getTable ( )

inline

Definition at line 366 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.table_.

Referenced by com.cloudera.impala.catalog.HdfsPartition.getConjunctSql(), com.cloudera.impala.catalog.HdfsPartition.getPartitionName(), com.cloudera.impala.catalog.HdfsPartition.getPartitionValuesAsStrings(), com.cloudera.impala.catalog.HdfsPartition.getValuesAsString(), and com.cloudera.impala.catalog.HdfsPartition.toHmsPartition().

String com.cloudera.impala.catalog.HdfsPartition.getValuesAsString ( )

inline

Returns a string of the form part_key1=value1/part_key2=value2...

Definition at line 347 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.getPartitionValues(), and com.cloudera.impala.catalog.HdfsPartition.getTable().

boolean com.cloudera.impala.catalog.HdfsPartition.hasFileDescriptors ( )

inline

Definition at line 433 of file HdfsPartition.java.

Referenced by com.cloudera.impala.planner.HdfsScanNode.prunePartitions().

boolean com.cloudera.impala.catalog.HdfsPartition.hasIncrementalStats ( )

inline

Definition at line 395 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.getPartitionStats().

boolean com.cloudera.impala.catalog.HdfsPartition.isDefaultPartition ( )

inline

Definition at line 274 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.id_.

boolean com.cloudera.impala.catalog.HdfsPartition.isDirty ( )

inline

Definition at line 423 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.isDirty_.

boolean com.cloudera.impala.catalog.HdfsPartition.isMarkedCached ( )

inline

Definition at line 369 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.isMarkedCached_.

Referenced by com.cloudera.impala.service.CatalogOpExecutor.alterPartitionSetCached(), com.cloudera.impala.service.CatalogOpExecutor.alterTableDropPartition(), and com.cloudera.impala.analysis.AlterTableSetLocationStmt.analyze().

void com.cloudera.impala.catalog.HdfsPartition.markCached ( )

inlinepackage

Definition at line 370 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.isMarkedCached_.

void com.cloudera.impala.catalog.HdfsPartition.markDirty ( )

inline

Marks this partition's metadata as "dirty" indicating that changes have been made and this partition's metadata should not be reused during the next incremental metadata refresh.

Definition at line 422 of file HdfsPartition.java.

References com.cloudera.impala.catalog.HdfsPartition.isDirty_.