Collaboration diagram for com.cloudera.impala.common.FileSystemUtil:

Static Public Member Functions
static int	deleteAllVisibleFiles (Path directory) throws IOException

static int	getTotalNumVisibleFiles (Path directory) throws IOException

static int	relocateAllVisibleFiles (Path sourceDir, Path destDir) throws IOException

static void	relocateFile (Path sourceFile, Path dest, boolean renameIfAlreadyExists) throws IOException

static String	readFile (Path file) throws IOException

static boolean	containsSubdirectory (Path directory) throws FileNotFoundException, IOException

static Path	makeTmpSubdirectory (Path directory) throws IOException

static boolean	isHiddenFile (String fileName)

static boolean	hasGetFileBlockLocations (FileSystem fs)

static boolean	isDistributedFileSystem (FileSystem fs)

static boolean	isDistributedFileSystem (Path path) throws IOException

static DistributedFileSystem	getDistributedFileSystem () throws IOException

static Path	createFullyQualifiedPath (Path location)

static Boolean	isPathOnFileSystem (Path path, FileSystem fs)

static Boolean	isPathReachable (Path path, FileSystem fs, StringBuilder error_msg)

static Configuration	getConfiguration ()

Static Private Member Functions
static boolean	arePathsInSameEncryptionZone (FileSystem fs, Path p1, Path p2) throws IOException

static String	appendToBaseFileName (String baseFileName, String appendStr)

Static Private Attributes
static final Configuration	CONF = new Configuration()

static final Logger	LOG = Logger.getLogger(FileSystemUtil.class)

Detailed Description

Common utility functions for operating on FileSystem objects.

Definition at line 43 of file FileSystemUtil.java.

Member Function Documentation

static String com.cloudera.impala.common.FileSystemUtil.appendToBaseFileName	(	String	baseFileName,
		String	appendStr
	)

inlinestaticprivate

Builds a new file name based on a base file name. This is done by inserting the given appendStr into the base file name, preserving the file extension (if one exists). For example, this could be passed a UUID string to uniquify files: file1.snap -> file1_<uuid>.snap file1 -> file1_<uuid>

Definition at line 200 of file FileSystemUtil.java.

Referenced by com.cloudera.impala.common.FileSystemUtil.relocateAllVisibleFiles(), and com.cloudera.impala.common.FileSystemUtil.relocateFile().

static boolean com.cloudera.impala.common.FileSystemUtil.arePathsInSameEncryptionZone	(	FileSystem	fs,
		Path	p1,
		Path	p2
	)		throws IOException

inlinestaticprivate

Returns true if path p1 and path p2 are in the same encryption zone.

Definition at line 86 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF.

Referenced by com.cloudera.impala.common.FileSystemUtil.relocateFile().

static boolean com.cloudera.impala.common.FileSystemUtil.containsSubdirectory ( Path directory ) throws FileNotFoundException, IOException

inlinestatic

Returns true if the given Path contains any sub directories, otherwise false.

Definition at line 215 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF.

Referenced by com.cloudera.impala.analysis.LoadDataStmt.analyzePaths().

static Path com.cloudera.impala.common.FileSystemUtil.createFullyQualifiedPath ( Path location )

inlinestatic

Fully-qualifies the given path based on the FileSystem configuration.

Definition at line 281 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF.

static int com.cloudera.impala.common.FileSystemUtil.deleteAllVisibleFiles ( Path directory ) throws IOException

inlinestatic

Performs a non-recursive delete of all visible (non-hidden) files in a given directory. Returns the number of files deleted as part of this operation.

Definition at line 51 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF, and com.cloudera.impala.common.FileSystemUtil.isHiddenFile().

static Configuration com.cloudera.impala.common.FileSystemUtil.getConfiguration ( )

inlinestatic

Returns the configuration.

Definition at line 330 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF.

static DistributedFileSystem com.cloudera.impala.common.FileSystemUtil.getDistributedFileSystem ( ) throws IOException

inlinestatic

Definition at line 271 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF, and path().

static int com.cloudera.impala.common.FileSystemUtil.getTotalNumVisibleFiles ( Path directory ) throws IOException

inlinestatic

Returns the total number of visible (non-hidden) files in a directory.

Definition at line 70 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF, and com.cloudera.impala.common.FileSystemUtil.isHiddenFile().

Referenced by com.cloudera.impala.analysis.LoadDataStmt.analyzePaths().

static boolean com.cloudera.impala.common.FileSystemUtil.hasGetFileBlockLocations ( FileSystem fs )

inlinestatic

Returns true if the filesystem might override getFileBlockLocations().

Definition at line 249 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.isDistributedFileSystem().

Referenced by com.cloudera.impala.catalog.HdfsTable.loadBlockMetadata().

static boolean com.cloudera.impala.common.FileSystemUtil.isDistributedFileSystem ( FileSystem fs )

inlinestatic

Returns true iff the filesystem is a DistributedFileSystem.

Definition at line 260 of file FileSystemUtil.java.

Referenced by com.cloudera.impala.common.FileSystemUtil.hasGetFileBlockLocations(), com.cloudera.impala.common.FileSystemUtil.isDistributedFileSystem(), and com.cloudera.impala.analysis.InsertStmt.setTargetTable().

static boolean com.cloudera.impala.common.FileSystemUtil.isDistributedFileSystem ( Path path ) throws IOException

inlinestatic

Return true iff path is on a DFS filesystem.

Definition at line 267 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF, com.cloudera.impala.common.FileSystemUtil.isDistributedFileSystem(), and path().

static boolean com.cloudera.impala.common.FileSystemUtil.isHiddenFile ( String fileName )

inlinestatic

Definition at line 237 of file FileSystemUtil.java.

Referenced by com.cloudera.impala.analysis.LoadDataStmt.analyzePaths(), com.cloudera.impala.common.FileSystemUtil.deleteAllVisibleFiles(), com.cloudera.impala.common.FileSystemUtil.getTotalNumVisibleFiles(), and com.cloudera.impala.common.FileSystemUtil.relocateAllVisibleFiles().

static Boolean com.cloudera.impala.common.FileSystemUtil.isPathOnFileSystem	(	Path	path,
		FileSystem	fs
	)

inlinestatic

Return true iff the path is on the given filesystem.

Definition at line 298 of file FileSystemUtil.java.

References path().

Referenced by com.cloudera.impala.analysis.LoadDataStmt.analyzePaths(), and com.cloudera.impala.common.FileSystemUtil.relocateFile().

static Boolean com.cloudera.impala.common.FileSystemUtil.isPathReachable	(	Path	path,
		FileSystem	fs,
		StringBuilder	error_msg
	)

inlinestatic

Return true if the path can be reached, false for all other cases File doesn't exist, cannot access the FileSystem, etc.

Definition at line 314 of file FileSystemUtil.java.

Referenced by com.cloudera.impala.analysis.HdfsUri.analyze(), and com.cloudera.impala.catalog.HdfsTable.getAvroSchema().

static Path com.cloudera.impala.common.FileSystemUtil.makeTmpSubdirectory ( Path directory ) throws IOException

inlinestatic

Makes a temporary unique directory within the given directory.

Definition at line 230 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF.

static String com.cloudera.impala.common.FileSystemUtil.readFile ( Path file ) throws IOException

inlinestatic

Reads the file at path and returns the contents.

Definition at line 182 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.CONF.

static int com.cloudera.impala.common.FileSystemUtil.relocateAllVisibleFiles	(	Path	sourceDir,
		Path	destDir
	)		throws IOException

inlinestatic

Relocates all visible (non-hidden) files from a source directory to a destination directory. Files are moved (renamed) to the new location unless the source and destination directories are in different encryption zones, in which case the files are copied so that they are decrypted and/or encrypted. Naming conflicts are resolved by appending a UUID to the base file name. Any sub-directories within the source directory are skipped. Returns the number of files relocated as part of this operation.

Definition at line 105 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.appendToBaseFileName(), com.cloudera.impala.common.FileSystemUtil.CONF, and com.cloudera.impala.common.FileSystemUtil.isHiddenFile().

static void com.cloudera.impala.common.FileSystemUtil.relocateFile	(	Path	sourceFile,
		Path	dest,
		boolean	renameIfAlreadyExists
	)		throws IOException

inlinestatic

Relocates the given file to a new location (either another directory or a file. The file is moved (renamed) to the new location unless the source and destination are in different encryption zones, in which case the file is copied so that the file can be decrypted and/or encrypted. If renameIfAlreadyExists is true, no error will be thrown if a file with the same name already exists in the destination location. Instead, a UUID will be appended to the base file name, preserving the the existing file extension. If renameIfAlreadyExists is false, an IOException will be thrown if there is a file name conflict.

Definition at line 147 of file FileSystemUtil.java.

References com.cloudera.impala.common.FileSystemUtil.appendToBaseFileName(), com.cloudera.impala.common.FileSystemUtil.arePathsInSameEncryptionZone(), com.cloudera.impala.common.FileSystemUtil.CONF, and com.cloudera.impala.common.FileSystemUtil.isPathOnFileSystem().

Member Data Documentation

final Configuration com.cloudera.impala.common.FileSystemUtil.CONF = new Configuration()

staticprivate

Definition at line 44 of file FileSystemUtil.java.

final Logger com.cloudera.impala.common.FileSystemUtil.LOG = Logger.getLogger(FileSystemUtil.class)

staticprivate

Definition at line 45 of file FileSystemUtil.java.

The documentation for this class was generated from the following file:

fe/src/main/java/com/cloudera/impala/common/FileSystemUtil.java

Static Public Member Functions

Static Private Member Functions

Static Private Attributes

Detailed Description

Member Function Documentation

Member Data Documentation