org.apache.hadoop.tools.distcp2

类 SimpleCopyListing

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.tools.distcp2.CopyListing

org.apache.hadoop.tools.distcp2.SimpleCopyListing

所有已实现的接口:

Configurable

public class SimpleCopyListing
extends CopyListing

The SimpleCopyListing is responsible for making the exhaustive list of all files/directories under its specified list of input-paths. These are written into the specified copy-listing file. Note: The SimpleCopyListing doesn't handle wild-cards in the input-paths.

构造器概要

构造器

限定符	构造器和说明
protected	SimpleCopyListing(Configuration configuration, Credentials credentials) Protected constructor, to initialize configuration.

方法概要

方法

限定符和类型	方法和说明
void	doBuildListing(Path pathToListingFile, DistCpOptions options) The interface to be implemented by sub-classes, to create the source/target file listing.
protected long	getBytesToCopy() Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy
protected long	getNumberOfPaths() Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy
protected boolean	shouldCopy(Path path, DistCpOptions options) Provide an option to skip copy of a path, Allows for exclusion of files such as FileOutputCommitter.SUCCEEDED_FILE_NAME
protected void	validatePaths(DistCpOptions options) Validate input and output paths

从类继承的方法 org.apache.hadoop.tools.distcp2.CopyListing

buildListing, getCopyListing, getCredentials, setCredentials

从类继承的方法 org.apache.hadoop.conf.Configured

getConf, setConf

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

SimpleCopyListing

protected SimpleCopyListing(Configuration configuration,
                 Credentials credentials)

Protected constructor, to initialize configuration.

参数:

configuration - The input configuration, with which the source/target FileSystems may be accessed.

credentials - - Credentials object on which the FS delegation tokens are cached. If null delegation token caching is skipped

方法详细资料

validatePaths

protected void validatePaths(DistCpOptions options)
                      throws IOException,
                             org.apache.hadoop.tools.distcp2.CopyListing.InvalidInputException

从类复制的说明: CopyListing

Validate input and output paths

指定者:

validatePaths 在类中 CopyListing

参数:

options - - Input options

抛出:

IOException

org.apache.hadoop.tools.distcp2.CopyListing.InvalidInputException

doBuildListing

public void doBuildListing(Path pathToListingFile,
                  DistCpOptions options)
                    throws IOException

The interface to be implemented by sub-classes, to create the source/target file listing.

指定者:

doBuildListing 在类中 CopyListing

参数:

pathToListingFile - Path on HDFS where the listing file is written.

options - Input Options for DistCp (indicating source/target paths.)

抛出:

IOException

shouldCopy

protected boolean shouldCopy(Path path,
                 DistCpOptions options)

Provide an option to skip copy of a path, Allows for exclusion of files such as FileOutputCommitter.SUCCEEDED_FILE_NAME

参数:

path - - Path being considered for copy while building the file listing

options - - Input options passed during DistCp invocation

返回:

- True if the path should be considered for copy, false otherwise

getBytesToCopy

protected long getBytesToCopy()

Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy

指定者:

getBytesToCopy 在类中 CopyListing

返回:

total bytes to copy

getNumberOfPaths

protected long getNumberOfPaths()

Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy

指定者:

getNumberOfPaths 在类中 CopyListing

返回:

Total number of paths to distcp