org.apache.hadoop.tools.distcp2

类 CopyListing

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.tools.distcp2.CopyListing

所有已实现的接口:

Configurable

直接已知子类:

FileBasedCopyListing, GlobbedCopyListing, SimpleCopyListing

public abstract class CopyListing
extends Configured

The CopyListing abstraction is responsible for how the list of sources and targets is constructed, for DistCp's copy function. The copy-listing should be a SequenceFile, located at the path specified to buildListing(), each entry being a pair of (Source relative path, source file status), all the paths being fully qualified.

构造器概要

构造器

限定符	构造器和说明
protected	CopyListing(Configuration configuration, Credentials credentials) Protected constructor, to initialize configuration.

方法概要

方法

限定符和类型	方法和说明
void	buildListing(Path pathToListFile, DistCpOptions options) Build listing function creates the input listing that distcp uses to perform the copy.
protected abstract void	doBuildListing(Path pathToListFile, DistCpOptions options) The interface to be implemented by sub-classes, to create the source/target file listing.
protected abstract long	getBytesToCopy() Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy
static CopyListing	getCopyListing(Configuration configuration, Credentials credentials, DistCpOptions options) Public Factory method with which the appropriate CopyListing implementation may be retrieved.
protected Credentials	getCredentials() get credentials to update the delegation tokens for accessed FS objects
protected abstract long	getNumberOfPaths() Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy
protected void	setCredentials(Credentials credentials) set Credentials store, on which FS delegatin token will be cached
protected abstract void	validatePaths(DistCpOptions options) Validate input and output paths

从类继承的方法 org.apache.hadoop.conf.Configured

getConf, setConf

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

CopyListing

protected CopyListing(Configuration configuration,
           Credentials credentials)

Protected constructor, to initialize configuration.

参数:

configuration - The input configuration, with which the source/target FileSystems may be accessed.

credentials - - Credentials object on which the FS delegation tokens are cached.If null delegation token caching is skipped

方法详细资料

buildListing

public final void buildListing(Path pathToListFile,
                DistCpOptions options)
                        throws IOException

Build listing function creates the input listing that distcp uses to perform the copy. The build listing is a sequence file that has relative path of a file in the key and the file status information of the source file in the value For instance if the source path is /tmp/data and the traversed path is /tmp/data/dir1/dir2/file1, then the sequence file would contain key: /dir1/dir2/file1 and value: FileStatus(/tmp/data/dir1/dir2/file1) File would also contain directory entries. Meaning, if /tmp/data/dir1/dir2/file1 is the only file under /tmp/data, the resulting sequence file would contain the following entries key: /dir1 and value: FileStatus(/tmp/data/dir1) key: /dir1/dir2 and value: FileStatus(/tmp/data/dir1/dir2) key: /dir1/dir2/file1 and value: FileStatus(/tmp/data/dir1/dir2/file1) Cases requiring special handling: If source path is a file (/tmp/file1), contents of the file will be as follows TARGET DOES NOT EXIST: Key-"", Value-FileStatus(/tmp/file1) TARGET IS FILE : Key-"", Value-FileStatus(/tmp/file1) TARGET IS DIR : Key-"/file1", Value-FileStatus(/tmp/file1)

参数:

pathToListFile - - Output file where the listing would be stored

options - - Input options to distcp

抛出:

IOException - - Exception if any

validatePaths

protected abstract void validatePaths(DistCpOptions options)
                               throws IOException,
                                      org.apache.hadoop.tools.distcp2.CopyListing.InvalidInputException

Validate input and output paths

参数:

options - - Input options

抛出:

InvalidInputException: - If inputs are invalid

IOException: - any Exception with FS

IOException

org.apache.hadoop.tools.distcp2.CopyListing.InvalidInputException

doBuildListing

protected abstract void doBuildListing(Path pathToListFile,
                  DistCpOptions options)
                                throws IOException

The interface to be implemented by sub-classes, to create the source/target file listing.

参数:

pathToListFile - Path on HDFS where the listing file is written.

options - Input Options for DistCp (indicating source/target paths.)

抛出:

IOException: - Thrown on failure to create the listing file.

IOException

getBytesToCopy

protected abstract long getBytesToCopy()

Return the total bytes that distCp should copy for the source paths This doesn't consider whether file is same should be skipped during copy

返回:

total bytes to copy

getNumberOfPaths

protected abstract long getNumberOfPaths()

Return the total number of paths to distcp, includes directories as well This doesn't consider whether file/dir is already present and should be skipped during copy

返回:

Total number of paths to distcp

setCredentials

protected void setCredentials(Credentials credentials)

set Credentials store, on which FS delegatin token will be cached

参数:

credentials - - Credentials object

getCredentials

protected Credentials getCredentials()

get credentials to update the delegation tokens for accessed FS objects

返回:

Credentials object

getCopyListing

public static CopyListing getCopyListing(Configuration configuration,
                         Credentials credentials,
                         DistCpOptions options)
                                  throws IOException

Public Factory method with which the appropriate CopyListing implementation may be retrieved.

参数:

configuration - The input configuration.

credentials - Credentials object on which the FS delegation tokens are cached

options - The input Options, to help choose the appropriate CopyListing Implementation.

返回:

An instance of the appropriate CopyListing implementation.

抛出:

IOException - - Exception if any