org.apache.hadoop.tools.distcp2.util

类 DistCpUtils

java.lang.Object

org.apache.hadoop.tools.distcp2.util.DistCpUtils

public class DistCpUtils
extends Object

Utility functions used in DistCp.

构造器概要

构造器

构造器和说明
DistCpUtils()

方法概要

方法

限定符和类型	方法和说明
static boolean	checksumsAreEqual(FileSystem sourceFS, Path source, FileSystem targetFS, Path target) Utility to compare checksums for the paths specified.
static boolean	compareFs(FileSystem srcFs, FileSystem destFs)
static long	getFileSize(Path path, Configuration configuration) Retrieves size of the file at the specified path.
static DecimalFormat	getFormatter()
static int	getInt(Configuration configuration, String label) Utility to retrieve a specified key from a Configuration.
static long	getLong(Configuration configuration, String label) Utility to retrieve a specified key from a Configuration.
static int	getNumMapTasks(Configuration configuration)
static String	getRelativePath(Path sourceRootPath, Path childPath) Gets relative path of child path with respect to a root path For ex.
static Class<? extends InputFormat>	getStrategy(Configuration conf, DistCpOptions options) Returns the class that implements a copy strategy.
static String	getStringDescriptionFor(long nBytes)
static String	packAttributes(EnumSet<DistCpOptions.FileAttribute> attributes) Pack file preservation attributes into a string, containing just the first character of each preservation attribute
static void	preserve(FileSystem targetFS, Path path, FileStatus srcFileStatus, EnumSet<DistCpOptions.FileAttribute> attributes) Preserve attribute on file matching that of the file status being sent as argument.
static <T> void	publish(Configuration configuration, String label, T value) Utility to publish a value to a configuration.
static Path	sortListing(FileSystem fs, Configuration conf, Path sourceListing) Sort sequence file containing FileStatus and Text as key and value respecitvely
static EnumSet<DistCpOptions.FileAttribute>	unpackAttributes(String attributes) Un packs preservation attribute string containing the first character of each preservation attribute back to a set of attributes to preserve

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

DistCpUtils

public DistCpUtils()

方法详细资料

getFileSize

public static long getFileSize(Path path,
               Configuration configuration)
                        throws IOException

Retrieves size of the file at the specified path.

参数:

path - The path of the file whose size is sought.

configuration - Configuration, to retrieve the appropriate FileSystem.

返回:

The file-size, in number of bytes.

抛出:

IOException, - on failure.

IOException

publish

public static <T> void publish(Configuration configuration,
               String label,
               T value)

Utility to publish a value to a configuration.

类型参数:

T - The type of the value.

参数:

configuration - The Configuration to which the value must be written.

label - The label for the value being published.

value - The value being published.

getInt

public static int getInt(Configuration configuration,
         String label)

Utility to retrieve a specified key from a Configuration. Throw exception if not found.

参数:

configuration - The Configuration in which the key is sought.

label - The key being sought.

返回:

Integer value of the key.

getNumMapTasks

public static int getNumMapTasks(Configuration configuration)

getLong

public static long getLong(Configuration configuration,
           String label)

Utility to retrieve a specified key from a Configuration. Throw exception if not found.

参数:

configuration - The Configuration in which the key is sought.

label - The key being sought.

返回:

Long value of the key.

getStrategy

public static Class<? extends InputFormat> getStrategy(Configuration conf,
                                       DistCpOptions options)

Returns the class that implements a copy strategy. Looks up the implementation for a particular strategy from distcp-default.xml

参数:

conf - - Configuration object

options - - Handle to input options

返回:

Class implementing the strategy specified in options.

getRelativePath

public static String getRelativePath(Path sourceRootPath,
                     Path childPath)

Gets relative path of child path with respect to a root path For ex. If childPath = /tmp/abc/xyz/file and sourceRootPath = /tmp/abc Relative path would be /xyz/file If childPath = /file and sourceRootPath = / Relative path would be /file

参数:

sourceRootPath - - Source root path

childPath - - Path for which relative path is required

返回:

- Relative portion of the child path (always prefixed with / unless it is empty

packAttributes

public static String packAttributes(EnumSet<DistCpOptions.FileAttribute> attributes)

Pack file preservation attributes into a string, containing just the first character of each preservation attribute

参数:

attributes - - Attribute set to preserve

返回:

- String containing first letters of each attribute to preserve

unpackAttributes

public static EnumSet<DistCpOptions.FileAttribute> unpackAttributes(String attributes)

Un packs preservation attribute string containing the first character of each preservation attribute back to a set of attributes to preserve

参数:

attributes - - Attribute string

返回:

- Attribute set

preserve

public static void preserve(FileSystem targetFS,
            Path path,
            FileStatus srcFileStatus,
            EnumSet<DistCpOptions.FileAttribute> attributes)
                     throws IOException

Preserve attribute on file matching that of the file status being sent as argument. Barring the block size, all the other attributes are preserved by this function

参数:

targetFS - - File system

path - - Path that needs to preserve original file status

srcFileStatus - - Original file status

attributes - - Attribute set that need to be preserved

抛出:

IOException - - Exception if any (particularly relating to group/owner change or any transient error)

sortListing

public static Path sortListing(FileSystem fs,
               Configuration conf,
               Path sourceListing)
                        throws IOException

Sort sequence file containing FileStatus and Text as key and value respecitvely

参数:

fs - - File System

conf - - Configuration

sourceListing - - Source listing file

返回:

Path of the sorted file. Is source file with _sorted appended to the name

抛出:

IOException - - Any exception during sort.

getFormatter

public static DecimalFormat getFormatter()

getStringDescriptionFor

public static String getStringDescriptionFor(long nBytes)

checksumsAreEqual

public static boolean checksumsAreEqual(FileSystem sourceFS,
                        Path source,
                        FileSystem targetFS,
                        Path target)
                                 throws IOException

Utility to compare checksums for the paths specified. If checksums's can't be retrieved, it doesn't fail the test Only time the comparison would fail is when checksums are available and they don't match

参数:

sourceFS - FileSystem for the source path.

source - The source path.

targetFS - FileSystem for the target path.

target - The target path.

返回:

If either checksum couldn't be retrieved, the function returns false. If checksums are retrieved, the function returns true if they match, and false otherwise.

抛出:

IOException - if there's an exception while retrieving checksums.

compareFs

public static boolean compareFs(FileSystem srcFs,
                FileSystem destFs)