org.apache.hadoop.fs

类 ChecksumFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FilterFileSystem

org.apache.hadoop.fs.ChecksumFileSystem

所有已实现的接口:

Closeable, AutoCloseable, Configurable

直接已知子类:

ChecksumDistributedFileSystem, InMemoryFileSystem, LocalFileSystem

public abstract class ChecksumFileSystem
extends FilterFileSystem

Abstract Checksumed FileSystem. It provide a basice implementation of a Checksumed FileSystem, which creates a checksum file for each raw file. It generates & verifies checksums at the client side.

嵌套类概要

从类继承的嵌套类/接口 org.apache.hadoop.fs.FileSystem

FileSystem.Statistics

字段概要

从类继承的字段 org.apache.hadoop.fs.FilterFileSystem

fs

从类继承的字段 org.apache.hadoop.fs.FileSystem

FS_DEFAULT_NAME_KEY, LOG, statistics

构造器概要

构造器

构造器和说明
ChecksumFileSystem(FileSystem fs)

方法概要

方法

限定符和类型	方法和说明
FSDataOutputStream	append(Path f, int bufferSize, Progressable progress) Append to an existing file (optional operation).
void	completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) Called when we're all done writing to the target.
void	copyFromLocalFile(boolean delSrc, Path src, Path dst) The src file is on the local disk.
void	copyToLocalFile(boolean delSrc, Path src, Path dst) The src file is under FS, and the dst is on the local disk.
void	copyToLocalFile(Path src, Path dst, boolean copyCrc) The src file is under FS, and the dst is on the local disk.
FSDataOutputStream	create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
FSDataOutputStream	createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
boolean	delete(Path f, boolean recursive) Implement the delete(Path, boolean) in checksum file system.
static double	getApproxChkSumLength(long size)
int	getBytesPerSum() Return the bytes Per Checksum
Path	getChecksumFile(Path file) Return the name of the checksum file associated with a file.
long	getChecksumFileLength(Path file, long fileSize) Return the length of the checksum file given the size of the actual file.
static long	getChecksumLength(long size, int bytesPerSum) Calculated the length of the checksum file in bytes.
FileSystem	getRawFileSystem() get the raw file system
static boolean	isChecksumFile(Path file) Return true iff file is a checksum file name.
FileStatus[]	listStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.
boolean	mkdirs(Path f) Call FileSystem.mkdirs(Path, FsPermission) with default permission.
FSDataInputStream	open(Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
boolean	rename(Path src, Path dst) Rename files/dirs
boolean	reportChecksumFailure(Path f, FSDataInputStream in, long inPos, FSDataInputStream sums, long sumsPos) Report a checksum error to the file system.
void	setConf(Configuration conf) Set the configuration to be used by this object.
boolean	setReplication(Path src, short replication) Set replication for an existing file.
void	setVerifyChecksum(boolean verifyChecksum) Set whether to verify checksum.
Path	startLocalOutput(Path fsOutputFile, Path tmpLocalFile) Returns a local File that the user can write output to.

从类继承的方法 org.apache.hadoop.fs.FilterFileSystem

checkPath, close, concat, delete, getCanonicalServiceName, getConf, getDefaultBlockSize, getDefaultReplication, getFileBlockLocations, getFileChecksum, getFileStatus, getHomeDirectory, getName, getUri, getWorkingDirectory, initialize, makeQualified, mkdirs, setOwner, setPermission, setWorkingDirectory

从类继承的方法 org.apache.hadoop.fs.FileSystem

addFileSystemForTesting, append, append, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, deleteOnExit, exists, get, get, get, getAllStatistics, getBlockSize, getCacheSize, getCanonicalUri, getContentSummary, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultUri, getDelegationToken, getLength, getLocal, getNamed, getReplication, getStatistics, getStatistics, getUsed, globStatus, globStatus, isDirectory, isFile, listStatus, listStatus, listStatus, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, open, printStatistics, processDeleteOnExit, setDefaultUri, setDefaultUri, setTimes

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

ChecksumFileSystem

public ChecksumFileSystem(FileSystem fs)

方法详细资料

getApproxChkSumLength

public static double getApproxChkSumLength(long size)

setConf

public void setConf(Configuration conf)

从接口复制的说明: Configurable

Set the configuration to be used by this object.

指定者:

setConf 在接口中 Configurable

覆盖:

setConf 在类中 Configured

setVerifyChecksum

public void setVerifyChecksum(boolean verifyChecksum)

Set whether to verify checksum.

覆盖:

setVerifyChecksum 在类中 FilterFileSystem

getRawFileSystem

public FileSystem getRawFileSystem()

get the raw file system

getChecksumFile

public Path getChecksumFile(Path file)

Return the name of the checksum file associated with a file.

isChecksumFile

public static boolean isChecksumFile(Path file)

Return true iff file is a checksum file name.

getChecksumFileLength

public long getChecksumFileLength(Path file,
                         long fileSize)

Return the length of the checksum file given the size of the actual file.

getBytesPerSum

public int getBytesPerSum()

Return the bytes Per Checksum

open

public FSDataInputStream open(Path f,
                     int bufferSize)
                       throws IOException

Opens an FSDataInputStream at the indicated Path.

覆盖:

open 在类中 FilterFileSystem

参数:

f - the file name to open

bufferSize - the size of the buffer to be used.

抛出:

IOException

append

public FSDataOutputStream append(Path f,
                        int bufferSize,
                        Progressable progress)
                          throws IOException

Append to an existing file (optional operation).

覆盖:

append 在类中 FilterFileSystem

参数:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used.

progress - for reporting progress if it is not null.

抛出:

IOException

getChecksumLength

public static long getChecksumLength(long size,
                     int bytesPerSum)

Calculated the length of the checksum file in bytes.

参数:

size - the length of the data file in bytes

bytesPerSum - the number of bytes in a checksum block

返回:

the number of bytes in the checksum file

create

public FSDataOutputStream create(Path f,
                        FsPermission permission,
                        boolean overwrite,
                        int bufferSize,
                        short replication,
                        long blockSize,
                        Progressable progress)
                          throws IOException

Opens an FSDataOutputStream at the indicated Path with write-progress reporting.

覆盖:

create 在类中 FilterFileSystem

参数:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

抛出:

IOException

另请参阅:

FileSystem.setPermission(Path, FsPermission)

createNonRecursive

public FSDataOutputStream createNonRecursive(Path f,
                                    FsPermission permission,
                                    boolean overwrite,
                                    int bufferSize,
                                    short replication,
                                    long blockSize,
                                    Progressable progress)
                                      throws IOException

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

覆盖:

createNonRecursive 在类中 FileSystem

参数:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

抛出:

IOException

另请参阅:

FileSystem.setPermission(Path, FsPermission)

setReplication

public boolean setReplication(Path src,
                     short replication)
                       throws IOException

Set replication for an existing file. Implement the abstract setReplication of FileSystem

覆盖:

setReplication 在类中 FilterFileSystem

参数:

src - file name

replication - new replication

返回:

true if successful; false if file does not exist or is a directory

抛出:

IOException

rename

public boolean rename(Path src,
             Path dst)
               throws IOException

Rename files/dirs

覆盖:

rename 在类中 FilterFileSystem

抛出:

IOException

delete

public boolean delete(Path f,
             boolean recursive)
               throws IOException

Implement the delete(Path, boolean) in checksum file system.

覆盖:

delete 在类中 FilterFileSystem

参数:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

返回:

true if delete is successful else false.

抛出:

IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

List the statuses of the files/directories in the given path if the path is a directory.

覆盖:

listStatus 在类中 FilterFileSystem

参数:

f - given path

返回:

the statuses of the files/directories in the given patch

抛出:

IOException

mkdirs

public boolean mkdirs(Path f)
               throws IOException

从类复制的说明: FileSystem

Call FileSystem.mkdirs(Path, FsPermission) with default permission.

覆盖:

mkdirs 在类中 FileSystem

抛出:

IOException

copyFromLocalFile

public void copyFromLocalFile(boolean delSrc,
                     Path src,
                     Path dst)
                       throws IOException

从类复制的说明: FilterFileSystem

The src file is on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed

覆盖:

copyFromLocalFile 在类中 FilterFileSystem

抛出:

IOException

copyToLocalFile

public void copyToLocalFile(boolean delSrc,
                   Path src,
                   Path dst)
                     throws IOException

The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name.

覆盖:

copyToLocalFile 在类中 FilterFileSystem

抛出:

IOException

copyToLocalFile

public void copyToLocalFile(Path src,
                   Path dst,
                   boolean copyCrc)
                     throws IOException

The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. If src and dst are directories, the copyCrc parameter determines whether to copy CRC files.

抛出:

IOException

startLocalOutput

public Path startLocalOutput(Path fsOutputFile,
                    Path tmpLocalFile)
                      throws IOException

从类复制的说明: FilterFileSystem

Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

覆盖:

startLocalOutput 在类中 FilterFileSystem

抛出:

IOException

completeLocalOutput

public void completeLocalOutput(Path fsOutputFile,
                       Path tmpLocalFile)
                         throws IOException

从类复制的说明: FilterFileSystem

Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.

覆盖:

completeLocalOutput 在类中 FilterFileSystem

抛出:

IOException

reportChecksumFailure

public boolean reportChecksumFailure(Path f,
                            FSDataInputStream in,
                            long inPos,
                            FSDataInputStream sums,
                            long sumsPos)

Report a checksum error to the file system.

参数:

f - the file name containing the error

in - the stream open on the file

inPos - the position of the beginning of the bad data in the file

sums - the stream open on the checksum file

sumsPos - the position of the beginning of the bad data in the checksum file

返回:

if retry is neccessary