org.apache.hadoop.hdfs.server.datanode

类 FSDataset

java.lang.Object

org.apache.hadoop.hdfs.server.datanode.FSDataset

所有已实现的接口:

FSConstants, FSDatasetInterface, FSDatasetMBean

public class FSDataset
extends Object
implements FSConstants, FSDatasetInterface

FSDataset manages a set of data blocks. Each block has a unique name and an extent on disk.

嵌套类概要

从接口继承的嵌套类/接口 org.apache.hadoop.hdfs.protocol.FSConstants

FSConstants.DatanodeReportType, FSConstants.SafeModeAction, FSConstants.UpgradeAction

从接口继承的嵌套类/接口 org.apache.hadoop.hdfs.server.datanode.FSDatasetInterface

FSDatasetInterface.BlockInputStreams, FSDatasetInterface.BlockWriteStreams, FSDatasetInterface.MetaDataInputStream

字段概要

字段

限定符和类型	字段和说明
static String	METADATA_EXTENSION
static int	METADATA_EXTENSION_LENGTH
static short	METADATA_VERSION

从接口继承的字段 org.apache.hadoop.hdfs.protocol.FSConstants

BLOCKREPORT_INITIAL_DELAY, BLOCKREPORT_INTERVAL, BUFFER_SIZE, DEFAULT_BLOCK_SIZE, DEFAULT_DATA_SOCKET_SIZE, HEARTBEAT_INTERVAL, LAYOUT_VERSION, LEASE_HARDLIMIT_PERIOD, LEASE_RECOVER_PERIOD, LEASE_SOFTLIMIT_PERIOD, MAX_PATH_DEPTH, MAX_PATH_LENGTH, MIN_BLOCKS_FOR_WRITE, QUOTA_DONT_SET, QUOTA_RESET, SIZE_OF_INTEGER, SMALL_BUFFER_SIZE

构造器概要

构造器

构造器和说明
FSDataset(DataStorage storage, Configuration conf) An FSDataset has a directory where it loads its data files.

方法概要

方法

限定符和类型	方法和说明
void	checkDataDir() check if a data directory is healthy if some volumes failed - make sure to remove all the blocks that belong to these volumes
boolean	detachBlock(Block block, int numLinks) Make a copy of the block if this block is linked to an existing snapshot.
void	finalizeBlock(Block b) Finalizes the block previously opened for writing using writeToBlock.
void	finalizeBlockIfNeeded(Block b) Finalizes the block previously opened for writing using writeToBlock if not already finalized
File	findBlockFile(long blockId) Return the block file for the given ID
static File	findMetaFile(File blockFile) Find the corresponding meta data file from a given block file
File	getBlockFile(Block b) Get File name for a given block.
InputStream	getBlockInputStream(Block b) Returns an input stream to read the contents of the specified block
InputStream	getBlockInputStream(Block b, long seekOffset) Returns an input stream at specified offset of the specified block
BlockLocalPathInfo	getBlockLocalPathInfo(Block block) Get BlockLocalPathInfo for the given block.
Block[]	getBlockReport() Return a table of block data.
Block[]	getBlocksBeingWrittenReport() Return a table of blocks being written data
long	getCapacity() Return total capacity, used and unused
long	getChannelPosition(Block b, FSDatasetInterface.BlockWriteStreams streams) Retrieves the offset in the block to which the the next write will write data to.
long	getDfsUsed() Return the total space used by dfs datanode
File	getFile(Block b) Turn the block identifier into a filename.
long	getLength(Block b) Find the block's on-disk length
FSDatasetInterface.MetaDataInputStream	getMetaDataInputStream(Block b) Returns metaData of block b as an input stream (and its length)
long	getMetaDataLength(Block b) Returns the length of the metadata file of the specified block
protected File	getMetaFile(Block b)
long	getRemaining() Return how many bytes can still be stored in the FSDataset
String	getStorageInfo() Returns the storage id of the underlying storage
Block	getStoredBlock(long blkid)
FSDatasetInterface.BlockInputStreams	getTmpInputStreams(Block b, long blkOffset, long ckoff) Returns handles to the block file and its metadata file
long	getVisibleLength(Block b) Returns the specified block's visible length (has metadata for this)
boolean	hasEnoughResource() Return true - if there are still valid volumes on the DataNode.
void	invalidate(Block[] invalidBlks) We're informed that a block is no longer valid.
boolean	isAsyncBlockReportReady()
boolean	isValidBlock(Block b) Check whether the given block is a valid one.
boolean	metaFileExists(Block b) Does the meta file exist for this block?
void	requestAsyncBlockReport() Request that a block report be prepared.
Block[]	retrieveAsyncBlockReport() Retrieve an asynchronously prepared block report.
void	setChannelPosition(Block b, FSDatasetInterface.BlockWriteStreams streams, long dataOffset, long ckOffset) Sets the offset in the block to which the the next write will write data to.
void	setVisibleLength(Block b, long length) update the specified blocks visible meta data.
void	shutdown() Shutdown the FSDataset
BlockRecoveryInfo	startBlockRecovery(long blockId)
String	toString() Stringifies the name of the storage
void	unfinalizeBlock(Block b) Remove the temporary block file (if any)
void	updateBlock(Block oldblock, Block newblock) Update the block to the new generation stamp and length.
void	validateBlockMetadata(Block b) Validate that the contents in the Block matches the file on disk.
FSDatasetInterface.BlockWriteStreams	writeToBlock(Block b, boolean isRecovery, boolean replicationRequest) Start writing to a block file If isRecovery is true and the block pre-exists, then we kill all volumeMap.put(b, v); volumeMap.put(b, v); other threads that might be writing to this block, and then reopen the file.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

字段详细资料

METADATA_EXTENSION

public static final String METADATA_EXTENSION

另请参阅:

常量字段值

METADATA_EXTENSION_LENGTH

public static final int METADATA_EXTENSION_LENGTH

METADATA_VERSION

public static final short METADATA_VERSION

另请参阅:

常量字段值

构造器详细资料

FSDataset

public FSDataset(DataStorage storage,
         Configuration conf)
          throws IOException

An FSDataset has a directory where it loads its data files.

抛出:

IOException

方法详细资料

getMetaFile

protected File getMetaFile(Block b)
                    throws IOException

抛出:

IOException

findMetaFile

public static File findMetaFile(File blockFile)
                         throws IOException

Find the corresponding meta data file from a given block file

抛出:

IOException

findBlockFile

public File findBlockFile(long blockId)

Return the block file for the given ID

getStoredBlock

public Block getStoredBlock(long blkid)
                     throws IOException

指定者:

getStoredBlock 在接口中 FSDatasetInterface

返回:

the generation stamp stored with the block.

抛出:

IOException

metaFileExists

public boolean metaFileExists(Block b)
                       throws IOException

从接口复制的说明: FSDatasetInterface

Does the meta file exist for this block?

指定者:

metaFileExists 在接口中 FSDatasetInterface

参数:

b - - the block

返回:

true of the metafile for specified block exits

抛出:

IOException

getMetaDataLength

public long getMetaDataLength(Block b)
                       throws IOException

从接口复制的说明: FSDatasetInterface

Returns the length of the metadata file of the specified block

指定者:

getMetaDataLength 在接口中 FSDatasetInterface

参数:

b - - the block for which the metadata length is desired

返回:

the length of the metadata file for the specified block.

抛出:

IOException

getMetaDataInputStream

public FSDatasetInterface.MetaDataInputStream getMetaDataInputStream(Block b)
                                                              throws IOException

从接口复制的说明: FSDatasetInterface

Returns metaData of block b as an input stream (and its length)

指定者:

getMetaDataInputStream 在接口中 FSDatasetInterface

参数:

b - - the block

返回:

the metadata input stream;

抛出:

IOException

getDfsUsed

public long getDfsUsed()
                throws IOException

Return the total space used by dfs datanode

指定者:

getDfsUsed 在接口中 FSDatasetMBean

返回:

the total space used by dfs datanode

抛出:

IOException

hasEnoughResource

public boolean hasEnoughResource()

Return true - if there are still valid volumes on the DataNode.

指定者:

hasEnoughResource 在接口中 FSDatasetInterface

返回:

true if more then minimum valid volumes left in the FSDataSet

getCapacity

public long getCapacity()
                 throws IOException

Return total capacity, used and unused

指定者:

getCapacity 在接口中 FSDatasetMBean

返回:

total capacity of storage (used and unused)

抛出:

IOException

getRemaining

public long getRemaining()
                  throws IOException

Return how many bytes can still be stored in the FSDataset

指定者:

getRemaining 在接口中 FSDatasetMBean

返回:

The amount of free storage space

抛出:

IOException

getLength

public long getLength(Block b)
               throws IOException

Find the block's on-disk length

指定者:

getLength 在接口中 FSDatasetInterface

返回:

the specified block's on-disk length (excluding metadta)

抛出:

IOException

getVisibleLength

public long getVisibleLength(Block b)
                      throws IOException

从接口复制的说明: FSDatasetInterface

Returns the specified block's visible length (has metadata for this)

指定者:

getVisibleLength 在接口中 FSDatasetInterface

返回:

the specified block's visible length

抛出:

IOException

setVisibleLength

public void setVisibleLength(Block b,
                    long length)
                      throws IOException

从接口复制的说明: FSDatasetInterface

update the specified blocks visible meta data. NOTE: only applies to blocks that are being written to. If called on closed blocks, throws IOException

指定者:

setVisibleLength 在接口中 FSDatasetInterface

参数:

b - block to update the length for

length - value to set visible length to

抛出:

IOException - if the block is not in ongoingCreates

getBlockFile

public File getBlockFile(Block b)
                  throws IOException

Get File name for a given block.

抛出:

IOException

getBlockLocalPathInfo

public BlockLocalPathInfo getBlockLocalPathInfo(Block block)
                                         throws IOException

从接口复制的说明: FSDatasetInterface

Get BlockLocalPathInfo for the given block.

指定者:

getBlockLocalPathInfo 在接口中 FSDatasetInterface

抛出:

IOException

getBlockInputStream

public InputStream getBlockInputStream(Block b)
                                throws IOException

从接口复制的说明: FSDatasetInterface

Returns an input stream to read the contents of the specified block

指定者:

getBlockInputStream 在接口中 FSDatasetInterface

返回:

an input stream to read the contents of the specified block

抛出:

IOException

getBlockInputStream

public InputStream getBlockInputStream(Block b,
                              long seekOffset)
                                throws IOException

从接口复制的说明: FSDatasetInterface

Returns an input stream at specified offset of the specified block

指定者:

getBlockInputStream 在接口中 FSDatasetInterface

返回:

an input stream to read the contents of the specified block, starting at the offset

抛出:

IOException

getTmpInputStreams

public FSDatasetInterface.BlockInputStreams getTmpInputStreams(Block b,
                                                      long blkOffset,
                                                      long ckoff)
                                                        throws IOException

Returns handles to the block file and its metadata file

指定者:

getTmpInputStreams 在接口中 FSDatasetInterface

返回:

an input stream to read the contents of the specified block, starting at the offset

抛出:

IOException

detachBlock

public boolean detachBlock(Block block,
                  int numLinks)
                    throws IOException

Make a copy of the block if this block is linked to an existing snapshot. This ensures that modifying this block does not modify data in any existing snapshots.

参数:

block - Block

numLinks - Detach if the number of links exceed this value

返回:

- true if the specified block was detached

抛出:

IOException

updateBlock

public void updateBlock(Block oldblock,
               Block newblock)
                 throws IOException

Update the block to the new generation stamp and length.

指定者:

updateBlock 在接口中 FSDatasetInterface

抛出:

IOException

writeToBlock

public FSDatasetInterface.BlockWriteStreams writeToBlock(Block b,
                                                boolean isRecovery,
                                                boolean replicationRequest)
                                                  throws IOException

Start writing to a block file If isRecovery is true and the block pre-exists, then we kill all volumeMap.put(b, v); volumeMap.put(b, v); other threads that might be writing to this block, and then reopen the file. If replicationRequest is true, then this operation is part of a block replication request.

指定者:

writeToBlock 在接口中 FSDatasetInterface

isRecovery - True if this is part of error recovery, otherwise false

replicationRequest - True if this is part of block replication request

返回:

a BlockWriteStreams object to allow writing the block data and CRC

抛出:

IOException

getChannelPosition

public long getChannelPosition(Block b,
                      FSDatasetInterface.BlockWriteStreams streams)
                        throws IOException

Retrieves the offset in the block to which the the next write will write data to.

指定者:

getChannelPosition 在接口中 FSDatasetInterface

streams - The stream to the data file and checksum file

返回:

the position of the file pointer in the data stream

抛出:

IOException

setChannelPosition

public void setChannelPosition(Block b,
                      FSDatasetInterface.BlockWriteStreams streams,
                      long dataOffset,
                      long ckOffset)
                        throws IOException

Sets the offset in the block to which the the next write will write data to.

指定者:

setChannelPosition 在接口中 FSDatasetInterface

streams - The stream for the data file and checksum file

dataOffset - The position to which the file pointre for the data stream should be set

ckOffset - The position to which the file pointre for the checksum stream should be set

抛出:

IOException

finalizeBlock

public void finalizeBlock(Block b)
                   throws IOException

从接口复制的说明: FSDatasetInterface

Finalizes the block previously opened for writing using writeToBlock. The block size is what is in the parameter b and it must match the amount of data written

指定者:

finalizeBlock 在接口中 FSDatasetInterface

抛出:

IOException

finalizeBlockIfNeeded

public void finalizeBlockIfNeeded(Block b)
                           throws IOException

从接口复制的说明: FSDatasetInterface

Finalizes the block previously opened for writing using writeToBlock if not already finalized

指定者:

finalizeBlockIfNeeded 在接口中 FSDatasetInterface

抛出:

IOException

unfinalizeBlock

public void unfinalizeBlock(Block b)
                     throws IOException

Remove the temporary block file (if any)

指定者:

unfinalizeBlock 在接口中 FSDatasetInterface

抛出:

IOException

getBlocksBeingWrittenReport

public Block[] getBlocksBeingWrittenReport()

Return a table of blocks being written data

指定者:

getBlocksBeingWrittenReport 在接口中 FSDatasetInterface

返回:

- the blocks being written report

requestAsyncBlockReport

public void requestAsyncBlockReport()

从接口复制的说明: FSDatasetInterface

Request that a block report be prepared.

指定者:

requestAsyncBlockReport 在接口中 FSDatasetInterface

isAsyncBlockReportReady

public boolean isAsyncBlockReportReady()

指定者:

isAsyncBlockReportReady 在接口中 FSDatasetInterface

返回:

true if an asynchronous block report is ready

retrieveAsyncBlockReport

public Block[] retrieveAsyncBlockReport()

从接口复制的说明: FSDatasetInterface

Retrieve an asynchronously prepared block report. Callers should first call FSDatasetInterface.requestAsyncBlockReport(), and then poll FSDatasetInterface.isAsyncBlockReportReady() until it returns true. Retrieving the asynchronous block report also resets it; a new one must be prepared before this method may be called again.

指定者:

retrieveAsyncBlockReport 在接口中 FSDatasetInterface

getBlockReport

public Block[] getBlockReport()

Return a table of block data. This method is synchronous, and is used by tests and during block scanner startup.

指定者:

getBlockReport 在接口中 FSDatasetInterface

返回:

- the block report - the full list of blocks stored

isValidBlock

public boolean isValidBlock(Block b)

Check whether the given block is a valid one.

指定者:

isValidBlock 在接口中 FSDatasetInterface

返回:

- true if the specified block is valid

validateBlockMetadata

public void validateBlockMetadata(Block b)
                           throws IOException

Validate that the contents in the Block matches the file on disk. Returns true if everything is fine.

指定者:

validateBlockMetadata 在接口中 FSDatasetInterface

参数:

b - The block to be verified.

抛出:

IOException

invalidate

public void invalidate(Block[] invalidBlks)
                throws IOException

We're informed that a block is no longer valid. We could lazily garbage-collect the block, but why bother? just get rid of it.

指定者:

invalidate 在接口中 FSDatasetInterface

参数:

invalidBlks - - the blocks to be invalidated

抛出:

IOException

getFile

public File getFile(Block b)

Turn the block identifier into a filename.

checkDataDir

public void checkDataDir()
                  throws DiskChecker.DiskErrorException

check if a data directory is healthy if some volumes failed - make sure to remove all the blocks that belong to these volumes

指定者:

checkDataDir 在接口中 FSDatasetInterface

抛出:

DiskChecker.DiskErrorException

toString

public String toString()

从接口复制的说明: FSDatasetInterface

Stringifies the name of the storage

指定者:

toString 在接口中 FSDatasetInterface

覆盖:

toString 在类中 Object

shutdown

public void shutdown()

从接口复制的说明: FSDatasetInterface

Shutdown the FSDataset

指定者:

shutdown 在接口中 FSDatasetInterface

getStorageInfo

public String getStorageInfo()

从接口复制的说明: FSDatasetMBean

Returns the storage id of the underlying storage

指定者:

getStorageInfo 在接口中 FSDatasetMBean

startBlockRecovery

public BlockRecoveryInfo startBlockRecovery(long blockId)
                                     throws IOException

指定者:

startBlockRecovery 在接口中 FSDatasetInterface

抛出:

IOException