org.apache.hadoop.hdfs.server.namenode

类 BlockPlacementPolicyDefault

java.lang.Object

org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy

org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyDefault

直接已知子类:

BlockPlacementPolicyWithNodeGroup

@InterfaceAudience.Private
public class BlockPlacementPolicyDefault
extends BlockPlacementPolicy

The class is responsible for choosing the desired number of targets for placing block replicas. The replica placement strategy is that if the writer is on a datanode, the 1st replica is placed on the local machine, otherwise a random datanode. The 2nd replica is placed on a datanode that is on a different rack. The 3rd replica is placed on a datanode which is on a different node of the rack as the second replica.

嵌套类概要

从类继承的嵌套类/接口 org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy

BlockPlacementPolicy.NotEnoughReplicasException

字段概要

字段

限定符和类型	字段和说明
protected NetworkTopology	clusterMap
protected boolean	considerLoad

方法概要

方法

限定符和类型	方法和说明
protected int	addToExcludedNodes(DatanodeDescriptor localMachine, HashMap<Node,Node> excludedNodes) Add localMachine and related nodes to excludedNodes for next replica choosing.
protected void	adjustExcludedNodes(HashMap<Node,Node> excludedNodes, Node chosenNode) After choosing a node to place replica, adjust excluded nodes accordingly.
protected DatanodeDescriptor	chooseLocalNode(DatanodeDescriptor localMachine, HashMap<Node,Node> excludedNodes, long blocksize, int maxNodesPerRack, List<DatanodeDescriptor> results, boolean avoidStaleNodes)
protected DatanodeDescriptor	chooseLocalRack(DatanodeDescriptor localMachine, HashMap<Node,Node> excludedNodes, long blocksize, int maxNodesPerRack, List<DatanodeDescriptor> results, boolean avoidStaleNodes)
protected void	chooseRandom(int numOfReplicas, String nodes, HashMap<Node,Node> excludedNodes, long blocksize, int maxNodesPerRack, List<DatanodeDescriptor> results, boolean avoidStaleNodes)
protected DatanodeDescriptor	chooseRandom(String nodes, HashMap<Node,Node> excludedNodes, long blocksize, int maxNodesPerRack, List<DatanodeDescriptor> results, boolean avoidStaleNodes)
protected void	chooseRemoteRack(int numOfReplicas, DatanodeDescriptor localMachine, HashMap<Node,Node> excludedNodes, long blocksize, int maxReplicasPerRack, List<DatanodeDescriptor> results, boolean avoidStaleNodes)
DatanodeDescriptor	chooseReplicaToDelete(FSInodeInfo inode, Block block, short replicationFactor, Collection<DatanodeDescriptor> first, Collection<DatanodeDescriptor> second) Decide whether deleting the specified replica of the block still makes the block conform to the configured block placement policy.
DatanodeDescriptor[]	chooseTarget(FSInodeInfo srcInode, int numOfReplicas, DatanodeDescriptor writer, List<DatanodeDescriptor> chosenNodes, long blocksize) choose numOfReplicas data nodes for writer If not, return as many as we can.
DatanodeDescriptor[]	chooseTarget(String srcPath, int numOfReplicas, DatanodeDescriptor writer, List<DatanodeDescriptor> chosenNodes, HashMap<Node,Node> excludedNodes, long blocksize) choose numOfReplicas data nodes for writer to re-replicate a block with size blocksize If not, return as many as we can.
DatanodeDescriptor[]	chooseTarget(String srcPath, int numOfReplicas, DatanodeDescriptor writer, List<DatanodeDescriptor> chosenNodes, long blocksize) choose numOfReplicas data nodes for writer to re-replicate a block with size blocksize If not, return as many as we can.
void	initialize(Configuration conf, FSClusterStats stats, NetworkTopology clusterMap) Used to setup a BlockPlacementPolicy object.
protected boolean	isGoodTarget(DatanodeDescriptor node, long blockSize, int maxTargetPerLoc, boolean considerLoad, List<DatanodeDescriptor> results, boolean avoidStaleNodes) Determine if a node is a good target.
protected Iterator<DatanodeDescriptor>	pickupReplicaSet(Collection<DatanodeDescriptor> first, Collection<DatanodeDescriptor> second) Pick up replica node set for deleting replica as over-replicated.
int	verifyBlockPlacement(String srcPath, LocatedBlock lBlk, int minRacks) Verify that the block is replicated on at least minRacks different racks if there is more than minRacks rack in the system.

从类继承的方法 org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy

adjustSetsWithChosenReplica, chooseTarget, getInstance, getRack, splitNodesWithRack

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

considerLoad

protected boolean considerLoad

clusterMap

protected NetworkTopology clusterMap

方法详细资料

initialize

public void initialize(Configuration conf,
              FSClusterStats stats,
              NetworkTopology clusterMap)

Used to setup a BlockPlacementPolicy object. This should be defined by all implementations of a BlockPlacementPolicy.

指定者:

initialize 在类中 BlockPlacementPolicy

参数:

conf - the configuration object

stats - retrieve cluster status from here

clusterMap - cluster topology

chooseTarget

public DatanodeDescriptor[] chooseTarget(String srcPath,
                                int numOfReplicas,
                                DatanodeDescriptor writer,
                                List<DatanodeDescriptor> chosenNodes,
                                long blocksize)

choose numOfReplicas data nodes for writer to re-replicate a block with size blocksize If not, return as many as we can.

参数:

srcPath - the file to which this chooseTargets is being invoked.

numOfReplicas - additional number of replicas wanted.

writer - the writer's machine, null if not in the cluster.

chosenNodes - datanodes that have been chosen as targets.

blocksize - size of the data to be written.

返回:

array of DatanodeDescriptor instances chosen as target and sorted as a pipeline.

chooseTarget

public DatanodeDescriptor[] chooseTarget(String srcPath,
                                int numOfReplicas,
                                DatanodeDescriptor writer,
                                List<DatanodeDescriptor> chosenNodes,
                                HashMap<Node,Node> excludedNodes,
                                long blocksize)

choose numOfReplicas data nodes for writer to re-replicate a block with size blocksize If not, return as many as we can.

指定者:

chooseTarget 在类中 BlockPlacementPolicy

参数:

srcPath - the file to which this chooseTargets is being invoked.

numOfReplicas - additional number of replicas wanted.

writer - the writer's machine, null if not in the cluster.

chosenNodes - datanodes that have been chosen as targets.

blocksize - size of the data to be written.

返回:

array of DatanodeDescriptor instances chosen as target and sorted as a pipeline.

chooseTarget

public DatanodeDescriptor[] chooseTarget(FSInodeInfo srcInode,
                                int numOfReplicas,
                                DatanodeDescriptor writer,
                                List<DatanodeDescriptor> chosenNodes,
                                long blocksize)

choose numOfReplicas data nodes for writer If not, return as many as we can. The base implemenatation extracts the pathname of the file from the specified srcInode, but this could be a costly operation depending on the file system implementation. Concrete implementations of this class should override this method to avoid this overhead.

参数:

srcInode - The inode of the file for which chooseTarget is being invoked.

numOfReplicas - additional number of replicas wanted.

writer - the writer's machine, null if not in the cluster.

chosenNodes - datanodes that have been chosen as targets.

blocksize - size of the data to be written.

返回:

array of DatanodeDescriptor instances chosen as target and sorted as a pipeline.

chooseLocalNode

protected DatanodeDescriptor chooseLocalNode(DatanodeDescriptor localMachine,
                                 HashMap<Node,Node> excludedNodes,
                                 long blocksize,
                                 int maxNodesPerRack,
                                 List<DatanodeDescriptor> results,
                                 boolean avoidStaleNodes)
                                      throws BlockPlacementPolicy.NotEnoughReplicasException

抛出:

BlockPlacementPolicy.NotEnoughReplicasException

addToExcludedNodes

protected int addToExcludedNodes(DatanodeDescriptor localMachine,
                     HashMap<Node,Node> excludedNodes)

Add localMachine and related nodes to excludedNodes for next replica choosing. In sub class, we can add more nodes within the same failure domain of localMachine

返回:

number of new excluded nodes

chooseLocalRack

protected DatanodeDescriptor chooseLocalRack(DatanodeDescriptor localMachine,
                                 HashMap<Node,Node> excludedNodes,
                                 long blocksize,
                                 int maxNodesPerRack,
                                 List<DatanodeDescriptor> results,
                                 boolean avoidStaleNodes)
                                      throws BlockPlacementPolicy.NotEnoughReplicasException

抛出:

BlockPlacementPolicy.NotEnoughReplicasException

chooseRemoteRack

protected void chooseRemoteRack(int numOfReplicas,
                    DatanodeDescriptor localMachine,
                    HashMap<Node,Node> excludedNodes,
                    long blocksize,
                    int maxReplicasPerRack,
                    List<DatanodeDescriptor> results,
                    boolean avoidStaleNodes)
                         throws BlockPlacementPolicy.NotEnoughReplicasException

抛出:

BlockPlacementPolicy.NotEnoughReplicasException

chooseRandom

protected DatanodeDescriptor chooseRandom(String nodes,
                              HashMap<Node,Node> excludedNodes,
                              long blocksize,
                              int maxNodesPerRack,
                              List<DatanodeDescriptor> results,
                              boolean avoidStaleNodes)
                                   throws BlockPlacementPolicy.NotEnoughReplicasException

抛出:

BlockPlacementPolicy.NotEnoughReplicasException

chooseRandom

protected void chooseRandom(int numOfReplicas,
                String nodes,
                HashMap<Node,Node> excludedNodes,
                long blocksize,
                int maxNodesPerRack,
                List<DatanodeDescriptor> results,
                boolean avoidStaleNodes)
                     throws BlockPlacementPolicy.NotEnoughReplicasException

抛出:

BlockPlacementPolicy.NotEnoughReplicasException

adjustExcludedNodes

protected void adjustExcludedNodes(HashMap<Node,Node> excludedNodes,
                       Node chosenNode)

After choosing a node to place replica, adjust excluded nodes accordingly. It should do nothing here as chosenNode is already put into exlcudeNodes, but it can be overridden in subclass to put more related nodes into excludedNodes.

参数:

excludedNodes -

chosenNode -

isGoodTarget

protected boolean isGoodTarget(DatanodeDescriptor node,
                   long blockSize,
                   int maxTargetPerLoc,
                   boolean considerLoad,
                   List<DatanodeDescriptor> results,
                   boolean avoidStaleNodes)

Determine if a node is a good target.

参数:

node - The target node

blockSize - Size of block

maxTargetPerLoc - Maximum number of targets per rack. The value of this parameter depends on the number of racks in the cluster and total number of replicas for a block

considerLoad - whether or not to consider load of the target node

results - A list containing currently chosen nodes. Used to check if too many nodes has been chosen in the target rack.

avoidStaleNodes - Whether or not to avoid choosing stale nodes.

返回:

Return true if node has enough space, does not have too much load, and the rack does not have too many nodes.

verifyBlockPlacement

public int verifyBlockPlacement(String srcPath,
                       LocatedBlock lBlk,
                       int minRacks)

Verify that the block is replicated on at least minRacks different racks if there is more than minRacks rack in the system.

指定者:

verifyBlockPlacement 在类中 BlockPlacementPolicy

参数:

srcPath - the full pathname of the file to be verified

lBlk - block with locations

minRacks - number of racks the block should be replicated to

返回:

the difference between the required and the actual number of racks the block is replicated to.

chooseReplicaToDelete

public DatanodeDescriptor chooseReplicaToDelete(FSInodeInfo inode,
                                       Block block,
                                       short replicationFactor,
                                       Collection<DatanodeDescriptor> first,
                                       Collection<DatanodeDescriptor> second)

Decide whether deleting the specified replica of the block still makes the block conform to the configured block placement policy.

指定者:

chooseReplicaToDelete 在类中 BlockPlacementPolicy

参数:

inode - The inode of the file to which the block-to-be-deleted belongs

block - The block to be deleted

replicationFactor - The required number of replicas for this block

first - The replica locations of this block that are present on at least two unique racks.

second - Replica locations of this block that are not listed in the previous parameter.

返回:

the replica that is the best candidate for deletion

pickupReplicaSet

protected Iterator<DatanodeDescriptor> pickupReplicaSet(Collection<DatanodeDescriptor> first,
                                            Collection<DatanodeDescriptor> second)

Pick up replica node set for deleting replica as over-replicated. First set contains replica nodes on rack with more than one replica while second set contains remaining replica nodes. So pick up first set if not empty. If first is empty, then pick second.