org.apache.hadoop.hdfs.server.namenode

类 BlockPlacementPolicyWithNodeGroup

java.lang.Object

org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy

org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyDefault

org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyWithNodeGroup

public class BlockPlacementPolicyWithNodeGroup
extends BlockPlacementPolicyDefault

The class is responsible for choosing the desired number of targets for placing block replicas on environment with node-group layer. The replica placement strategy is adjusted to: If the writer is on a datanode, the 1st replica is placed on the local node (or local node-group), otherwise a random datanode. The 2nd replica is placed on a datanode that is on a different rack with 1st replica node. The 3rd replica is placed on a datanode which is on a different node-group but the same rack as the second replica node.

嵌套类概要

从类继承的嵌套类/接口 org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy

BlockPlacementPolicy.NotEnoughReplicasException

字段概要

从类继承的字段 org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyDefault

clusterMap, considerLoad

方法概要

方法

限定符和类型	方法和说明
protected int	addToExcludedNodes(DatanodeDescriptor localMachine, HashMap<Node,Node> excludedNodes) Find other nodes in the same nodegroup of localMachine and add them into excludeNodes as replica should not be duplicated for nodes within the same nodegroup
protected void	adjustExcludedNodes(HashMap<Node,Node> excludedNodes, Node chosenNode) After choosing a node to place replica, adjust excluded nodes accordingly.
protected DatanodeDescriptor	chooseLocalNode(DatanodeDescriptor localMachine, HashMap<Node,Node> excludedNodes, long blocksize, int maxNodesPerRack, List<DatanodeDescriptor> results, boolean avoidStaleNodes) choose local node of localMachine as the target.
protected DatanodeDescriptor	chooseLocalRack(DatanodeDescriptor localMachine, HashMap<Node,Node> excludedNodes, long blocksize, int maxNodesPerRack, List<DatanodeDescriptor> results, boolean avoidStaleNodes)
protected void	chooseRemoteRack(int numOfReplicas, DatanodeDescriptor localMachine, HashMap<Node,Node> excludedNodes, long blocksize, int maxReplicasPerRack, List<DatanodeDescriptor> results, boolean avoidStaleNodes)
protected String	getRack(DatanodeInfo cur) Get rack string from a data node
void	initialize(Configuration conf, FSClusterStats stats, NetworkTopology clusterMap) Used to setup a BlockPlacementPolicy object.
Iterator<DatanodeDescriptor>	pickupReplicaSet(Collection<DatanodeDescriptor> first, Collection<DatanodeDescriptor> second) Pick up replica node set for deleting replica as over-replicated.

从类继承的方法 org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyDefault

chooseRandom, chooseRandom, chooseReplicaToDelete, chooseTarget, chooseTarget, chooseTarget, isGoodTarget, verifyBlockPlacement

从类继承的方法 org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy

adjustSetsWithChosenReplica, chooseTarget, getInstance, splitNodesWithRack

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

方法详细资料

initialize

public void initialize(Configuration conf,
              FSClusterStats stats,
              NetworkTopology clusterMap)

从类复制的说明: BlockPlacementPolicyDefault

Used to setup a BlockPlacementPolicy object. This should be defined by all implementations of a BlockPlacementPolicy.

覆盖:

initialize 在类中 BlockPlacementPolicyDefault

参数:

conf - the configuration object

stats - retrieve cluster status from here

clusterMap - cluster topology

chooseLocalNode

protected DatanodeDescriptor chooseLocalNode(DatanodeDescriptor localMachine,
                                 HashMap<Node,Node> excludedNodes,
                                 long blocksize,
                                 int maxNodesPerRack,
                                 List<DatanodeDescriptor> results,
                                 boolean avoidStaleNodes)
                                      throws BlockPlacementPolicy.NotEnoughReplicasException

choose local node of localMachine as the target. if localMachine is not available, choose a node on the same nodegroup or rack instead.

覆盖:

chooseLocalNode 在类中 BlockPlacementPolicyDefault

返回:

the chosen node

抛出:

BlockPlacementPolicy.NotEnoughReplicasException

adjustExcludedNodes

protected void adjustExcludedNodes(HashMap<Node,Node> excludedNodes,
                       Node chosenNode)

从类复制的说明: BlockPlacementPolicyDefault

After choosing a node to place replica, adjust excluded nodes accordingly. It should do nothing here as chosenNode is already put into exlcudeNodes, but it can be overridden in subclass to put more related nodes into excludedNodes.

覆盖:

adjustExcludedNodes 在类中 BlockPlacementPolicyDefault

chooseLocalRack

protected DatanodeDescriptor chooseLocalRack(DatanodeDescriptor localMachine,
                                 HashMap<Node,Node> excludedNodes,
                                 long blocksize,
                                 int maxNodesPerRack,
                                 List<DatanodeDescriptor> results,
                                 boolean avoidStaleNodes)
                                      throws BlockPlacementPolicy.NotEnoughReplicasException

覆盖:

chooseLocalRack 在类中 BlockPlacementPolicyDefault

抛出:

BlockPlacementPolicy.NotEnoughReplicasException

chooseRemoteRack

protected void chooseRemoteRack(int numOfReplicas,
                    DatanodeDescriptor localMachine,
                    HashMap<Node,Node> excludedNodes,
                    long blocksize,
                    int maxReplicasPerRack,
                    List<DatanodeDescriptor> results,
                    boolean avoidStaleNodes)
                         throws BlockPlacementPolicy.NotEnoughReplicasException

覆盖:

chooseRemoteRack 在类中 BlockPlacementPolicyDefault

抛出:

BlockPlacementPolicy.NotEnoughReplicasException

getRack

protected String getRack(DatanodeInfo cur)

从类复制的说明: BlockPlacementPolicy

Get rack string from a data node

覆盖:

getRack 在类中 BlockPlacementPolicy

返回:

rack of data node

addToExcludedNodes

protected int addToExcludedNodes(DatanodeDescriptor localMachine,
                     HashMap<Node,Node> excludedNodes)

Find other nodes in the same nodegroup of localMachine and add them into excludeNodes as replica should not be duplicated for nodes within the same nodegroup

覆盖:

addToExcludedNodes 在类中 BlockPlacementPolicyDefault

返回:

number of new excluded nodes

pickupReplicaSet

public Iterator<DatanodeDescriptor> pickupReplicaSet(Collection<DatanodeDescriptor> first,
                                            Collection<DatanodeDescriptor> second)

Pick up replica node set for deleting replica as over-replicated. First set contains replica nodes on rack with more than one replica while second set contains remaining replica nodes. If first is not empty, divide first set into two subsets: moreThanOne contains nodes on nodegroup with more than one replica exactlyOne contains the remaining nodes in first set then pickup priSet if not empty. If first is empty, then pick second.

覆盖:

pickupReplicaSet 在类中 BlockPlacementPolicyDefault