org.apache.hadoop.hbase.master.balancer

Class StochasticLoadBalancer

java.lang.Object

org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, LoadBalancer, Stoppable

@InterfaceAudience.LimitedPrivate(value="Configuration")
public class StochasticLoadBalancer
extends BaseLoadBalancer

This is a best effort load balancer. Given a Cost function F(C) => x It will randomly try and mutate the cluster to Cprime. If F(Cprime) < F(C) then the new cluster state becomes the plan. It includes costs functions to compute the cost of:

• Region Load
• Table Load
• Data Locality
• Memstore Sizes
• Storefile Sizes

Every cost function returns a number between 0 and 1 inclusive; where 0 is the lowest cost best solution, and 1 is the highest possible cost and the worst solution. The computed costs are scaled by their respective multipliers:

• hbase.master.balancer.stochastic.regionLoadCost
• hbase.master.balancer.stochastic.moveCost
• hbase.master.balancer.stochastic.tableLoadCost
• hbase.master.balancer.stochastic.localityCost
• hbase.master.balancer.stochastic.memstoreSizeCost
• hbase.master.balancer.stochastic.storefileSizeCost

In addition to the above configurations, the balancer can be tuned by the following configuration values:

• hbase.master.balancer.stochastic.maxMoveRegions which controls what the max number of regions that can be moved in a single invocation of this balancer.
• hbase.master.balancer.stochastic.stepsPerRegion is the coefficient by which the number of regions is multiplied to try and get the number of times the balancer will mutate all servers.
• hbase.master.balancer.stochastic.maxSteps which controls the maximum number of times that the balancer will try and mutate all the servers. The balancer will use the minimum of this value and the above computation.

This balancer is best used with hbase.master.loadbalance.bytable set to false so that the balancer gets the full picture of all loads on the cluster.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StochasticLoadBalancer.CostFromRegionLoadFunction Base class the allows writing costs functions from rolling average of some number from RegionLoad.
static class	StochasticLoadBalancer.CostFunction Base class of StochasticLoadBalancer's Cost Functions.
static class	StochasticLoadBalancer.LoadPicker
static class	StochasticLoadBalancer.LocalityCostFunction Compute a cost of a potential cluster configuration based upon where StoreFiles are located.
static class	StochasticLoadBalancer.MemstoreSizeCostFunction Compute the cost of total memstore size.
static class	StochasticLoadBalancer.MoveCostFunction Given the starting state of the regions and a potential ending state compute cost based upon the number of regions that have moved.
static class	StochasticLoadBalancer.ReadRequestCostFunction Compute the cost of total number of read requests The more unbalanced the higher the computed cost will be.
static class	StochasticLoadBalancer.RegionCountSkewCostFunction Compute the cost of a potential cluster state from skew in number of regions on a cluster.
static class	StochasticLoadBalancer.StoreFileCostFunction Compute the cost of total open storefiles size.
static class	StochasticLoadBalancer.TableSkewCostFunction Compute the cost of a potential cluster configuration based upon how evenly distributed tables are.
static class	StochasticLoadBalancer.WriteRequestCostFunction Compute the cost of total number of write requests.

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

BaseLoadBalancer.Cluster

Field Summary

Fields inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

metricsBalancer, services, slop

Constructor Summary

Constructors

Constructor and Description
StochasticLoadBalancer()

Method Summary

Methods

Modifier and Type	Method and Description
List<RegionPlan>	balanceCluster(Map<ServerName,List<HRegionInfo>> clusterState) Given the cluster state this will try and approach an optimal balance.
protected double	computeCost(BaseLoadBalancer.Cluster cluster, double previousCost) This is the main cost function.
void	setClusterStatus(ClusterStatus st) Set the current cluster status.
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setMasterServices(MasterServices masterServices) Set the master service.
protected void	setSlop(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

getConf, immediateAssignment, initialize, isStopped, needsBalance, randomAssignment, regionOffline, regionOnline, retainAssignment, roundRobinAssignment, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StochasticLoadBalancer

public StochasticLoadBalancer()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

Overrides:

setConf in class BaseLoadBalancer

setSlop

protected void setSlop(org.apache.hadoop.conf.Configuration conf)

Overrides:

setSlop in class BaseLoadBalancer

setClusterStatus

public void setClusterStatus(ClusterStatus st)

Description copied from interface: LoadBalancer

Set the current cluster status. This allows a LoadBalancer to map host name to a server

Specified by:

setClusterStatus in interface LoadBalancer

Overrides:

setClusterStatus in class BaseLoadBalancer

setMasterServices

public void setMasterServices(MasterServices masterServices)

Description copied from interface: LoadBalancer

Set the master service.

Specified by:

setMasterServices in interface LoadBalancer

Overrides:

setMasterServices in class BaseLoadBalancer

balanceCluster

public List<RegionPlan> balanceCluster(Map<ServerName,List<HRegionInfo>> clusterState)

Given the cluster state this will try and approach an optimal balance. This should always approach the optimal state given enough steps.

Returns:

List of plans

computeCost

protected double computeCost(BaseLoadBalancer.Cluster cluster,
                 double previousCost)

This is the main cost function. It will compute a cost associated with a proposed cluster state. All different costs will be combined with their multipliers to produce a double cost.

Parameters:

cluster - The state of the cluster

previousCost - the previous cost. This is used as an early out.

Returns:

a double of a cost associated with the proposed cluster state. This cost is an aggregate of all individual cost functions.