org.apache.hadoop.hbase.replication.regionserver

Class ReplicationSource

java.lang.Object

java.lang.Thread

org.apache.hadoop.hbase.replication.regionserver.ReplicationSource

All Implemented Interfaces:

Runnable, ReplicationSourceInterface

@InterfaceAudience.Private
public class ReplicationSource
extends Thread
implements ReplicationSourceInterface

Class that handles the source of a replication stream. Currently does not handle more than 1 slave For each slave cluster it selects a random number of peers using a replication ratio. For example, if replication ration = 0.1 and slave cluster has 100 region servers, 10 will be selected.

A stream is considered down when we cannot contact a region server on the peer cluster for more than 55 seconds by default.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ReplicationSource.LogsComparator Comparator used to compare logs together based on their start time

Nested classes/interfaces inherited from class java.lang.Thread

Thread.State, Thread.UncaughtExceptionHandler

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	LOG

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor and Description
ReplicationSource()

Method Summary

Methods

Modifier and Type	Method and Description
void	enqueueLog(org.apache.hadoop.fs.Path log) Add a log to the list of logs to replicate
org.apache.hadoop.fs.Path	getCurrentPath() Get the current log that's replicated
protected boolean	getNextPath() Poll for the next path
String	getPeerClusterId() Get the id that the source is replicating to.
String	getPeerClusterZnode() Get the id that the source is replicating to
String	getStats() Get a string representation of the current statistics for this source
void	init(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, ReplicationSourceManager manager, ReplicationQueues replicationQueues, ReplicationPeers replicationPeers, Stoppable stopper, String peerClusterZnode, UUID clusterId) Instantiation method used by region servers
protected boolean	isPeerEnabled() check whether the peer is enabled or not
protected boolean	openReader(int sleepMultiplier) Open a reader on the current path
protected boolean	processEndOfFile() If the queue isn't empty, switch to the next one Else if this is a recovered queue, it means we're done! Else we'll just continue to try reading the log file
protected boolean	readAllEntriesToReplicateOrNextFile(boolean currentWALisBeingWrittenTo, List<HLog.Entry> entries) Read all the entries from the current log files and retain those that need to be replicated.
protected void	removeNonReplicableEdits(HLog.Entry entry) We only want KVs that are scoped other than local
void	run()
protected void	shipEdits(boolean currentWALisBeingWrittenTo, List<HLog.Entry> entries) Do the shipping logic
protected boolean	sleepForRetries(String msg, int sleepMultiplier) Do the sleeping logic
void	startup() Start the replication
void	terminate(String reason) End the replication
void	terminate(String reason, Exception cause) End the replication

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

Constructor Detail

ReplicationSource

public ReplicationSource()

Method Detail

init

public void init(org.apache.hadoop.conf.Configuration conf,
        org.apache.hadoop.fs.FileSystem fs,
        ReplicationSourceManager manager,
        ReplicationQueues replicationQueues,
        ReplicationPeers replicationPeers,
        Stoppable stopper,
        String peerClusterZnode,
        UUID clusterId)
          throws IOException

Instantiation method used by region servers

Specified by:

init in interface ReplicationSourceInterface

Parameters:

conf - configuration to use

fs - file system to use

manager - replication manager to ping to

stopper - the atomic boolean to use to stop the regionserver

peerClusterZnode - the name of our znode

Throws:

IOException

enqueueLog

public void enqueueLog(org.apache.hadoop.fs.Path log)

Description copied from interface: ReplicationSourceInterface

Add a log to the list of logs to replicate

Specified by:

enqueueLog in interface ReplicationSourceInterface

Parameters:

log - path to the log to replicate

run

public void run()

Specified by:

run in interface Runnable

Overrides:

run in class Thread

readAllEntriesToReplicateOrNextFile

protected boolean readAllEntriesToReplicateOrNextFile(boolean currentWALisBeingWrittenTo,
                                          List<HLog.Entry> entries)
                                               throws IOException

Read all the entries from the current log files and retain those that need to be replicated. Else, process the end of the current file.

Parameters:

currentWALisBeingWrittenTo - is the current WAL being written to

entries - resulting entries to be replicated

Returns:

true if we got nothing and went to the next file, false if we got entries

Throws:

IOException

getNextPath

protected boolean getNextPath()

Poll for the next path

Returns:

true if a path was obtained, false if not

openReader

protected boolean openReader(int sleepMultiplier)

Open a reader on the current path

Parameters:

sleepMultiplier - by how many times the default sleeping time is augmented

Returns:

true if we should continue with that file, false if we are over with it

sleepForRetries

protected boolean sleepForRetries(String msg,
                      int sleepMultiplier)

Do the sleeping logic

Parameters:

msg - Why we sleep

sleepMultiplier - by how many times the default sleeping time is augmented

Returns:

True if sleepMultiplier is < maxRetriesMultiplier

removeNonReplicableEdits

protected void removeNonReplicableEdits(HLog.Entry entry)

We only want KVs that are scoped other than local

Parameters:

entry - The entry to check for replication

shipEdits

protected void shipEdits(boolean currentWALisBeingWrittenTo,
             List<HLog.Entry> entries)

Do the shipping logic

Parameters:

currentWALisBeingWrittenTo - was the current WAL being (seemingly) written to when this method was called

isPeerEnabled

protected boolean isPeerEnabled()

check whether the peer is enabled or not

Returns:

true if the peer is enabled, otherwise false

processEndOfFile

protected boolean processEndOfFile()

If the queue isn't empty, switch to the next one Else if this is a recovered queue, it means we're done! Else we'll just continue to try reading the log file

Returns:

true if we're done with the current file, false if we should continue trying to read from it

startup

public void startup()

Description copied from interface: ReplicationSourceInterface

Start the replication

Specified by:

startup in interface ReplicationSourceInterface

terminate

public void terminate(String reason)

Description copied from interface: ReplicationSourceInterface

End the replication

Specified by:

terminate in interface ReplicationSourceInterface

Parameters:

reason - why it's terminating

terminate

public void terminate(String reason,
             Exception cause)

Description copied from interface: ReplicationSourceInterface

End the replication

Specified by:

terminate in interface ReplicationSourceInterface

Parameters:

reason - why it's terminating

cause - the error that's causing it

getPeerClusterZnode

public String getPeerClusterZnode()

Description copied from interface: ReplicationSourceInterface

Get the id that the source is replicating to

Specified by:

getPeerClusterZnode in interface ReplicationSourceInterface

Returns:

peer cluster id

getPeerClusterId

public String getPeerClusterId()

Description copied from interface: ReplicationSourceInterface

Get the id that the source is replicating to.

Specified by:

getPeerClusterId in interface ReplicationSourceInterface

Returns:

peer cluster id

getCurrentPath

public org.apache.hadoop.fs.Path getCurrentPath()

Description copied from interface: ReplicationSourceInterface

Get the current log that's replicated

Specified by:

getCurrentPath in interface ReplicationSourceInterface

Returns:

the current log

getStats

public String getStats()

Description copied from interface: ReplicationSourceInterface

Get a string representation of the current statistics for this source

Specified by:

getStats in interface ReplicationSourceInterface

Returns:

printable stats