org.apache.hadoop.hive.ql.txn.compactor

Class Worker

java.lang.Object

java.lang.Thread

org.apache.hadoop.hive.ql.txn.compactor.Worker

All Implemented Interfaces:

Runnable, MetaStoreThread

public class Worker
extends Thread

A class to do compactions. This will run in a separate thread. It will spin on the compaction queue and look for new work to do.

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Thread

Thread.State, Thread.UncaughtExceptionHandler

Field Summary

Fields

Modifier and Type	Field and Description
protected HiveConf	conf
protected AtomicBoolean	looped
protected RawStore	rs
protected AtomicBoolean	stop
protected int	threadId
protected CompactionTxnHandler	txnHandler

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor and Description
Worker()

Method Summary

Methods

Modifier and Type	Method and Description
protected String	findUserToRunAs(String location, Table t) Determine which user to run an operation as, based on the owner of the directory to be compacted.
static String	hostname() Get the hostname that this worker is run on.
void	init(AtomicBoolean stop, AtomicBoolean looped) Initialize the thread.
protected Partition	resolvePartition(CompactionInfo ci) Get the partition being compacted.
protected StorageDescriptor	resolveStorageDescriptor(Table t, Partition p) Get the storage descriptor for a compaction.
protected Table	resolveTable(CompactionInfo ci) Find the table being compacted
void	run()
protected boolean	runJobAsSelf(String owner) Determine whether to run this job as the current user or whether we need a doAs to switch users.
void	setHiveConf(HiveConf conf) Set the Hive configuration for this thread.
void	setThreadId(int threadId) Set the id for this thread.
protected String	tableName(Table t)

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.metastore.MetaStoreThread

start

Field Detail

conf

protected HiveConf conf

txnHandler

protected CompactionTxnHandler txnHandler

rs

protected RawStore rs

threadId

protected int threadId

stop

protected AtomicBoolean stop

looped

protected AtomicBoolean looped

Constructor Detail

Worker

public Worker()

Method Detail

hostname

public static String hostname()

Get the hostname that this worker is run on. Made static and public so that other classes can use the same method to know what host their worker threads are running on.

Returns:

hostname

run

public void run()

Specified by:

run in interface Runnable

Overrides:

run in class Thread

init

public void init(AtomicBoolean stop,
        AtomicBoolean looped)
          throws MetaException

Description copied from interface: MetaStoreThread

Initialize the thread. This must not be called until after MetaStoreThread.setHiveConf(org.apache.hadoop.hive.conf.HiveConf) and MetaStoreThread.setThreadId(int) have been called.

Specified by:

init in interface MetaStoreThread

Parameters:

stop - a flag to watch for when to stop. If this value is set to true, the thread will terminate the next time through its main loop.

looped - a flag that is set to true everytime a thread goes through it's main loop. This is purely for testing so that tests can assure themselves that the thread has run through it's loop once. The test can set this value to false. The thread should then assure that the loop has been gone completely through at least once.

Throws:

MetaException

setHiveConf

public void setHiveConf(HiveConf conf)

Description copied from interface: MetaStoreThread

Set the Hive configuration for this thread.

Specified by:

setHiveConf in interface MetaStoreThread

setThreadId

public void setThreadId(int threadId)

Description copied from interface: MetaStoreThread

Set the id for this thread.

Specified by:

setThreadId in interface MetaStoreThread

resolveTable

protected Table resolveTable(CompactionInfo ci)
                      throws MetaException

Find the table being compacted

Parameters:

ci - compaction info returned from the compaction queue

Returns:

metastore table

Throws:

MetaException - if the table cannot be found.

resolvePartition

protected Partition resolvePartition(CompactionInfo ci)
                              throws Exception

Get the partition being compacted.

Parameters:

ci - compaction info returned from the compaction queue

Returns:

metastore partition, or null if there is not partition in this compaction info

Throws:

Exception - if underlying calls throw, or if the partition name resolves to more than one partition.

resolveStorageDescriptor

protected StorageDescriptor resolveStorageDescriptor(Table t,
                                         Partition p)

Get the storage descriptor for a compaction.

Parameters:

t - table from resolveTable(org.apache.hadoop.hive.metastore.txn.CompactionInfo)

p - table from resolvePartition(org.apache.hadoop.hive.metastore.txn.CompactionInfo)

Returns:

metastore storage descriptor.

findUserToRunAs

protected String findUserToRunAs(String location,
                     Table t)
                          throws IOException,
                                 InterruptedException

Determine which user to run an operation as, based on the owner of the directory to be compacted. It is asserted that either the user running the hive metastore or the table owner must be able to stat the directory and determine the owner.

Parameters:

location - directory that will be read or written to.

t - metastore table object

Returns:

username of the owner of the location.

Throws:

IOException - if neither the hive metastore user nor the table owner can stat the location.

InterruptedException

runJobAsSelf

protected boolean runJobAsSelf(String owner)

Determine whether to run this job as the current user or whether we need a doAs to switch users.

Parameters:

owner - of the directory we will be working in, as determined by findUserToRunAs(String, org.apache.hadoop.hive.metastore.api.Table)

Returns:

true if the job should run as the current user, false if a doAs is needed.

tableName

protected String tableName(Table t)