org.apache.hadoop.mapred

类 Task

java.lang.Object

org.apache.hadoop.mapred.Task

所有已实现的接口:

Configurable, Writable

public abstract class Task
extends Object
implements Writable, Configurable

Base class for tasks. This is NOT a public interface.

嵌套类概要

嵌套类

限定符和类型	类和说明
protected static class	Task.CombineOutputCollector<K,V> OutputCollector for the combiner.
protected static class	Task.CombinerRunner<K,V>
protected static class	Task.CombineValuesIterator<KEY,VALUE>
static class	Task.Counter
protected static class	Task.NewCombinerRunner<K,V>
protected static class	Task.OldCombinerRunner<K,V>
protected class	Task.TaskReporter

字段概要

字段

限定符和类型	字段和说明
protected OutputCommitter	committer
protected JobConf	conf
static long	DEFAULT_MR_COMBINE_RECORDS_BEFORE_PROGRESS
protected static String	FILESYSTEM_COUNTER_GROUP Name of the FileSystem counters' group
protected boolean	jobCleanup
protected JobContext	jobContext
protected JobStatus.State	jobRunStateForCleanup
protected boolean	jobSetup
protected org.apache.hadoop.mapred.JvmContext	jvmContext
protected LocalDirAllocator	lDirAlloc
protected org.apache.hadoop.mapred.MapOutputFile	mapOutputFile
static String	MR_COMBINE_RECORDS_BEFORE_PROGRESS
protected OutputFormat<?,?>	outputFormat
static int	PROGRESS_INTERVAL The number of milliseconds between progress reports.
protected Counters.Counter	spilledRecordsCounter
protected boolean	taskCleanup
protected TaskAttemptContext	taskContext
protected SecretKey	tokenSecret
protected TaskUmbilicalProtocol	umbilical

构造器概要

构造器

构造器和说明
Task()
Task(String jobFile, TaskAttemptID taskId, int partition, int numSlotsRequired)

方法概要

方法

限定符和类型	方法和说明
protected static <INKEY,INVALUE,OUTKEY,OUTVALUE> Reducer.Context	createReduceContext(Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE> reducer, Configuration job, TaskAttemptID taskId, RawKeyValueIterator rIter, Counter inputKeyCounter, Counter inputValueCounter, RecordWriter<OUTKEY,OUTVALUE> output, OutputCommitter committer, StatusReporter reporter, RawComparator<INKEY> comparator, Class<INKEY> keyClass, Class<INVALUE> valueClass)
abstract org.apache.hadoop.mapred.TaskRunner	createRunner(TaskTracker tracker, org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip, org.apache.hadoop.mapred.TaskTracker.RunningJob rjob) Return an approprate thread runner for this task.
void	done(TaskUmbilicalProtocol umbilical, Task.TaskReporter reporter)
Configuration	getConf() Return the configuration used by this object.
protected static String[]	getFileSystemCounterNames(String uriScheme) Counters to measure the usage of the different file systems.
protected static FileSystem.Statistics	getFsStatistics(Path path, Configuration conf) Gets a handle to the Statistics instance based on the scheme associated with path.
String	getJobFile()
JobID	getJobID() Get the job name for this task.
SecretKey	getJobTokenSecret() Get the job token secret
org.apache.hadoop.mapred.JvmContext	getJvmContext() Gets the task JvmContext
int	getNumSlotsRequired()
int	getPartition() Get the index of this task within the job.
TaskStatus.Phase	getPhase() Return current phase of the task.
Progress	getProgress()
org.apache.hadoop.mapred.SortedRanges	getSkipRanges() Get skipRanges.
TaskAttemptID	getTaskID()
String	getUser() Get the name of the user running the job/task.
void	initialize(JobConf job, JobID id, Reporter reporter, boolean useNewApi)
abstract boolean	isMapTask()
boolean	isSkipping() Is Task in skipping mode.
void	localizeConfiguration(JobConf conf) Localize the given JobConf to be specific for this task.
void	readFields(DataInput in) Deserialize the fields of this object from in.
protected void	reportFatalError(TaskAttemptID id, Throwable throwable, String logMsg) Report a fatal error to the parent (task) tracker.
protected void	reportNextRecordRange(TaskUmbilicalProtocol umbilical, long nextRecIndex) Reports the next executing record range to TaskTracker.
abstract void	run(JobConf job, TaskUmbilicalProtocol umbilical) Run this task as a part of the named job.
protected void	runJobCleanupTask(TaskUmbilicalProtocol umbilical, Task.TaskReporter reporter)
protected void	runJobSetupTask(TaskUmbilicalProtocol umbilical, Task.TaskReporter reporter)
protected void	runTaskCleanupTask(TaskUmbilicalProtocol umbilical, Task.TaskReporter reporter)
void	setConf(Configuration conf) Set the configuration to be used by this object.
void	setJobFile(String jobFile)
void	setJobTokenSecret(SecretKey tokenSecret) Set the job token secret
void	setJvmContext(org.apache.hadoop.mapred.JvmContext jvmContext) Set the task JvmContext
protected void	setPhase(TaskStatus.Phase phase) Set current phase of the task.
void	setSkipping(boolean skipping) Sets whether to run Task in skipping mode.
void	setSkipRanges(org.apache.hadoop.mapred.SortedRanges skipRanges) Set skipRanges.
protected void	setWriteSkipRecs(boolean writeSkipRecs) Set whether to write skip records.
protected void	statusUpdate(TaskUmbilicalProtocol umbilical)
protected boolean	supportIsolationRunner(JobConf conf)
String	toString()
protected boolean	toWriteSkipRecs() Get whether to write skip records.
void	write(DataOutput out) Serialize the fields of this object to out.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

字段详细资料

MR_COMBINE_RECORDS_BEFORE_PROGRESS

public static final String MR_COMBINE_RECORDS_BEFORE_PROGRESS

另请参阅:

常量字段值

DEFAULT_MR_COMBINE_RECORDS_BEFORE_PROGRESS

public static final long DEFAULT_MR_COMBINE_RECORDS_BEFORE_PROGRESS

另请参阅:

常量字段值

FILESYSTEM_COUNTER_GROUP

protected static final String FILESYSTEM_COUNTER_GROUP

Name of the FileSystem counters' group

另请参阅:

常量字段值

jobRunStateForCleanup

protected JobStatus.State jobRunStateForCleanup

jobCleanup

protected boolean jobCleanup

jobSetup

protected boolean jobSetup

taskCleanup

protected boolean taskCleanup

conf

protected JobConf conf

mapOutputFile

protected org.apache.hadoop.mapred.MapOutputFile mapOutputFile

lDirAlloc

protected LocalDirAllocator lDirAlloc

jobContext

protected JobContext jobContext

taskContext

protected TaskAttemptContext taskContext

outputFormat

protected OutputFormat<?,?> outputFormat

committer

protected OutputCommitter committer

spilledRecordsCounter

protected final Counters.Counter spilledRecordsCounter

umbilical

protected TaskUmbilicalProtocol umbilical

tokenSecret

protected SecretKey tokenSecret

jvmContext

protected org.apache.hadoop.mapred.JvmContext jvmContext

PROGRESS_INTERVAL

public static final int PROGRESS_INTERVAL

The number of milliseconds between progress reports.

另请参阅:

常量字段值

构造器详细资料

Task

public Task()

Task

public Task(String jobFile,
    TaskAttemptID taskId,
    int partition,
    int numSlotsRequired)

方法详细资料

getFileSystemCounterNames

protected static String[] getFileSystemCounterNames(String uriScheme)

Counters to measure the usage of the different file systems. Always return the String array with two elements. First one is the name of BYTES_READ counter and second one is of the BYTES_WRITTEN counter.

setJobFile

public void setJobFile(String jobFile)

getJobFile

public String getJobFile()

getTaskID

public TaskAttemptID getTaskID()

getNumSlotsRequired

public int getNumSlotsRequired()

getJobID

public JobID getJobID()

Get the job name for this task.

返回:

the job name

setJobTokenSecret

public void setJobTokenSecret(SecretKey tokenSecret)

Set the job token secret

参数:

tokenSecret - the secret

getJobTokenSecret

public SecretKey getJobTokenSecret()

Get the job token secret

返回:

the token secret

setJvmContext

public void setJvmContext(org.apache.hadoop.mapred.JvmContext jvmContext)

Set the task JvmContext

参数:

jvmContext -

getJvmContext

public org.apache.hadoop.mapred.JvmContext getJvmContext()

Gets the task JvmContext

返回:

the jvm context

getPartition

public int getPartition()

Get the index of this task within the job.

返回:

the integer part of the task id

getPhase

public TaskStatus.Phase getPhase()

Return current phase of the task. needs to be synchronized as communication thread sends the phase every second

返回:

the curent phase of the task

setPhase

protected void setPhase(TaskStatus.Phase phase)

Set current phase of the task.

参数:

phase - task phase

toWriteSkipRecs

protected boolean toWriteSkipRecs()

Get whether to write skip records.

setWriteSkipRecs

protected void setWriteSkipRecs(boolean writeSkipRecs)

Set whether to write skip records.

reportFatalError

protected void reportFatalError(TaskAttemptID id,
                    Throwable throwable,
                    String logMsg)

Report a fatal error to the parent (task) tracker.

getSkipRanges

public org.apache.hadoop.mapred.SortedRanges getSkipRanges()

Get skipRanges.

setSkipRanges

public void setSkipRanges(org.apache.hadoop.mapred.SortedRanges skipRanges)

Set skipRanges.

isSkipping

public boolean isSkipping()

Is Task in skipping mode.

setSkipping

public void setSkipping(boolean skipping)

Sets whether to run Task in skipping mode.

参数:

skipping -

getUser

public String getUser()

Get the name of the user running the job/task. TaskTracker needs task's user name even before it's JobConf is localized. So we explicitly serialize the user name.

返回:

user

write

public void write(DataOutput out)
           throws IOException

从接口复制的说明: Writable

Serialize the fields of this object to out.

指定者:

write 在接口中 Writable

参数:

out - DataOuput to serialize this object into.

抛出:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

从接口复制的说明: Writable

Deserialize the fields of this object from in.

For efficiency, implementations should attempt to re-use storage in the existing object where possible.

指定者:

readFields 在接口中 Writable

参数:

in - DataInput to deseriablize this object from.

抛出:

IOException

toString

public String toString()

覆盖:

toString 在类中 Object

localizeConfiguration

public void localizeConfiguration(JobConf conf)
                           throws IOException

Localize the given JobConf to be specific for this task.

抛出:

IOException

run

public abstract void run(JobConf job,
       TaskUmbilicalProtocol umbilical)
                  throws IOException,
                         ClassNotFoundException,
                         InterruptedException

Run this task as a part of the named job. This method is executed in the child process and is what invokes user-supplied map, reduce, etc. methods.

参数:

umbilical - for progress reports

抛出:

IOException

ClassNotFoundException

InterruptedException

createRunner

public abstract org.apache.hadoop.mapred.TaskRunner createRunner(TaskTracker tracker,
                                               org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip,
                                               org.apache.hadoop.mapred.TaskTracker.RunningJob rjob)
                                                          throws IOException

Return an approprate thread runner for this task.

参数:

tip - TODO

抛出:

IOException

isMapTask

public abstract boolean isMapTask()

getProgress

public Progress getProgress()

initialize

public void initialize(JobConf job,
              JobID id,
              Reporter reporter,
              boolean useNewApi)
                throws IOException,
                       ClassNotFoundException,
                       InterruptedException

抛出:

IOException

ClassNotFoundException

InterruptedException

reportNextRecordRange

protected void reportNextRecordRange(TaskUmbilicalProtocol umbilical,
                         long nextRecIndex)
                              throws IOException

Reports the next executing record range to TaskTracker.

参数:

umbilical -

nextRecIndex - the record index which would be fed next.

抛出:

IOException

done

public void done(TaskUmbilicalProtocol umbilical,
        Task.TaskReporter reporter)
          throws IOException,
                 InterruptedException

抛出:

IOException

InterruptedException

statusUpdate

protected void statusUpdate(TaskUmbilicalProtocol umbilical)
                     throws IOException

抛出:

IOException

runTaskCleanupTask

protected void runTaskCleanupTask(TaskUmbilicalProtocol umbilical,
                      Task.TaskReporter reporter)
                           throws IOException,
                                  InterruptedException

抛出:

IOException

InterruptedException

runJobCleanupTask

protected void runJobCleanupTask(TaskUmbilicalProtocol umbilical,
                     Task.TaskReporter reporter)
                          throws IOException,
                                 InterruptedException

抛出:

IOException

InterruptedException

supportIsolationRunner

protected boolean supportIsolationRunner(JobConf conf)

runJobSetupTask

protected void runJobSetupTask(TaskUmbilicalProtocol umbilical,
                   Task.TaskReporter reporter)
                        throws IOException,
                               InterruptedException

抛出:

IOException

InterruptedException

getFsStatistics

protected static FileSystem.Statistics getFsStatistics(Path path,
                                    Configuration conf)
                                                throws IOException

Gets a handle to the Statistics instance based on the scheme associated with path.

参数:

path - the path.

返回:

a Statistics instance, or null if none is found for the scheme.

抛出:

IOException

setConf

public void setConf(Configuration conf)

从接口复制的说明: Configurable

Set the configuration to be used by this object.

指定者:

setConf 在接口中 Configurable

getConf

public Configuration getConf()

从接口复制的说明: Configurable

Return the configuration used by this object.

指定者:

getConf 在接口中 Configurable

createReduceContext

protected static <INKEY,INVALUE,OUTKEY,OUTVALUE> Reducer.Context createReduceContext(Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE> reducer,
                                                                  Configuration job,
                                                                  TaskAttemptID taskId,
                                                                  RawKeyValueIterator rIter,
                                                                  Counter inputKeyCounter,
                                                                  Counter inputValueCounter,
                                                                  RecordWriter<OUTKEY,OUTVALUE> output,
                                                                  OutputCommitter committer,
                                                                  StatusReporter reporter,
                                                                  RawComparator<INKEY> comparator,
                                                                  Class<INKEY> keyClass,
                                                                  Class<INVALUE> valueClass)
                                              throws IOException,
                                                     ClassNotFoundException

抛出:

IOException

ClassNotFoundException