org.apache.hadoop.mapred

类 TaskTracker

java.lang.Object

org.apache.hadoop.mapred.TaskTracker

所有已实现的接口:

Runnable, VersionedProtocol, TaskTrackerMXBean, TaskUmbilicalProtocol

public class TaskTracker
extends Object
implements TaskUmbilicalProtocol, Runnable, TaskTrackerMXBean

TaskTracker is a process that starts and tracks MR Tasks in a networked environment. It contacts the JobTracker for Task assignments and reporting results.

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	TaskTracker.MapOutputServlet This class is used in TaskTracker's Jetty to serve the map outputs to other nodes.

字段概要

字段

限定符和类型	字段和说明
static org.apache.commons.logging.Log	ClientTraceLog
static long	COUNTER_UPDATE_INTERVAL
static long	DEFAULT_DISK_HEALTH_CHECK_INTERVAL How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
static int	FILE_NOT_FOUND
static String	FOR_REDUCE_TASK The reduce task number for which this map output is being transferred
static String	FROM_MAP_TASK The map task from which the map output data is being transferred
static int	HEARTBEAT_INTERVAL_MIN
static org.apache.commons.logging.Log	LOG
static String	MAP_OUTPUT_LENGTH The custom http header used for the map output length.
static String	MR_CLIENTTRACE_FORMAT
static String	RAW_MAP_OUTPUT_LENGTH The custom http header used for the "raw" map output length.
static String	SUBDIR
static int	SUCCESS
static String	TT_KEYTAB_FILE
static String	TT_LOG_TMP_DIR
static String	TT_RESOURCE_CALCULATOR_PLUGIN
static String	TT_USER_NAME
static String	WORKDIR

从接口继承的字段 org.apache.hadoop.mapred.TaskUmbilicalProtocol

versionID

构造器概要

构造器

构造器和说明
TaskTracker(JobConf conf) Start with the local machine name, and the default JobTracker

方法概要

方法

限定符和类型	方法和说明
boolean	canCommit(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext) Child checking whether it can commit
void	cleanupStorage() Removes all contents of temporary storage.
void	close() Close down the TaskTracker and all its components.
void	commitPending(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext) Task is reporting that it is in commit_pending and it is waiting for the commit Response
void	done(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext) The task is done.
void	fatalError(TaskAttemptID taskId, String msg, org.apache.hadoop.mapred.JvmContext jvmContext) A child task had a fatal error.
void	fsError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext) A child task had a local filesystem error.
String	getConfigVersion()
String	getHostname()
int	getHttpPort()
static String	getJobCacheSubdir(String user)
org.apache.hadoop.mapred.InterTrackerProtocol	getJobClient() The connection to the JobTracker, used by the TaskRunner for locating remote files.
static String	getJobJarFile(String user, String jobid)
String	getJobTrackerUrl()
org.apache.hadoop.mapred.JvmManager	getJvmManagerInstance()
static String	getLocalJobDir(String user, String jobid)
static String	getLocalTaskDir(String user, String jobid, String taskid)
static String	getLocalTaskDir(String user, String jobid, String taskid, boolean isCleanupAttempt)
MapTaskCompletionEventsUpdate	getMapCompletionEvents(JobID jobId, int fromEventId, int maxLocs, TaskAttemptID id, org.apache.hadoop.mapred.JvmContext jvmContext) Called by a reduce task to get the map output locations for finished maps.
static String	getPrivateDistributedCacheDir(String user)
long	getProtocolVersion(String protocol, long clientVersion) Return protocol version corresponding to protocol interface.
static String	getPublicDistributedCacheDir()
int	getRpcPort()
JvmTask	getTask(org.apache.hadoop.mapred.JvmContext context) Called upon startup by the child process, to fetch Task data.
TaskController	getTaskController()
org.apache.hadoop.mapred.TaskMemoryManagerThread	getTaskMemoryManager()
String	getTasksInfoJson()
org.apache.hadoop.mapred.TaskTrackerInstrumentation	getTaskTrackerInstrumentation()
InetSocketAddress	getTaskTrackerReportAddress() Return the port at which the tasktracker bound to
static String	getUserDir(String user)
String	getVersion()
boolean	isHealthy()
boolean	isIdle() Is this task tracker idle?
boolean	isIdleAndClean() Is this task tracker idle and clean?
boolean	isTaskMemoryManagerEnabled() Is the TaskMemoryManager Enabled on this system?
protected void	launchTaskForJob(org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip, JobConf jobConf, org.apache.hadoop.mapred.TaskTracker.RunningJob rjob)
static void	main(String[] argv) Start the TaskTracker, point toward the indicated JobTracker
void	mapOutputLost(TaskAttemptID taskid, String errorMsg) A completed map task's output has been lost.
boolean	ping(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext) Child checking to see if we're alive.
void	reportDiagnosticInfo(TaskAttemptID taskid, String info, org.apache.hadoop.mapred.JvmContext jvmContext) Called when the task dies before completion, and we want to report back diagnostic info
void	reportNextRecordRange(TaskAttemptID taskid, org.apache.hadoop.mapred.SortedRanges.Range range, org.apache.hadoop.mapred.JvmContext jvmContext) Report the record range which is going to process next by the Task.
void	run() The server retry loop.
void	shuffleError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext) A reduce-task failed to shuffle the map-outputs.
void	shutdown()
boolean	statusUpdate(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext) Called periodically to report Task progress, from 0.0 to 1.0.
void	updatePrivateDistributedCacheSizes(JobID jobId, long[] sizes) The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

LOG

public static final org.apache.commons.logging.Log LOG

MR_CLIENTTRACE_FORMAT

public static final String MR_CLIENTTRACE_FORMAT

另请参阅:

常量字段值

ClientTraceLog

public static final org.apache.commons.logging.Log ClientTraceLog

SUBDIR

public static final String SUBDIR

另请参阅:

常量字段值

TT_LOG_TMP_DIR

public static final String TT_LOG_TMP_DIR

另请参阅:

常量字段值

TT_RESOURCE_CALCULATOR_PLUGIN

public static final String TT_RESOURCE_CALCULATOR_PLUGIN

另请参阅:

常量字段值

TT_USER_NAME

public static final String TT_USER_NAME

另请参阅:

常量字段值

TT_KEYTAB_FILE

public static final String TT_KEYTAB_FILE

另请参阅:

常量字段值

HEARTBEAT_INTERVAL_MIN

public static final int HEARTBEAT_INTERVAL_MIN

另请参阅:

常量字段值

COUNTER_UPDATE_INTERVAL

public static final long COUNTER_UPDATE_INTERVAL

另请参阅:

常量字段值

DEFAULT_DISK_HEALTH_CHECK_INTERVAL

public static final long DEFAULT_DISK_HEALTH_CHECK_INTERVAL

How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval

另请参阅:

常量字段值

SUCCESS

public static final int SUCCESS

另请参阅:

常量字段值

FILE_NOT_FOUND

public static final int FILE_NOT_FOUND

另请参阅:

常量字段值

MAP_OUTPUT_LENGTH

public static final String MAP_OUTPUT_LENGTH

The custom http header used for the map output length.

另请参阅:

常量字段值

RAW_MAP_OUTPUT_LENGTH

public static final String RAW_MAP_OUTPUT_LENGTH

The custom http header used for the "raw" map output length.

另请参阅:

常量字段值

FROM_MAP_TASK

public static final String FROM_MAP_TASK

The map task from which the map output data is being transferred

另请参阅:

常量字段值

FOR_REDUCE_TASK

public static final String FOR_REDUCE_TASK

The reduce task number for which this map output is being transferred

另请参阅:

常量字段值

WORKDIR

public static final String WORKDIR

另请参阅:

常量字段值

构造器详细资料

TaskTracker

public TaskTracker(JobConf conf)
            throws IOException,
                   InterruptedException

Start with the local machine name, and the default JobTracker

抛出:

IOException

InterruptedException

方法详细资料

getTaskTrackerInstrumentation

public org.apache.hadoop.mapred.TaskTrackerInstrumentation getTaskTrackerInstrumentation()

getTaskController

public TaskController getTaskController()

getUserDir

public static String getUserDir(String user)

getPrivateDistributedCacheDir

public static String getPrivateDistributedCacheDir(String user)

getPublicDistributedCacheDir

public static String getPublicDistributedCacheDir()

getJobCacheSubdir

public static String getJobCacheSubdir(String user)

getLocalJobDir

public static String getLocalJobDir(String user,
                    String jobid)

getJobJarFile

public static String getJobJarFile(String user,
                   String jobid)

getLocalTaskDir

public static String getLocalTaskDir(String user,
                     String jobid,
                     String taskid)

getLocalTaskDir

public static String getLocalTaskDir(String user,
                     String jobid,
                     String taskid,
                     boolean isCleanupAttempt)

getProtocolVersion

public long getProtocolVersion(String protocol,
                      long clientVersion)
                        throws IOException

从接口复制的说明: VersionedProtocol

Return protocol version corresponding to protocol interface.

指定者:

getProtocolVersion 在接口中 VersionedProtocol

参数:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

返回:

the version that the server will speak

抛出:

IOException

cleanupStorage

public void cleanupStorage()
                    throws IOException

Removes all contents of temporary storage. Called upon startup, to remove any leftovers from previous run.

抛出:

IOException

launchTaskForJob

protected void launchTaskForJob(org.apache.hadoop.mapred.TaskTracker.TaskInProgress tip,
                    JobConf jobConf,
                    org.apache.hadoop.mapred.TaskTracker.RunningJob rjob)
                         throws IOException

抛出:

IOException

shutdown

public void shutdown()
              throws IOException,
                     InterruptedException

抛出:

IOException

InterruptedException

close

public void close()
           throws IOException,
                  InterruptedException

Close down the TaskTracker and all its components. We must also shutdown any running tasks or threads, and cleanup disk space. A new TaskTracker within the same process space might be restarted, so everything must be clean.

抛出:

InterruptedException

IOException

getJobClient

public org.apache.hadoop.mapred.InterTrackerProtocol getJobClient()

The connection to the JobTracker, used by the TaskRunner for locating remote files.

getTaskTrackerReportAddress

public InetSocketAddress getTaskTrackerReportAddress()

Return the port at which the tasktracker bound to

getJvmManagerInstance

public org.apache.hadoop.mapred.JvmManager getJvmManagerInstance()

run

public void run()

The server retry loop. This while-loop attempts to connect to the JobTracker. It only loops when the old TaskTracker has gone bad (its state is stale somehow) and we need to reinitialize everything.

指定者:

run 在接口中 Runnable

getTask

public JvmTask getTask(org.apache.hadoop.mapred.JvmContext context)
                throws IOException

Called upon startup by the child process, to fetch Task data.

指定者:

getTask 在接口中 TaskUmbilicalProtocol

参数:

context - the JvmContext of the JVM w.r.t the TaskTracker that launched it

返回:

Task object

抛出:

IOException

statusUpdate

public boolean statusUpdate(TaskAttemptID taskid,
                   TaskStatus taskStatus,
                   org.apache.hadoop.mapred.JvmContext jvmContext)
                     throws IOException

Called periodically to report Task progress, from 0.0 to 1.0.

指定者:

statusUpdate 在接口中 TaskUmbilicalProtocol

参数:

taskid - task-id of the child

taskStatus - status of the child

jvmContext - context the jvmContext running the task.

返回:

True if the task is known

抛出:

IOException

reportDiagnosticInfo

public void reportDiagnosticInfo(TaskAttemptID taskid,
                        String info,
                        org.apache.hadoop.mapred.JvmContext jvmContext)
                          throws IOException

Called when the task dies before completion, and we want to report back diagnostic info

指定者:

reportDiagnosticInfo 在接口中 TaskUmbilicalProtocol

参数:

taskid - the id of the task involved

info - the text to report

jvmContext - context the jvmContext running the task.

抛出:

IOException

reportNextRecordRange

public void reportNextRecordRange(TaskAttemptID taskid,
                         org.apache.hadoop.mapred.SortedRanges.Range range,
                         org.apache.hadoop.mapred.JvmContext jvmContext)
                           throws IOException

从接口复制的说明: TaskUmbilicalProtocol

Report the record range which is going to process next by the Task.

指定者:

reportNextRecordRange 在接口中 TaskUmbilicalProtocol

参数:

taskid - the id of the task involved

range - the range of record sequence nos

jvmContext - context the jvmContext running the task.

抛出:

IOException

ping

public boolean ping(TaskAttemptID taskid,
           org.apache.hadoop.mapred.JvmContext jvmContext)
             throws IOException

Child checking to see if we're alive. Normally does nothing.

指定者:

ping 在接口中 TaskUmbilicalProtocol

参数:

taskid - the id of the task involved

jvmContext - context the jvmContext running the task.

返回:

True if the task is known

抛出:

IOException

commitPending

public void commitPending(TaskAttemptID taskid,
                 TaskStatus taskStatus,
                 org.apache.hadoop.mapred.JvmContext jvmContext)
                   throws IOException

Task is reporting that it is in commit_pending and it is waiting for the commit Response

指定者:

commitPending 在接口中 TaskUmbilicalProtocol

参数:

taskid - task's id

taskStatus - status of the child

jvmContext - context the jvmContext running the task.

抛出:

IOException

canCommit

public boolean canCommit(TaskAttemptID taskid,
                org.apache.hadoop.mapred.JvmContext jvmContext)
                  throws IOException

Child checking whether it can commit

指定者:

canCommit 在接口中 TaskUmbilicalProtocol

jvmContext - context the jvmContext running the task.

返回:

true/false

抛出:

IOException

done

public void done(TaskAttemptID taskid,
        org.apache.hadoop.mapred.JvmContext jvmContext)
          throws IOException

The task is done.

指定者:

done 在接口中 TaskUmbilicalProtocol

参数:

taskid - task's id

jvmContext - context the jvmContext running the task.

抛出:

IOException

shuffleError

public void shuffleError(TaskAttemptID taskId,
                String message,
                org.apache.hadoop.mapred.JvmContext jvmContext)
                  throws IOException

A reduce-task failed to shuffle the map-outputs. Kill the task.

指定者:

shuffleError 在接口中 TaskUmbilicalProtocol

抛出:

IOException

fsError

public void fsError(TaskAttemptID taskId,
           String message,
           org.apache.hadoop.mapred.JvmContext jvmContext)
             throws IOException

A child task had a local filesystem error. Kill the task.

指定者:

fsError 在接口中 TaskUmbilicalProtocol

抛出:

IOException

fatalError

public void fatalError(TaskAttemptID taskId,
              String msg,
              org.apache.hadoop.mapred.JvmContext jvmContext)
                throws IOException

A child task had a fatal error. Kill the task.

指定者:

fatalError 在接口中 TaskUmbilicalProtocol

抛出:

IOException

getMapCompletionEvents

public MapTaskCompletionEventsUpdate getMapCompletionEvents(JobID jobId,
                                                   int fromEventId,
                                                   int maxLocs,
                                                   TaskAttemptID id,
                                                   org.apache.hadoop.mapred.JvmContext jvmContext)
                                                     throws IOException

从接口复制的说明: TaskUmbilicalProtocol

Called by a reduce task to get the map output locations for finished maps. Returns an update centered around the map-task-completion-events. The update also piggybacks the information whether the events copy at the task-tracker has changed or not. This will trigger some action at the child-process.

指定者:

getMapCompletionEvents 在接口中 TaskUmbilicalProtocol

参数:

jobId - the reducer job id

fromEventId - the index starting from which the locations should be fetched

maxLocs - the max number of locations to fetch

id - The attempt id of the task that is trying to communicate

返回:

A MapTaskCompletionEventsUpdate

抛出:

IOException

mapOutputLost

public void mapOutputLost(TaskAttemptID taskid,
                 String errorMsg)
                   throws IOException

A completed map task's output has been lost.

抛出:

IOException

isIdle

public boolean isIdle()

Is this task tracker idle?

返回:

has this task tracker finished all its assigned tasks?

isIdleAndClean

public boolean isIdleAndClean()

Is this task tracker idle and clean?

返回:

has this task tracker finished and cleaned up all of its tasks?

main

public static void main(String[] argv)
                 throws Exception

Start the TaskTracker, point toward the indicated JobTracker

抛出:

Exception

isTaskMemoryManagerEnabled

public boolean isTaskMemoryManagerEnabled()

Is the TaskMemoryManager Enabled on this system?

返回:

true if enabled, false otherwise.

getTaskMemoryManager

public org.apache.hadoop.mapred.TaskMemoryManagerThread getTaskMemoryManager()

getHostname

public String getHostname()

指定者:

getHostname 在接口中 TaskTrackerMXBean

返回:

the hostname of the tasktracker

getVersion

public String getVersion()

指定者:

getVersion 在接口中 TaskTrackerMXBean

返回:

the version of the code base

getConfigVersion

public String getConfigVersion()

指定者:

getConfigVersion 在接口中 TaskTrackerMXBean

返回:

the config version (from a config properties)

getJobTrackerUrl

public String getJobTrackerUrl()

指定者:

getJobTrackerUrl 在接口中 TaskTrackerMXBean

返回:

the URL of the jobtracker

getRpcPort

public int getRpcPort()

指定者:

getRpcPort 在接口中 TaskTrackerMXBean

返回:

the RPC port of the tasktracker

getHttpPort

public int getHttpPort()

指定者:

getHttpPort 在接口中 TaskTrackerMXBean

返回:

the HTTP port of the tasktracker

isHealthy

public boolean isHealthy()

指定者:

isHealthy 在接口中 TaskTrackerMXBean

返回:

the health status of the tasktracker

getTasksInfoJson

public String getTasksInfoJson()

指定者:

getTasksInfoJson 在接口中 TaskTrackerMXBean

返回:

a json formatted info about tasks of the tasktracker

updatePrivateDistributedCacheSizes

public void updatePrivateDistributedCacheSizes(JobID jobId,
                                      long[] sizes)
                                        throws IOException

从接口复制的说明: TaskUmbilicalProtocol

The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.

指定者:

updatePrivateDistributedCacheSizes 在接口中 TaskUmbilicalProtocol

参数:

jobId - the job to update

sizes - the array of sizes that were computed

抛出:

IOException