org.apache.hadoop.mapred

类 OutputCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapred.OutputCommitter

直接已知子类:

FileOutputCommitter

public abstract class OutputCommitter
extends OutputCommitter

OutputCommitter describes the commit of task output for a Map-Reduce job.

The Map-Reduce framework relies on the OutputCommitter of the job to:

1. Setup the job during initialization. For example, create the temporary output directory for the job during the initialization of the job.
2. Cleanup the job after the job completion. For example, remove the temporary output directory after the job completion.
3. Setup the task temporary output.
4. Check whether a task needs a commit. This is to avoid the commit procedure if a task does not need commit.
5. Commit of the task output.
6. Discard the task commit.

另请参阅:

FileOutputCommitter, JobContext, TaskAttemptContext

构造器概要

构造器

构造器和说明
OutputCommitter()

方法概要

方法

限定符和类型	方法和说明
void	abortJob(JobContext jobContext, int status) For cleaning up the job's output after job failure.
void	abortJob(JobContext context, JobStatus.State runState) This method implements the new interface by calling the old method.
abstract void	abortTask(TaskAttemptContext taskContext) Discard the task output
void	abortTask(TaskAttemptContext taskContext) This method implements the new interface by calling the old method.
void	cleanupJob(JobContext jobContext) 已过时。 use commitJob(JobContext) or abortJob(JobContext, int) instead
void	cleanupJob(JobContext context) 已过时。
void	commitJob(JobContext jobContext) For committing job's output after successful job completion.
void	commitJob(JobContext context) This method implements the new interface by calling the old method.
abstract void	commitTask(TaskAttemptContext taskContext) To promote the task's temporary output to final output location The task's output is moved to the job's output directory.
void	commitTask(TaskAttemptContext taskContext) This method implements the new interface by calling the old method.
abstract boolean	needsTaskCommit(TaskAttemptContext taskContext) Check whether task needs a commit
boolean	needsTaskCommit(TaskAttemptContext taskContext) This method implements the new interface by calling the old method.
abstract void	setupJob(JobContext jobContext) For the framework to setup the job output during initialization
void	setupJob(JobContext jobContext) This method implements the new interface by calling the old method.
abstract void	setupTask(TaskAttemptContext taskContext) Sets up output for the task.
void	setupTask(TaskAttemptContext taskContext) This method implements the new interface by calling the old method.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

OutputCommitter

public OutputCommitter()

方法详细资料

setupJob

public abstract void setupJob(JobContext jobContext)
                       throws IOException

For the framework to setup the job output during initialization

参数:

jobContext - Context of the job whose output is being written.

抛出:

IOException - if temporary output could not be created

cleanupJob

@Deprecated
public void cleanupJob(JobContext jobContext)
                throws IOException

已过时。 use commitJob(JobContext) or abortJob(JobContext, int) instead

For cleaning up the job's output after job completion

抛出:

IOException

commitJob

public void commitJob(JobContext jobContext)
               throws IOException

For committing job's output after successful job completion. Note that this is invoked for jobs with final run state as JobStatus.SUCCEEDED.

参数:

jobContext - Context of the job whose output is being written.

抛出:

IOException

abortJob

public void abortJob(JobContext jobContext,
            int status)
              throws IOException

For cleaning up the job's output after job failure.

参数:

jobContext - Context of the job whose output is being written.

status - Final run state of the job, should be JobStatus.KILLED or JobStatus.FAILED

抛出:

IOException

setupTask

public abstract void setupTask(TaskAttemptContext taskContext)
                        throws IOException

Sets up output for the task.

参数:

taskContext - Context of the task whose output is being written.

抛出:

IOException

needsTaskCommit

public abstract boolean needsTaskCommit(TaskAttemptContext taskContext)
                                 throws IOException

Check whether task needs a commit

参数:

taskContext -

返回:

true/false

抛出:

IOException

commitTask

public abstract void commitTask(TaskAttemptContext taskContext)
                         throws IOException

To promote the task's temporary output to final output location The task's output is moved to the job's output directory.

参数:

taskContext - Context of the task whose output is being written.

抛出:

IOException - if commit is not

abortTask

public abstract void abortTask(TaskAttemptContext taskContext)
                        throws IOException

Discard the task output

参数:

taskContext -

抛出:

IOException

setupJob

public final void setupJob(JobContext jobContext)
                    throws IOException

This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.

指定者:

setupJob 在类中 OutputCommitter

参数:

jobContext - Context of the job whose output is being written.

抛出:

IOException - if temporary output could not be created

cleanupJob

@Deprecated
public final void cleanupJob(JobContext context)
                      throws IOException

已过时。

This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.

覆盖:

cleanupJob 在类中 OutputCommitter

抛出:

IOException

commitJob

public final void commitJob(JobContext context)
                     throws IOException

This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.

覆盖:

commitJob 在类中 OutputCommitter

参数:

context - Context of the job whose output is being written.

抛出:

IOException

abortJob

public final void abortJob(JobContext context,
            JobStatus.State runState)
                    throws IOException

This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.

覆盖:

abortJob 在类中 OutputCommitter

参数:

context - Context of the job whose output is being written.

runState - final run state of the job, should be either JobStatus.State.KILLED or JobStatus.State.FAILED

抛出:

IOException

setupTask

public final void setupTask(TaskAttemptContext taskContext)
                     throws IOException

This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.

指定者:

setupTask 在类中 OutputCommitter

参数:

taskContext - Context of the task whose output is being written.

抛出:

IOException

needsTaskCommit

public final boolean needsTaskCommit(TaskAttemptContext taskContext)
                              throws IOException

This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.

指定者:

needsTaskCommit 在类中 OutputCommitter

返回:

true/false

抛出:

IOException

commitTask

public final void commitTask(TaskAttemptContext taskContext)
                      throws IOException

This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.

指定者:

commitTask 在类中 OutputCommitter

参数:

taskContext - Context of the task whose output is being written.

抛出:

IOException - if commit is not

abortTask

public final void abortTask(TaskAttemptContext taskContext)
                     throws IOException

This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.

指定者:

abortTask 在类中 OutputCommitter

抛出:

IOException