JobClient (Hadoop 1.2.2-SNAPSHOT API)

java.lang.Object
- org.apache.hadoop.conf.Configured
- - org.apache.hadoop.mapred.JobClient

所有已实现的接口:

Configurable, Tool
```
public class JobClient
extends Configured
implements Tool
```
JobClient is the primary interface for the user-job to interact with the JobTracker. JobClient provides facilities to submit jobs, track their progress, access component-tasks' reports/logs, get the Map-Reduce cluster status information etc.
The job submission process involves:
1. Checking the input and output specifications of the job.
2. Computing the InputSplits for the job.
3. Setup the requisite accounting information for the DistributedCache of the job, if necessary.
4. Copying the job's jar and configuration to the map-reduce system directory on the distributed file-system.
5. Submitting the job to the JobTracker and optionally monitoring it's status.
Normally the user creates the application, describes various facets of the job via JobConf and then uses the JobClient to submit the job and monitor its progress.
Here is an example on how to use JobClient:
```
     // Create a new JobConf
     JobConf job = new JobConf(new Configuration(), MyJob.class);
     
     // Specify various job-specific parameters     
     job.setJobName("myjob");
     
     job.setInputPath(new Path("in"));
     job.setOutputPath(new Path("out"));
     
     job.setMapperClass(MyJob.MyMapper.class);
     job.setReducerClass(MyJob.MyReducer.class);

     // Submit the job, then poll for progress until the job is complete
     JobClient.runJob(job);
 
```
Job Control

At times clients would chain map-reduce jobs to accomplish complex tasks which cannot be done via a single map-reduce job. This is fairly easy since the output of the job, typically, goes to distributed file-system and that can be used as the input for the next job.

However, this also means that the onus on ensuring jobs are complete (success/failure) lies squarely on the clients. In such situations the various job-control options are:
1. runJob(JobConf) : submits the job and returns only after the job has completed.
2. submitJob(JobConf) : only submits the job, then poll the returned handle to the RunningJob to query status and make scheduling decisions.
3. JobConf.setJobEndNotificationURI(String) : setup a notification on job-completion, thus avoiding polling.
另请参阅:
JobConf, ClusterStatus, Tool, DistributedCache

嵌套类概要

嵌套类
限定符和类型类和说明

static class JobClient.Renewer

static class JobClient.TaskStatusFilter

嵌套类
限定符和类型	类和说明
`static class`	`JobClient.Renewer`
`static class`	`JobClient.TaskStatusFilter`

字段概要

字段
限定符和类型	字段和说明
`static long`	`COUNTER_UPDATE_INTERVAL`
`static long`	`DEFAULT_DISK_HEALTH_CHECK_INTERVAL` How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
`static int`	`FILE_NOT_FOUND`
`static String`	`FOR_REDUCE_TASK` The reduce task number for which this map output is being transferred
`static String`	`FROM_MAP_TASK` The map task from which the map output data is being transferred
`static int`	`HEARTBEAT_INTERVAL_MIN`
`static String`	`MAP_OUTPUT_LENGTH` The custom http header used for the map output length.
`static boolean`	`MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_DEFAULT`
`static String`	`MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_KEY`
`static String`	`MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_DEFAULT`
`static String`	`MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_KEY`
`static String`	`RAW_MAP_OUTPUT_LENGTH` The custom http header used for the "raw" map output length.
`static int`	`SUCCESS`
`static String`	`WORKDIR`

构造器概要

构造器
构造器和说明
`JobClient()` Create a job client.
`JobClient(InetSocketAddress jobTrackAddr, Configuration conf)` Build a job client, connect to the indicated job tracker.
`JobClient(JobConf conf)` Build a job client with the given `JobConf`, and connect to the default `JobTracker`.

方法概要

方法
限定符和类型	方法和说明
`void`	`cancelDelegationToken(Token<DelegationTokenIdentifier> token)` Cancel a delegation token from the JobTracker
`void`	`close()` Close the `JobClient`.
`void`	`displayTasks(JobID jobId, String type, String state)` Display the information about a job's tasks, of a particular type and in a particular state
`JobStatus[]`	`getAllJobs()` Get the jobs that are submitted.
`TaskReport[]`	`getCleanupTaskReports(JobID jobId)` Get the information of the current state of the cleanup tasks of a job.
`ClusterStatus`	`getClusterStatus()` Get status information about the Map-Reduce cluster.
`ClusterStatus`	`getClusterStatus(boolean detailed)` Get status information about the Map-Reduce cluster.
`int`	`getDefaultMaps()` Get status information about the max available Maps in the cluster.
`int`	`getDefaultReduces()` Get status information about the max available Reduces in the cluster.
`Token<DelegationTokenIdentifier>`	`getDelegationToken(Text renewer)`
`FileSystem`	`getFs()` Get a filesystem handle.
`RunningJob`	`getJob(JobID jobid)` Get an `RunningJob` object to track an ongoing job.
`RunningJob`	`getJob(String jobid)` 已过时。 Applications should rather use `getJob(JobID)`.
`JobStatus[]`	`getJobsFromQueue(String queueName)` Gets all the jobs which were added to particular Job Queue
`TaskReport[]`	`getMapTaskReports(JobID jobId)` Get the information of the current state of the map tasks of a job.
`TaskReport[]`	`getMapTaskReports(String jobId)` 已过时。 Applications should rather use `getMapTaskReports(JobID)`
`QueueAclsInfo[]`	`getQueueAclsForCurrentUser()` Gets the Queue ACLs for current user
`JobQueueInfo`	`getQueueInfo(String queueName)` Gets the queue information associated to a particular Job Queue
`JobQueueInfo[]`	`getQueues()` Return an array of queue information objects about all the Job Queues configured.
`TaskReport[]`	`getReduceTaskReports(JobID jobId)` Get the information of the current state of the reduce tasks of a job.
`TaskReport[]`	`getReduceTaskReports(String jobId)` 已过时。 Applications should rather use `getReduceTaskReports(JobID)`
`TaskReport[]`	`getSetupTaskReports(JobID jobId)` Get the information of the current state of the setup tasks of a job.
`Path`	`getStagingAreaDir()` Grab the jobtracker's view of the staging directory path where job-specific files will be placed.
`Path`	`getSystemDir()` Grab the jobtracker system directory path where job-specific files are to be placed.
`JobClient.TaskStatusFilter`	`getTaskOutputFilter()` 已过时。
`static JobClient.TaskStatusFilter`	`getTaskOutputFilter(JobConf job)` Get the task output filter out of the JobConf.
`void`	`init(JobConf conf)` Connect to the default `JobTracker`.
`static boolean`	`isJobDirValid(Path jobDirPath, FileSystem fs)` Checks if the job directory is clean and has all the required components for (re) starting the job
`JobStatus[]`	`jobsToComplete()` Get the jobs that are not completed and not failed.
`static void`	`main(String[] argv)`
`boolean`	`monitorAndPrintJob(JobConf conf, RunningJob job)` Monitor a job and print status in real-time as progress is made and tasks fail.
`long`	`renewDelegationToken(Token<DelegationTokenIdentifier> token)` Renew a delegation token
`int`	`run(String[] argv)` Execute the command with the given arguments.
`static RunningJob`	`runJob(JobConf job)` Utility that submits a job, then polls for progress until the job is complete.
`void`	`setTaskOutputFilter(JobClient.TaskStatusFilter newValue)` 已过时。
`static void`	`setTaskOutputFilter(JobConf job, JobClient.TaskStatusFilter newValue)` Modify the JobConf to set the task output filter.
`RunningJob`	`submitJob(JobConf job)` Submit a job to the MR system.
`RunningJob`	`submitJob(String jobFile)` Submit a job to the MR system.
`RunningJob`	`submitJobInternal(JobConf job)` Internal method for submitting jobs to the system.

从类继承的方法 org.apache.hadoop.conf.Configured
getConf, setConf

从类继承的方法 java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.hadoop.conf.Configurable
getConf, setConf

- 字段详细资料
  - MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_KEY
```
public static final String MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_KEY
```
    另请参阅:
    常量字段值
  - MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_DEFAULT
```
public static final boolean MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_DEFAULT
```
    另请参阅:
    常量字段值
  - MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_KEY
```
public static final String MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_KEY
```
    另请参阅:
    常量字段值
  - MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_DEFAULT
```
public static final String MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_DEFAULT
```
    另请参阅:
    常量字段值
  - HEARTBEAT_INTERVAL_MIN
```
public static final int HEARTBEAT_INTERVAL_MIN
```
    另请参阅:
    常量字段值
  - COUNTER_UPDATE_INTERVAL
```
public static final long COUNTER_UPDATE_INTERVAL
```
    另请参阅:
    常量字段值
  - DEFAULT_DISK_HEALTH_CHECK_INTERVAL
```
public static final long DEFAULT_DISK_HEALTH_CHECK_INTERVAL
```
    How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
    
    另请参阅:
    常量字段值
  - SUCCESS
```
public static final int SUCCESS
```
    另请参阅:
    常量字段值
  - FILE_NOT_FOUND
```
public static final int FILE_NOT_FOUND
```
    另请参阅:
    常量字段值
  - MAP_OUTPUT_LENGTH
```
public static final String MAP_OUTPUT_LENGTH
```
    The custom http header used for the map output length.
    
    另请参阅:
    常量字段值
  - RAW_MAP_OUTPUT_LENGTH
```
public static final String RAW_MAP_OUTPUT_LENGTH
```
    The custom http header used for the "raw" map output length.
    
    另请参阅:
    常量字段值
  - FROM_MAP_TASK
```
public static final String FROM_MAP_TASK
```
    The map task from which the map output data is being transferred
    
    另请参阅:
    常量字段值
  - FOR_REDUCE_TASK
```
public static final String FOR_REDUCE_TASK
```
    The reduce task number for which this map output is being transferred
    
    另请参阅:
    常量字段值
  - WORKDIR
```
public static final String WORKDIR
```
    另请参阅:
    常量字段值
- 构造器详细资料
  - JobClient
```
public JobClient()
```
    Create a job client.
  - JobClient
```
public JobClient(JobConf conf)
          throws IOException
```
    Build a job client with the given JobConf, and connect to the default JobTracker.
    
    参数:
    conf - the job configuration.
    
    抛出:
    
    IOException
  - JobClient
```
public JobClient(InetSocketAddress jobTrackAddr,
         Configuration conf)
          throws IOException
```
    Build a job client, connect to the indicated job tracker.
    
    参数:
    jobTrackAddr - the job tracker to connect to.
    conf - configuration.
    
    抛出:
    
    IOException
- 方法详细资料
  - init
```
public void init(JobConf conf)
          throws IOException
```
    Connect to the default JobTracker.
    
    参数:
    conf - the job configuration.
    
    抛出:
    
    IOException
  - close
```
public void close()
           throws IOException
```
    Close the JobClient.
    
    抛出:
    
    IOException
  - getFs
```
public FileSystem getFs()
                 throws IOException
```
    Get a filesystem handle. We need this to prepare jobs for submission to the MapReduce system.
    
    返回:
    the filesystem handle.
    
    抛出:
    
    IOException
  - submitJob
```
public RunningJob submitJob(String jobFile)
                     throws FileNotFoundException,
                            InvalidJobConfException,
                            IOException
```
    Submit a job to the MR system. This returns a handle to the RunningJob which can be used to track the running-job.
    
    参数:
    jobFile - the job configuration.
    
    返回:
    a handle to the RunningJob which can be used to track the running-job.
    
    抛出:
    
    FileNotFoundException
    
    InvalidJobConfException
    
    IOException
  - submitJob
```
public RunningJob submitJob(JobConf job)
                     throws FileNotFoundException,
                            IOException
```
    Submit a job to the MR system. This returns a handle to the RunningJob which can be used to track the running-job.
    
    参数:
    job - the job configuration.
    
    返回:
    a handle to the RunningJob which can be used to track the running-job.
    
    抛出:
    
    FileNotFoundException
    
    IOException
  - submitJobInternal
```
public RunningJob submitJobInternal(JobConf job)
                             throws FileNotFoundException,
                                    ClassNotFoundException,
                                    InterruptedException,
                                    IOException
```
    Internal method for submitting jobs to the system.
    
    参数:
    job - the configuration to submit
    
    返回:
    a proxy object for the running job
    
    抛出:
    
    FileNotFoundException
    
    ClassNotFoundException
    
    InterruptedException
    
    IOException
  - isJobDirValid
```
public static boolean isJobDirValid(Path jobDirPath,
                    FileSystem fs)
                             throws IOException
```
    Checks if the job directory is clean and has all the required components for (re) starting the job
    
    抛出:
    
    IOException
  - getJob
```
public RunningJob getJob(JobID jobid)
                  throws IOException
```
    Get an RunningJob object to track an ongoing job. Returns null if the id does not correspond to any known job.
    
    参数:
    jobid - the jobid of the job.
    
    返回:
    the RunningJob handle to track the job, null if the jobid doesn't correspond to any known job.
    
    抛出:
    
    IOException
  - getJob
```
@Deprecated
public RunningJob getJob(String jobid)
                  throws IOException
```
    已过时。 Applications should rather use getJob(JobID).
    
    抛出:
    
    IOException
  - getMapTaskReports
```
public TaskReport[] getMapTaskReports(JobID jobId)
                               throws IOException
```
    Get the information of the current state of the map tasks of a job.
    
    参数:
    jobId - the job to query.
    
    返回:
    the list of all of the map tips.
    
    抛出:
    
    IOException
  - getMapTaskReports
```
@Deprecated
public TaskReport[] getMapTaskReports(String jobId)
                               throws IOException
```
    已过时。 Applications should rather use getMapTaskReports(JobID)
    
    抛出:
    
    IOException
  - getReduceTaskReports
```
public TaskReport[] getReduceTaskReports(JobID jobId)
                                  throws IOException
```
    Get the information of the current state of the reduce tasks of a job.
    
    参数:
    jobId - the job to query.
    
    返回:
    the list of all of the reduce tips.
    
    抛出:
    
    IOException
  - getCleanupTaskReports
```
public TaskReport[] getCleanupTaskReports(JobID jobId)
                                   throws IOException
```
    Get the information of the current state of the cleanup tasks of a job.
    
    参数:
    jobId - the job to query.
    
    返回:
    the list of all of the cleanup tips.
    
    抛出:
    
    IOException
  - getSetupTaskReports
```
public TaskReport[] getSetupTaskReports(JobID jobId)
                                 throws IOException
```
    Get the information of the current state of the setup tasks of a job.
    
    参数:
    jobId - the job to query.
    
    返回:
    the list of all of the setup tips.
    
    抛出:
    
    IOException
  - getReduceTaskReports
```
@Deprecated
public TaskReport[] getReduceTaskReports(String jobId)
                                  throws IOException
```
    已过时。 Applications should rather use getReduceTaskReports(JobID)
    
    抛出:
    
    IOException
  - displayTasks
```
public void displayTasks(JobID jobId,
                String type,
                String state)
                  throws IOException
```
    Display the information about a job's tasks, of a particular type and in a particular state
    
    参数:
    jobId - the ID of the job
    type - the type of the task (map/reduce/setup/cleanup)
    state - the state of the task (pending/running/completed/failed/killed)
    
    抛出:
    
    IOException
  - getClusterStatus
```
public ClusterStatus getClusterStatus()
                               throws IOException
```
    Get status information about the Map-Reduce cluster.
    
    返回:
    the status information about the Map-Reduce cluster as an object of ClusterStatus.
    
    抛出:
    
    IOException
  - getClusterStatus
```
public ClusterStatus getClusterStatus(boolean detailed)
                               throws IOException
```
    Get status information about the Map-Reduce cluster.
    
    参数:
    detailed - if true then get a detailed status including the tracker names and memory usage of the JobTracker
    
    返回:
    the status information about the Map-Reduce cluster as an object of ClusterStatus.
    
    抛出:
    
    IOException
  - getStagingAreaDir
```
public Path getStagingAreaDir()
                       throws IOException
```
    Grab the jobtracker's view of the staging directory path where job-specific files will be placed.
    
    返回:
    the staging directory where job-specific files are to be placed.
    
    抛出:
    
    IOException
  - jobsToComplete
```
public JobStatus[] jobsToComplete()
                           throws IOException
```
    Get the jobs that are not completed and not failed.
    
    返回:
    array of JobStatus for the running/to-be-run jobs.
    
    抛出:
    
    IOException
  - getAllJobs
```
public JobStatus[] getAllJobs()
                       throws IOException
```
    Get the jobs that are submitted.
    
    返回:
    array of JobStatus for the submitted jobs.
    
    抛出:
    
    IOException
  - runJob
```
public static RunningJob runJob(JobConf job)
                         throws IOException
```
    Utility that submits a job, then polls for progress until the job is complete.
    
    参数:
    job - the job configuration.
    
    抛出:
    
    IOException - if the job fails
  - monitorAndPrintJob
```
public boolean monitorAndPrintJob(JobConf conf,
                         RunningJob job)
                           throws IOException,
                                  InterruptedException
```
    Monitor a job and print status in real-time as progress is made and tasks fail.
    
    参数:
    conf - the job's configuration
    job - the job to track
    
    返回:
    true if the job succeeded
    
    抛出:
    
    IOException - if communication to the JobTracker fails
    
    InterruptedException
  - setTaskOutputFilter
```
@Deprecated
public void setTaskOutputFilter(JobClient.TaskStatusFilter newValue)
```
    已过时。
    
    Sets the output filter for tasks. only those tasks are printed whose output matches the filter.
    
    参数:
    newValue - task filter.
  - getTaskOutputFilter
```
public static JobClient.TaskStatusFilter getTaskOutputFilter(JobConf job)
```
    Get the task output filter out of the JobConf.
    
    参数:
    job - the JobConf to examine.
    
    返回:
    the filter level.
  - setTaskOutputFilter
```
public static void setTaskOutputFilter(JobConf job,
                       JobClient.TaskStatusFilter newValue)
```
    Modify the JobConf to set the task output filter.
    
    参数:
    job - the JobConf to modify.
    newValue - the value to set.
  - getTaskOutputFilter
```
@Deprecated
public JobClient.TaskStatusFilter getTaskOutputFilter()
```
    已过时。
    
    Returns task output filter.
    
    返回:
    task filter.
  - run
```
public int run(String[] argv)
        throws Exception
```
    从接口复制的说明: Tool
    
    Execute the command with the given arguments.
    
    指定者:
    
    run 在接口中 Tool
    
    参数:
    argv - command specific arguments.
    
    返回:
    exit code.
    
    抛出:
    
    Exception
  - getDefaultMaps
```
public int getDefaultMaps()
                   throws IOException
```
    Get status information about the max available Maps in the cluster.
    
    返回:
    the max available Maps in the cluster
    
    抛出:
    
    IOException
  - getDefaultReduces
```
public int getDefaultReduces()
                      throws IOException
```
    Get status information about the max available Reduces in the cluster.
    
    返回:
    the max available Reduces in the cluster
    
    抛出:
    
    IOException
  - getSystemDir
```
public Path getSystemDir()
```
    Grab the jobtracker system directory path where job-specific files are to be placed.
    
    返回:
    the system directory where job-specific files are to be placed.
  - getQueues
```
public JobQueueInfo[] getQueues()
                         throws IOException
```
    Return an array of queue information objects about all the Job Queues configured.
    
    返回:
    Array of JobQueueInfo objects
    
    抛出:
    
    IOException
  - getJobsFromQueue
```
public JobStatus[] getJobsFromQueue(String queueName)
                             throws IOException
```
    Gets all the jobs which were added to particular Job Queue
    
    参数:
    queueName - name of the Job Queue
    
    返回:
    Array of jobs present in the job queue
    
    抛出:
    
    IOException
  - getQueueInfo
```
public JobQueueInfo getQueueInfo(String queueName)
                          throws IOException
```
    Gets the queue information associated to a particular Job Queue
    
    参数:
    queueName - name of the job queue.
    
    返回:
    Queue information associated to particular queue.
    
    抛出:
    
    IOException
  - getQueueAclsForCurrentUser
```
public QueueAclsInfo[] getQueueAclsForCurrentUser()
                                           throws IOException
```
    Gets the Queue ACLs for current user
    
    返回:
    array of QueueAclsInfo object for current user.
    
    抛出:
    
    IOException
  - getDelegationToken
```
public Token<DelegationTokenIdentifier> getDelegationToken(Text renewer)
                                                    throws IOException,
                                                           InterruptedException
```
    抛出:
    
    IOException
    
    InterruptedException
  - renewDelegationToken
```
public long renewDelegationToken(Token<DelegationTokenIdentifier> token)
                          throws SecretManager.InvalidToken,
                                 IOException,
                                 InterruptedException
```
    Renew a delegation token
    
    参数:
    token - the token to renew
    
    返回:
    the new expiration time
    
    抛出:
    
    SecretManager.InvalidToken
    
    IOException
    
    InterruptedException
  - cancelDelegationToken
```
public void cancelDelegationToken(Token<DelegationTokenIdentifier> token)
                           throws IOException,
                                  InterruptedException
```
    Cancel a delegation token from the JobTracker
    
    参数:
    token - the token to cancel
    
    抛出:
    
    IOException
    
    InterruptedException
  - main
```
public static void main(String[] argv)
                 throws Exception
```
    抛出:
    
    Exception

类 JobClient

Job Control

嵌套类概要

字段概要

构造器概要

方法概要

从类继承的方法 org.apache.hadoop.conf.Configured

从类继承的方法 java.lang.Object

从接口继承的方法 org.apache.hadoop.conf.Configurable

字段详细资料

MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_KEY

MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_DEFAULT

MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_KEY

MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_DEFAULT

HEARTBEAT_INTERVAL_MIN

COUNTER_UPDATE_INTERVAL

DEFAULT_DISK_HEALTH_CHECK_INTERVAL

SUCCESS

FILE_NOT_FOUND

MAP_OUTPUT_LENGTH

RAW_MAP_OUTPUT_LENGTH

FROM_MAP_TASK

FOR_REDUCE_TASK

WORKDIR

构造器详细资料

JobClient

JobClient

JobClient

方法详细资料

init

close

getFs

submitJob

submitJob

submitJobInternal

isJobDirValid

getJob

getJob

getMapTaskReports

getMapTaskReports

getReduceTaskReports

getCleanupTaskReports

getSetupTaskReports

getReduceTaskReports

displayTasks

getClusterStatus

getClusterStatus

getStagingAreaDir

jobsToComplete

getAllJobs

runJob

monitorAndPrintJob

setTaskOutputFilter

getTaskOutputFilter

setTaskOutputFilter

getTaskOutputFilter

run

getDefaultMaps

getDefaultReduces

getSystemDir

getQueues

getJobsFromQueue

getQueueInfo

getQueueAclsForCurrentUser

getDelegationToken

renewDelegationToken

cancelDelegationToken

main