org.apache.hadoop.tools.distcp2.mapred.lib

类 DynamicInputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.tools.distcp2.mapred.lib.DynamicInputFormat<K,V>

public class DynamicInputFormat<K,V>
extends InputFormat<K,V>

DynamicInputFormat implements the "Worker pattern" for DistCp. Rather than to split up the copy-list into a set of static splits, the DynamicInputFormat does the following: 1. Splits the copy-list into small chunks on the DFS. 2. Creates a set of empty "dynamic" splits, that each consume as many chunks as it can. This arrangement ensures that a single slow mapper won't slow down the entire job (since the slack will be picked up by other mappers, who consume more chunks.) By varying the split-ratio, one can vary chunk sizes to achieve different performance characteristics.

构造器概要

构造器

构造器和说明
DynamicInputFormat()

方法概要

方法

限定符和类型	方法和说明
RecordReader<K,V>	createRecordReader(InputSplit inputSplit, TaskAttemptContext taskAttemptContext) Implementation of Inputformat::createRecordReader().
List<InputSplit>	getSplits(JobContext jobContext) Implementation of InputFormat::getSplits().

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

DynamicInputFormat

public DynamicInputFormat()

方法详细资料

getSplits

public List<InputSplit> getSplits(JobContext jobContext)
                           throws IOException,
                                  InterruptedException

Implementation of InputFormat::getSplits(). This method splits up the copy-listing file into chunks, and assigns the first batch to different tasks.

指定者:

getSplits 在类中 InputFormat<K,V>

参数:

jobContext - JobContext for the map job.

返回:

The list of (empty) dynamic input-splits.

抛出:

IOException, - on failure.

InterruptedException

IOException

createRecordReader

public RecordReader<K,V> createRecordReader(InputSplit inputSplit,
                                   TaskAttemptContext taskAttemptContext)
                                     throws IOException,
                                            InterruptedException

Implementation of Inputformat::createRecordReader().

指定者:

createRecordReader 在类中 InputFormat<K,V>

参数:

inputSplit - The split for which the RecordReader is required.

taskAttemptContext - TaskAttemptContext for the current attempt.

返回:

DynamicRecordReader instance.

抛出:

IOException, - on failure.

InterruptedException

IOException