org.apache.hadoop.mapred.lib

类 NLineInputFormat

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<LongWritable,Text>

org.apache.hadoop.mapred.lib.NLineInputFormat

所有已实现的接口:

InputFormat<LongWritable,Text>, JobConfigurable

public class NLineInputFormat
extends FileInputFormat<LongWritable,Text>
implements JobConfigurable

NLineInputFormat which splits N lines of input as one split. In many "pleasantly" parallel applications, each process/mapper processes the same input file (s), but with computations are controlled by different parameters.(Referred to as "parameter sweeps"). One way to achieve this, is to specify a set of parameters (one set per line) as input in a control file (which is the input path to the map-reduce application, where as the input dataset is specified via a config variable in JobConf.). The NLineInputFormat can be used in such applications, that splits the input file such that by default, one line is fed as a value to one map task, and key is the offset. i.e. (k,v) is (LongWritable, Text). The location hints will span the whole mapred cluster.

嵌套类概要

从类继承的嵌套类/接口 org.apache.hadoop.mapred.FileInputFormat

FileInputFormat.Counter

字段概要

从类继承的字段 org.apache.hadoop.mapred.FileInputFormat

LOG

构造器概要

构造器

构造器和说明
NLineInputFormat()

方法概要

方法

限定符和类型	方法和说明
void	configure(JobConf conf) Initializes a new instance from a JobConf.
protected static FileSplit	createFileSplit(Path fileName, long begin, long length) NLineInputFormat uses LineRecordReader, which always reads (and consumes) at least one character out of its upper split boundary.
RecordReader<LongWritable,Text>	getRecordReader(InputSplit genericSplit, JobConf job, Reporter reporter) Get the RecordReader for the given InputSplit.
InputSplit[]	getSplits(JobConf job, int numSplits) Logically splits the set of input files for the job, splits N lines of the input as one split.

从类继承的方法 org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, isSplitable, listStatus, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

NLineInputFormat

public NLineInputFormat()

方法详细资料

getRecordReader

public RecordReader<LongWritable,Text> getRecordReader(InputSplit genericSplit,
                                              JobConf job,
                                              Reporter reporter)
                                                throws IOException

从接口复制的说明: InputFormat

Get the RecordReader for the given InputSplit.

It is the responsibility of the RecordReader to respect record boundaries while processing the logical split to present a record-oriented view to the individual task.

指定者:

getRecordReader 在接口中 InputFormat<LongWritable,Text>

指定者:

getRecordReader 在类中 FileInputFormat<LongWritable,Text>

参数:

genericSplit - the InputSplit

job - the job that this split belongs to

返回:

a RecordReader

抛出:

IOException

getSplits

public InputSplit[] getSplits(JobConf job,
                     int numSplits)
                       throws IOException

Logically splits the set of input files for the job, splits N lines of the input as one split.

指定者:

getSplits 在接口中 InputFormat<LongWritable,Text>

覆盖:

getSplits 在类中 FileInputFormat<LongWritable,Text>

参数:

job - job configuration.

numSplits - the desired number of splits, a hint.

返回:

an array of InputSplits for the job.

抛出:

IOException

另请参阅:

FileInputFormat.getSplits(JobConf, int)

createFileSplit

protected static FileSplit createFileSplit(Path fileName,
                        long begin,
                        long length)

NLineInputFormat uses LineRecordReader, which always reads (and consumes) at least one character out of its upper split boundary. So to make sure that each mapper gets N lines, we move back the upper split limits of each split by one character here.

参数:

fileName - Path of file

begin - the position of the first byte in the file to process

length - number of bytes in InputSplit

返回:

FileSplit

configure

public void configure(JobConf conf)

从接口复制的说明: JobConfigurable

Initializes a new instance from a JobConf.

指定者:

configure 在接口中 JobConfigurable

参数:

conf - the configuration