org.apache.hadoop.examples.terasort

类 TeraInputFormat

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<Text,Text>

org.apache.hadoop.examples.terasort.TeraInputFormat

所有已实现的接口:

InputFormat<Text,Text>

public class TeraInputFormat
extends FileInputFormat<Text,Text>

An input format that reads the first 10 characters of each line as the key and the rest of the line as the value. Both key and value are represented as Text.

嵌套类概要

从类继承的嵌套类/接口 org.apache.hadoop.mapred.FileInputFormat

FileInputFormat.Counter

字段概要

从类继承的字段 org.apache.hadoop.mapred.FileInputFormat

LOG

构造器概要

构造器

构造器和说明
TeraInputFormat()

方法概要

方法

限定符和类型	方法和说明
RecordReader<Text,Text>	getRecordReader(InputSplit split, JobConf job, Reporter reporter) Get the RecordReader for the given InputSplit.
InputSplit[]	getSplits(JobConf conf, int splits) Splits files returned by FileInputFormat.listStatus(JobConf) when they're too big.
static void	writePartitionFile(JobConf conf, Path partFile) Use the input splits to take samples of the input and generate sample keys.

从类继承的方法 org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, isSplitable, listStatus, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

TeraInputFormat

public TeraInputFormat()

方法详细资料

writePartitionFile

public static void writePartitionFile(JobConf conf,
                      Path partFile)
                               throws IOException

Use the input splits to take samples of the input and generate sample keys. By default reads 100,000 keys from 10 locations in the input, sorts them and picks N-1 keys to generate N equally sized partitions.

参数:

conf - the job to sample

partFile - where to write the output file to

抛出:

IOException - if something goes wrong

getRecordReader

public RecordReader<Text,Text> getRecordReader(InputSplit split,
                                      JobConf job,
                                      Reporter reporter)
                                        throws IOException

从接口复制的说明: InputFormat

Get the RecordReader for the given InputSplit.

It is the responsibility of the RecordReader to respect record boundaries while processing the logical split to present a record-oriented view to the individual task.

指定者:

getRecordReader 在接口中 InputFormat<Text,Text>

指定者:

getRecordReader 在类中 FileInputFormat<Text,Text>

参数:

split - the InputSplit

job - the job that this split belongs to

返回:

a RecordReader

抛出:

IOException

getSplits

public InputSplit[] getSplits(JobConf conf,
                     int splits)
                       throws IOException

从类复制的说明: FileInputFormat

Splits files returned by FileInputFormat.listStatus(JobConf) when they're too big.

指定者:

getSplits 在接口中 InputFormat<Text,Text>

覆盖:

getSplits 在类中 FileInputFormat<Text,Text>

参数:

conf - job configuration.

splits - the desired number of splits, a hint.

返回:

an array of InputSplits for the job.

抛出:

IOException