org.apache.hadoop.mapreduce.lib.db

类 DataDrivenDBInputFormat<T extends DBWritable>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<LongWritable,T>

org.apache.hadoop.mapreduce.lib.db.DBInputFormat<T>

org.apache.hadoop.mapreduce.lib.db.DataDrivenDBInputFormat<T>

所有已实现的接口:

Configurable

直接已知子类:

OracleDataDrivenDBInputFormat

@InterfaceAudience.Public
@InterfaceStability.Evolving
public class DataDrivenDBInputFormat<T extends DBWritable>
extends DBInputFormat<T>
implements Configurable

A InputFormat that reads input data from an SQL table. Operates like DBInputFormat, but instead of using LIMIT and OFFSET to demarcate splits, it tries to generate WHERE clauses which separate the data into roughly equivalent shards.

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	DataDrivenDBInputFormat.DataDrivenDBInputSplit A InputSplit that spans a set of rows

从类继承的嵌套类/接口 org.apache.hadoop.mapreduce.lib.db.DBInputFormat

DBInputFormat.DBInputSplit, DBInputFormat.NullDBWritable

字段概要

字段

限定符和类型	字段和说明
static String	SUBSTITUTE_TOKEN If users are providing their own query, the following string is expected to appear in the WHERE clause, which will be substituted with a pair of conditions on the input to allow input splits to parallelise the import.

构造器概要

构造器

构造器和说明
DataDrivenDBInputFormat()

方法概要

方法

限定符和类型	方法和说明
protected RecordReader<LongWritable,T>	createDBRecordReader(DBInputFormat.DBInputSplit split, Configuration conf)
protected String	getBoundingValsQuery()
List<InputSplit>	getSplits(JobContext job) Logically split the set of input files for the job.
protected DBSplitter	getSplitter(int sqlDataType)
static void	setBoundingQuery(Configuration conf, String query) Set the user-defined bounding query to use with a user-defined query.
static void	setInput(Job job, Class<? extends DBWritable> inputClass, String inputQuery, String inputBoundingQuery) setInput() takes a custom query and a separate "bounding query" to use instead of the custom "count query" used by DBInputFormat.
static void	setInput(Job job, Class<? extends DBWritable> inputClass, String tableName, String conditions, String splitBy, String... fieldNames) Note that the "orderBy" column is called the "splitBy" in this version.

从类继承的方法 org.apache.hadoop.mapreduce.lib.db.DBInputFormat

closeConnection, createRecordReader, getConf, getConnection, getCountQuery, getDBConf, getDBProductName, setConf

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.hadoop.conf.Configurable

getConf, setConf

字段详细资料

SUBSTITUTE_TOKEN

public static final String SUBSTITUTE_TOKEN

If users are providing their own query, the following string is expected to appear in the WHERE clause, which will be substituted with a pair of conditions on the input to allow input splits to parallelise the import.

另请参阅:

常量字段值

构造器详细资料

DataDrivenDBInputFormat

public DataDrivenDBInputFormat()

方法详细资料

getSplitter

protected DBSplitter getSplitter(int sqlDataType)

返回:

the DBSplitter implementation to use to divide the table/query into InputSplits.

getSplits

public List<InputSplit> getSplits(JobContext job)
                           throws IOException

Logically split the set of input files for the job.

Each InputSplit is then assigned to an individual Mapper for processing.

Note: The split is a logical split of the inputs and the input files are not physically split into chunks. For e.g. a split could be <input-file-path, start, offset> tuple. The InputFormat also creates the RecordReader to read the InputSplit.

覆盖:

getSplits 在类中 DBInputFormat<T extends DBWritable>

参数:

job - job configuration.

返回:

an array of InputSplits for the job.

抛出:

IOException

getBoundingValsQuery

protected String getBoundingValsQuery()

返回:

a query which returns the minimum and maximum values for the order-by column. The min value should be in the first column, and the max value should be in the second column of the results.

setBoundingQuery

public static void setBoundingQuery(Configuration conf,
                    String query)

Set the user-defined bounding query to use with a user-defined query. This *must* include the substring "$CONDITIONS" (DataDrivenDBInputFormat.SUBSTITUTE_TOKEN) inside the WHERE clause, so that DataDrivenDBInputFormat knows where to insert split clauses. e.g., "SELECT foo FROM mytable WHERE $CONDITIONS" This will be expanded to something like: SELECT foo FROM mytable WHERE (id > 100) AND (id < 250) inside each split.

createDBRecordReader

protected RecordReader<LongWritable,T> createDBRecordReader(DBInputFormat.DBInputSplit split,
                                                Configuration conf)
                                                                        throws IOException

覆盖:

createDBRecordReader 在类中 DBInputFormat<T extends DBWritable>

抛出:

IOException

setInput

public static void setInput(Job job,
            Class<? extends DBWritable> inputClass,
            String tableName,
            String conditions,
            String splitBy,
            String... fieldNames)

Note that the "orderBy" column is called the "splitBy" in this version. We reuse the same field, but it's not strictly ordering it -- just partitioning the results.

setInput

public static void setInput(Job job,
            Class<? extends DBWritable> inputClass,
            String inputQuery,
            String inputBoundingQuery)

setInput() takes a custom query and a separate "bounding query" to use instead of the custom "count query" used by DBInputFormat.