org.apache.hive.hcatalog.streaming

Class DelimitedInputWriter

java.lang.Object

org.apache.hive.hcatalog.streaming.DelimitedInputWriter

All Implemented Interfaces:

RecordWriter

public class DelimitedInputWriter
extends Object

Streaming Writer handles delimited input (eg. CSV). Delimited input is parsed & reordered to match column order in table Uses Lazy Simple Serde to process delimited input

Constructor Summary

Constructors

Constructor and Description
DelimitedInputWriter(String[] colNamesForFields, String delimiter, HiveEndPoint endPoint) Constructor.
DelimitedInputWriter(String[] colNamesForFields, String delimiter, HiveEndPoint endPoint, HiveConf conf) Constructor.
DelimitedInputWriter(String[] colNamesForFields, String delimiter, HiveEndPoint endPoint, HiveConf conf, char serdeSeparator) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	clear() Clear bufferred writes.
void	closeBatch() Close the RecordUpdater.
protected LazySimpleSerDe	createSerde(Table tbl, HiveConf conf) Creates LazySimpleSerde
void	flush() Flush records from buffer.
char	getSerdeSeparator()
protected ArrayList<String>	getTableColumns()
void	newBatch(Long minTxnId, Long maxTxnID) Creates a new record updater for the new batch
protected byte[]	reorderFields(byte[] record)
void	write(long transactionId, byte[] record) Writes using a hive RecordUpdater

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DelimitedInputWriter

public DelimitedInputWriter(String[] colNamesForFields,
                    String delimiter,
                    HiveEndPoint endPoint)
                     throws ClassNotFoundException,
                            ConnectionError,
                            SerializationError,
                            InvalidColumn,
                            StreamingException

Constructor. Uses default separator of the LazySimpleSerde

Parameters:

colNamesForFields - Column name assignment for input fields. nulls or empty strings in the array indicates the fields to be skipped

delimiter - input field delimiter

endPoint - Hive endpoint

Throws:

ConnectionError - Problem talking to Hive

ClassNotFoundException - Serde class not found

SerializationError - Serde initialization/interaction failed

StreamingException - Problem acquiring file system path for partition

InvalidColumn - any element in colNamesForFields refers to a non existing column

DelimitedInputWriter

public DelimitedInputWriter(String[] colNamesForFields,
                    String delimiter,
                    HiveEndPoint endPoint,
                    HiveConf conf)
                     throws ClassNotFoundException,
                            ConnectionError,
                            SerializationError,
                            InvalidColumn,
                            StreamingException

Constructor. Uses default separator of the LazySimpleSerde

Parameters:

colNamesForFields - Column name assignment for input fields. nulls or empty strings in the array indicates the fields to be skipped

delimiter - input field delimiter

endPoint - Hive endpoint

conf - a Hive conf object. Can be null if not using advanced hive settings.

Throws:

ConnectionError - Problem talking to Hive

ClassNotFoundException - Serde class not found

SerializationError - Serde initialization/interaction failed

StreamingException - Problem acquiring file system path for partition

InvalidColumn - any element in colNamesForFields refers to a non existing column

DelimitedInputWriter

public DelimitedInputWriter(String[] colNamesForFields,
                    String delimiter,
                    HiveEndPoint endPoint,
                    HiveConf conf,
                    char serdeSeparator)
                     throws ClassNotFoundException,
                            ConnectionError,
                            SerializationError,
                            InvalidColumn,
                            StreamingException

Constructor. Allows overriding separator of the LazySimpleSerde

Parameters:

colNamesForFields - Column name assignment for input fields

delimiter - input field delimiter

endPoint - Hive endpoint

conf - a Hive conf object. Set to null if not using advanced hive settings.

serdeSeparator - separator used when encoding data that is fed into the LazySimpleSerde. Ensure this separator does not occur in the field data

Throws:

ConnectionError - Problem talking to Hive

ClassNotFoundException - Serde class not found

SerializationError - Serde initialization/interaction failed

StreamingException - Problem acquiring file system path for partition

InvalidColumn - any element in colNamesForFields refers to a non existing column

Method Detail

reorderFields

protected byte[] reorderFields(byte[] record)
                        throws UnsupportedEncodingException

Throws:

UnsupportedEncodingException

getTableColumns

protected ArrayList<String> getTableColumns()

write

public void write(long transactionId,
         byte[] record)
           throws SerializationError,
                  StreamingIOFailure

Description copied from interface: RecordWriter

Writes using a hive RecordUpdater

Parameters:

transactionId - the ID of the Txn in which the write occurs

record - the record to be written

Throws:

SerializationError

StreamingIOFailure

createSerde

protected LazySimpleSerDe createSerde(Table tbl,
                          HiveConf conf)
                               throws SerializationError

Creates LazySimpleSerde

Parameters:

tbl -

Returns:

Throws:

SerializationError - if serde could not be initialized

getSerdeSeparator

public char getSerdeSeparator()

flush

public void flush()
           throws StreamingIOFailure

Description copied from interface: RecordWriter

Flush records from buffer. Invoked by TransactionBatch.commit()

Specified by:

flush in interface RecordWriter

Throws:

StreamingIOFailure

clear

public void clear()
           throws StreamingIOFailure

Description copied from interface: RecordWriter

Clear bufferred writes. Invoked by TransactionBatch.abort()

Specified by:

clear in interface RecordWriter

Throws:

StreamingIOFailure

newBatch

public void newBatch(Long minTxnId,
            Long maxTxnID)
              throws StreamingIOFailure,
                     SerializationError

Creates a new record updater for the new batch

Specified by:

newBatch in interface RecordWriter

Parameters:

minTxnId - smallest Txnid in the batch

maxTxnID - largest Txnid in the batch

Throws:

StreamingIOFailure - if failed to create record updater

SerializationError

closeBatch

public void closeBatch()
                throws StreamingIOFailure

Description copied from interface: RecordWriter

Close the RecordUpdater. Invoked by TransactionBatch.close()

Specified by:

closeBatch in interface RecordWriter

Throws:

StreamingIOFailure