org.apache.hadoop.hbase.io.hfile

Class AbstractHFileWriter

java.lang.Object

org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter

All Implemented Interfaces:

Closeable, AutoCloseable, HFile.Writer

Direct Known Subclasses:

HFileWriterV2

@InterfaceAudience.Private
public abstract class AbstractHFileWriter
extends Object
implements HFile.Writer

Common functionality needed by all versions of HFile writers.

Field Summary

Fields

Modifier and Type	Field and Description
protected HFileDataBlockEncoder	blockEncoder The data block encoding which will be used.
protected CacheConfig	cacheConf Cache configuration for caching data on write.
protected boolean	closeOutputStream True if we opened the outputStream (and so will close it).
protected KeyValue.KVComparator	comparator Key comparator.
protected long	entryCount Total # of key/value entries, i.e.
protected HFile.FileInfo	fileInfo A "file info" block: a key-value map of file-wide metadata.
protected byte[]	firstKeyInBlock First key in a block.
protected HFileContext	hFileContext
protected byte[]	lastKeyBuffer Key previously appended.
protected int	lastKeyLength
protected int	lastKeyOffset
protected List<org.apache.hadoop.io.Writable>	metaData Writables representing meta block data.
protected List<byte[]>	metaNames Meta block names.
protected String	name Name for this object used when logging or in toString.
protected org.apache.hadoop.fs.FSDataOutputStream	outputStream FileSystem stream to write into.
protected org.apache.hadoop.fs.Path	path May be null if we were passed a stream.
protected long	totalKeyLength Used for calculating the average key length.
protected long	totalUncompressedBytes Total uncompressed bytes, maybe calculate a compression ratio later.
protected long	totalValueLength Used for calculating the average value length.

Constructor Summary

Constructors

Constructor and Description
AbstractHFileWriter(CacheConfig cacheConf, org.apache.hadoop.fs.FSDataOutputStream outputStream, org.apache.hadoop.fs.Path path, KeyValue.KVComparator comparator, HFileContext fileContext)

Method Summary

Methods

Modifier and Type	Method and Description
void	appendFileInfo(byte[] k, byte[] v) Add to the file info.
protected boolean	checkKey(byte[] key, int offset, int length) Checks that the given key does not violate the key order.
protected void	checkValue(byte[] value, int offset, int length) Checks the given value for validity.
static Compression.Algorithm	compressionByName(String algoName)
protected static org.apache.hadoop.fs.FSDataOutputStream	createOutputStream(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, InetSocketAddress[] favoredNodes) A helper method to create HFile output streams in constructors
protected void	finishClose(FixedFileTrailer trailer) Sets remaining trailer fields, writes the trailer to disk, and optionally closes the output stream.
protected void	finishFileInfo() Add last bits of metadata to file info before it is written out.
org.apache.hadoop.fs.Path	getPath()
String	toString()
protected void	writeFileInfo(FixedFileTrailer trailer, DataOutputStream out) Sets the file info offset in the trailer, finishes up populating fields in the file info, and writes the file info into the given data output.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.io.hfile.HFile.Writer

addDeleteFamilyBloomFilter, addGeneralBloomFilter, addInlineBlockWriter, append, append, append, appendMetaBlock, getFileContext

Methods inherited from interface java.io.Closeable

close

Field Detail

lastKeyBuffer

protected byte[] lastKeyBuffer

Key previously appended. Becomes the last key in the file.

lastKeyOffset

protected int lastKeyOffset

lastKeyLength

protected int lastKeyLength

outputStream

protected org.apache.hadoop.fs.FSDataOutputStream outputStream

FileSystem stream to write into.

closeOutputStream

protected final boolean closeOutputStream

True if we opened the outputStream (and so will close it).

fileInfo

protected HFile.FileInfo fileInfo

A "file info" block: a key-value map of file-wide metadata.

entryCount

protected long entryCount

Total # of key/value entries, i.e. how many times add() was called.

totalKeyLength

protected long totalKeyLength

Used for calculating the average key length.

totalValueLength

protected long totalValueLength

Used for calculating the average value length.

totalUncompressedBytes

protected long totalUncompressedBytes

Total uncompressed bytes, maybe calculate a compression ratio later.

comparator

protected final KeyValue.KVComparator comparator

Key comparator. Used to ensure we write in order.

metaNames

protected List<byte[]> metaNames

Meta block names.

metaData

protected List<org.apache.hadoop.io.Writable> metaData

Writables representing meta block data.

firstKeyInBlock

protected byte[] firstKeyInBlock

First key in a block.

path

protected final org.apache.hadoop.fs.Path path

May be null if we were passed a stream.

cacheConf

protected final CacheConfig cacheConf

Cache configuration for caching data on write.

name

protected final String name

Name for this object used when logging or in toString. Is either the result of a toString on stream or else name of passed file Path.

blockEncoder

protected final HFileDataBlockEncoder blockEncoder

The data block encoding which will be used. NoOpDataBlockEncoder.INSTANCE if there is no encoding.

hFileContext

protected final HFileContext hFileContext

Constructor Detail

AbstractHFileWriter

public AbstractHFileWriter(CacheConfig cacheConf,
                   org.apache.hadoop.fs.FSDataOutputStream outputStream,
                   org.apache.hadoop.fs.Path path,
                   KeyValue.KVComparator comparator,
                   HFileContext fileContext)

Method Detail

finishFileInfo

protected void finishFileInfo()
                       throws IOException

Add last bits of metadata to file info before it is written out.

Throws:

IOException

appendFileInfo

public void appendFileInfo(byte[] k,
                  byte[] v)
                    throws IOException

Add to the file info. All added key/value pairs can be obtained using HFile.Reader.loadFileInfo().

Specified by:

appendFileInfo in interface HFile.Writer

Parameters:

k - Key

v - Value

Throws:

IOException - in case the key or the value are invalid

writeFileInfo

protected final void writeFileInfo(FixedFileTrailer trailer,
                 DataOutputStream out)
                            throws IOException

Sets the file info offset in the trailer, finishes up populating fields in the file info, and writes the file info into the given data output. The reason the data output is not always outputStream is that we store file info as a block in version 2.

Parameters:

trailer - fixed file trailer

out - the data output to write the file info to

Throws:

IOException

checkKey

protected boolean checkKey(byte[] key,
               int offset,
               int length)
                    throws IOException

Checks that the given key does not violate the key order.

Parameters:

key - Key to check.

Returns:

true if the key is duplicate

Throws:

IOException - if the key or the key order is wrong

checkValue

protected void checkValue(byte[] value,
              int offset,
              int length)
                   throws IOException

Checks the given value for validity.

Throws:

IOException

getPath

public org.apache.hadoop.fs.Path getPath()

Specified by:

getPath in interface HFile.Writer

Returns:

Path or null if we were passed a stream rather than a Path.

toString

public String toString()

Overrides:

toString in class Object

finishClose

protected void finishClose(FixedFileTrailer trailer)
                    throws IOException

Sets remaining trailer fields, writes the trailer to disk, and optionally closes the output stream.

Throws:

IOException

compressionByName

public static Compression.Algorithm compressionByName(String algoName)

createOutputStream

protected static org.apache.hadoop.fs.FSDataOutputStream createOutputStream(org.apache.hadoop.conf.Configuration conf,
                                                         org.apache.hadoop.fs.FileSystem fs,
                                                         org.apache.hadoop.fs.Path path,
                                                         InetSocketAddress[] favoredNodes)
                                                                     throws IOException

A helper method to create HFile output streams in constructors

Throws:

IOException