Table of Contents
- H.1. HBase File Format (version 1)
- H.2. HBase file format with inline blocks (version 2)
- H.2.1. Motivation
- H.2.2. Overview of Version 2
- H.2.3. Unified version 2 block format
- H.2.4. Block index in version 2
- H.2.5. Root block index format in version 2
- H.2.6. Non-root block index format in version 2
- H.2.7. Bloom filters in version 2
- H.2.8. File Info format in versions 1 and 2
- H.2.9. Fixed file trailer format differences between versions 1 and 2
- H.2.10. getShortMidpointKey(an optimization for data index block)
- H.3. HBase File Format with Security Enhancements (version 3)
This appendix describes the evolution of the HFile format.
As we will be discussing changes to the HFile format, it is useful to give a short overview of the original (HFile version 1) format.
An HFile in version 1 format is structured as follows:
The block index in version 1 is very straightforward. For each entry, it contains:
Offset (long)
Uncompressed size (int)
Key (a serialized byte array written using Bytes.writeByteArray)
Key length as a variable-length integer (VInt)
Key bytes
The number of entries in the block index is stored in the fixed file trailer, and has to be passed in to the method that reads the block index. One of the limitations of the block index in version 1 is that it does not provide the compressed size of a block, which turns out to be necessary for decompression. Therefore, the HFile reader has to infer this compressed size from the offset difference between blocks. We fix this limitation in version 2, where we store on-disk block size instead of uncompressed size, and get uncompressed size from the block header.