When repairing a corrupted HBase, it is best to repair the lowest risk inconsistencies first. These are generally region consistency repairs -- localized single region repairs, that only modify in-memory data, ephemeral zookeeper data, or patch holes in the META table. Region consistency requires that the HBase instance has the state of the region’s data in HDFS (.regioninfo files), the region’s row in the hbase:meta table., and region’s deployment/assignments on region servers and the master in accordance. Options for repairing region consistency include:
-fixAssignments (equivalent to the 0.90 -fix option) repairs unassigned, incorrectly
assigned or multiply assigned regions.
-fixMeta which removes meta rows when corresponding regions are not present in
HDFS and adds new meta rows if they regions are present in HDFS while not in META.
To fix deployment and assignment problems you can run this command:
$ ./bin/hbase hbck -fixAssignments
To fix deployment and assignment problems as well as repairing incorrect meta rows you can run this command:
$ ./bin/hbase hbck -fixAssignments -fixMeta
There are a few classes of table integrity problems that are low risk repairs. The first two are degenerate (startkey == endkey) regions and backwards regions (startkey > endkey). These are automatically handled by sidelining the data to a temporary directory (/hbck/xxxx). The third low-risk class is hdfs region holes. This can be repaired by using the:
-fixHdfsHoles option for fabricating new empty regions on the file system.
If holes are detected you can use -fixHdfsHoles and should include -fixMeta and -fixAssignments to make the new region consistent.
$ ./bin/hbase hbck -fixAssignments -fixMeta -fixHdfsHoles
Since this is a common operation, we’ve added a the -repairHoles flag that is equivalent to the
previous command:
$ ./bin/hbase hbck -repairHoles
If inconsistencies still remain after these steps, you most likely have table integrity problems related to orphaned or overlapping regions.