org.apache.hadoop.hive.metastore

Class Warehouse

java.lang.Object

org.apache.hadoop.hive.metastore.Warehouse

public class Warehouse
extends Object

This class represents a warehouse where data of Hive tables is stored

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	LOG

Constructor Summary

Constructors

Constructor and Description
Warehouse(org.apache.hadoop.conf.Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
static void	closeFs(org.apache.hadoop.fs.FileSystem fs)
boolean	deleteDir(org.apache.hadoop.fs.Path f, boolean recursive)
boolean	deleteDir(org.apache.hadoop.fs.Path f, boolean recursive, boolean ifPurge)
org.apache.hadoop.fs.Path	getDatabasePath(Database db)
org.apache.hadoop.fs.Path	getDefaultDatabasePath(String dbName)
org.apache.hadoop.fs.Path	getDnsPath(org.apache.hadoop.fs.Path path)
static org.apache.hadoop.fs.Path	getDnsPath(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Hadoop File System reverse lookups paths with raw ip addresses The File System URI always contains the canonical DNS name of the Namenode.
org.apache.hadoop.fs.FileStatus[]	getFileStatusesForLocation(String location)
org.apache.hadoop.fs.FileStatus[]	getFileStatusesForSD(StorageDescriptor desc)
org.apache.hadoop.fs.FileStatus[]	getFileStatusesForUnpartitionedTable(Database db, Table table)
org.apache.hadoop.fs.FileSystem	getFs(org.apache.hadoop.fs.Path f)
static org.apache.hadoop.fs.FileSystem	getFs(org.apache.hadoop.fs.Path f, org.apache.hadoop.conf.Configuration conf) Helper functions to convert IOException to MetaException
org.apache.hadoop.fs.Path	getPartitionPath(Database db, String tableName, LinkedHashMap<String,String> pm)
org.apache.hadoop.fs.Path	getPartitionPath(org.apache.hadoop.fs.Path tblPath, LinkedHashMap<String,String> pm)
static List<String>	getPartValuesFromPartName(String partName)
static String	getQualifiedName(Partition partition)
static String	getQualifiedName(Table table)
org.apache.hadoop.fs.Path	getTablePath(Database db, String tableName)
org.apache.hadoop.fs.Path	getTablePath(String whRootString, String tableName)
org.apache.hadoop.fs.Path	getWhRoot() Resolve the configured warehouse root dir with respect to the configuration This involves opening the FileSystem corresponding to the warehouse root dir (but that should be ok given that this is only called during DDL statements for non-external tables).
boolean	isDir(org.apache.hadoop.fs.Path f)
boolean	isEmpty(org.apache.hadoop.fs.Path path)
boolean	isWritable(org.apache.hadoop.fs.Path path)
static String	makeDynamicPartName(Map<String,String> spec) Given a dynamic partition specification, return the path corresponding to the static part of partition specification.
static Map<String,String>	makeEscSpecFromName(String name)
static String	makePartName(List<FieldSchema> partCols, List<String> vals)
static String	makePartName(List<FieldSchema> partCols, List<String> vals, String defaultStr) Makes a valid partition name.
static String	makePartName(Map<String,String> spec, boolean addTrailingSeperator) Makes a partition name from a specification
static String	makePartPath(Map<String,String> spec) Given a partition specification, return the path corresponding to the partition spec.
static void	makeSpecFromName(Map<String,String> partSpec, org.apache.hadoop.fs.Path currPath)
static LinkedHashMap<String,String>	makeSpecFromName(String name)
static Map<String,String>	makeSpecFromValues(List<FieldSchema> partCols, List<String> values)
static void	makeValsFromName(String name, AbstractList<String> result) Extracts values from partition name without the column names.
boolean	mkdirs(org.apache.hadoop.fs.Path f, boolean inheritPermCandidate)
boolean	renameDir(org.apache.hadoop.fs.Path sourcePath, org.apache.hadoop.fs.Path destPath)
boolean	renameDir(org.apache.hadoop.fs.Path sourcePath, org.apache.hadoop.fs.Path destPath, boolean inheritPerms)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

Constructor Detail

Warehouse

public Warehouse(org.apache.hadoop.conf.Configuration conf)
          throws MetaException

Throws:

MetaException

Method Detail

getFs

public static org.apache.hadoop.fs.FileSystem getFs(org.apache.hadoop.fs.Path f,
                                    org.apache.hadoop.conf.Configuration conf)
                                             throws MetaException

Helper functions to convert IOException to MetaException

Throws:

MetaException

getFs

public org.apache.hadoop.fs.FileSystem getFs(org.apache.hadoop.fs.Path f)
                                      throws MetaException

Throws:

MetaException

closeFs

public static void closeFs(org.apache.hadoop.fs.FileSystem fs)
                    throws MetaException

Throws:

MetaException

getDnsPath

public static org.apache.hadoop.fs.Path getDnsPath(org.apache.hadoop.fs.Path path,
                                   org.apache.hadoop.conf.Configuration conf)
                                            throws MetaException

Hadoop File System reverse lookups paths with raw ip addresses The File System URI always contains the canonical DNS name of the Namenode. Subsequently, operations on paths with raw ip addresses cause an exception since they don't match the file system URI. This routine solves this problem by replacing the scheme and authority of a path with the scheme and authority of the FileSystem that it maps to.

Parameters:

path - Path to be canonicalized

Returns:

Path with canonical scheme and authority

Throws:

MetaException

getDnsPath

public org.apache.hadoop.fs.Path getDnsPath(org.apache.hadoop.fs.Path path)
                                     throws MetaException

Throws:

MetaException

getWhRoot

public org.apache.hadoop.fs.Path getWhRoot()
                                    throws MetaException

Resolve the configured warehouse root dir with respect to the configuration This involves opening the FileSystem corresponding to the warehouse root dir (but that should be ok given that this is only called during DDL statements for non-external tables).

Throws:

MetaException

getTablePath

public org.apache.hadoop.fs.Path getTablePath(String whRootString,
                                     String tableName)
                                       throws MetaException

Throws:

MetaException

getDatabasePath

public org.apache.hadoop.fs.Path getDatabasePath(Database db)
                                          throws MetaException

Throws:

MetaException

getDefaultDatabasePath

public org.apache.hadoop.fs.Path getDefaultDatabasePath(String dbName)
                                                 throws MetaException

Throws:

MetaException

getTablePath

public org.apache.hadoop.fs.Path getTablePath(Database db,
                                     String tableName)
                                       throws MetaException

Throws:

MetaException

getQualifiedName

public static String getQualifiedName(Table table)

getQualifiedName

public static String getQualifiedName(Partition partition)

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path f,
             boolean inheritPermCandidate)
               throws MetaException

Throws:

MetaException

renameDir

public boolean renameDir(org.apache.hadoop.fs.Path sourcePath,
                org.apache.hadoop.fs.Path destPath)
                  throws MetaException

Throws:

MetaException

renameDir

public boolean renameDir(org.apache.hadoop.fs.Path sourcePath,
                org.apache.hadoop.fs.Path destPath,
                boolean inheritPerms)
                  throws MetaException

Throws:

MetaException

deleteDir

public boolean deleteDir(org.apache.hadoop.fs.Path f,
                boolean recursive)
                  throws MetaException

Throws:

MetaException

deleteDir

public boolean deleteDir(org.apache.hadoop.fs.Path f,
                boolean recursive,
                boolean ifPurge)
                  throws MetaException

Throws:

MetaException

isEmpty

public boolean isEmpty(org.apache.hadoop.fs.Path path)
                throws IOException,
                       MetaException

Throws:

IOException

MetaException

isWritable

public boolean isWritable(org.apache.hadoop.fs.Path path)
                   throws IOException

Throws:

IOException

makePartPath

public static String makePartPath(Map<String,String> spec)
                           throws MetaException

Given a partition specification, return the path corresponding to the partition spec. By default, the specification does not include dynamic partitions.

Parameters:

spec -

Returns:

string representation of the partition specification.

Throws:

MetaException

makePartName

public static String makePartName(Map<String,String> spec,
                  boolean addTrailingSeperator)
                           throws MetaException

Makes a partition name from a specification

Parameters:

spec -

addTrailingSeperator - if true, adds a trailing separator e.g. 'ds=1/'

Returns:

partition name

Throws:

MetaException

makeDynamicPartName

public static String makeDynamicPartName(Map<String,String> spec)

Given a dynamic partition specification, return the path corresponding to the static part of partition specification. This is basically a copy of makePartName but we get rid of MetaException since it is not serializable.

Parameters:

spec -

Returns:

string representation of the static part of the partition specification.

makeValsFromName

public static void makeValsFromName(String name,
                    AbstractList<String> result)
                             throws MetaException

Extracts values from partition name without the column names.

Parameters:

name - Partition name.

result - The result. Must be pre-sized to the expected number of columns.

Throws:

MetaException

makeSpecFromName

public static LinkedHashMap<String,String> makeSpecFromName(String name)
                                                     throws MetaException

Throws:

MetaException

makeSpecFromName

public static void makeSpecFromName(Map<String,String> partSpec,
                    org.apache.hadoop.fs.Path currPath)

makeEscSpecFromName

public static Map<String,String> makeEscSpecFromName(String name)
                                              throws MetaException

Throws:

MetaException

getPartitionPath

public org.apache.hadoop.fs.Path getPartitionPath(Database db,
                                         String tableName,
                                         LinkedHashMap<String,String> pm)
                                           throws MetaException

Throws:

MetaException

getPartitionPath

public org.apache.hadoop.fs.Path getPartitionPath(org.apache.hadoop.fs.Path tblPath,
                                         LinkedHashMap<String,String> pm)
                                           throws MetaException

Throws:

MetaException

isDir

public boolean isDir(org.apache.hadoop.fs.Path f)
              throws MetaException

Throws:

MetaException

makePartName

public static String makePartName(List<FieldSchema> partCols,
                  List<String> vals)
                           throws MetaException

Throws:

MetaException

getFileStatusesForSD

public org.apache.hadoop.fs.FileStatus[] getFileStatusesForSD(StorageDescriptor desc)
                                                       throws MetaException

Parameters:

desc -

Returns:

array of FileStatus objects corresponding to the files making up the passed storage description

Throws:

MetaException

getFileStatusesForLocation

public org.apache.hadoop.fs.FileStatus[] getFileStatusesForLocation(String location)
                                                             throws MetaException

Parameters:

location -

Returns:

array of FileStatus objects corresponding to the files making up the passed storage description

Throws:

MetaException

getFileStatusesForUnpartitionedTable

public org.apache.hadoop.fs.FileStatus[] getFileStatusesForUnpartitionedTable(Database db,
                                                                     Table table)
                                                                       throws MetaException

Parameters:

table -

Returns:

array of FileStatus objects corresponding to the files making up the passed unpartitioned table

Throws:

MetaException

makePartName

public static String makePartName(List<FieldSchema> partCols,
                  List<String> vals,
                  String defaultStr)
                           throws MetaException

Makes a valid partition name.

Parameters:

partCols - The partition columns

vals - The partition values

defaultStr - The default name given to a partition value if the respective value is empty or null.

Returns:

An escaped, valid partition name.

Throws:

MetaException

getPartValuesFromPartName

public static List<String> getPartValuesFromPartName(String partName)
                                              throws MetaException

Throws:

MetaException

makeSpecFromValues

public static Map<String,String> makeSpecFromValues(List<FieldSchema> partCols,
                                    List<String> values)