AdminManual Metastore Administration

Hive Metastore Administration

Introduction

All the metadata for Hive tables and partitions are accessed through the Hive Metastore. Metadata is persisted using JPOX ORM solution (Data Nucleus) so any database that is supported by it can be used by Hive. Most of the commercial relational databases and many open source databases are supported. See the list of supported databases in section below.

You can find an E/R diagram for the metastore here.

There are 2 different ways to setup the metastore server and metastore database using different Hive configurations:

Configuration options for metastore database where metadata is persisted:

• Local/Embedded Metastore Database (Derby)
• Remote Metastore Database

Configuration options for metastore server:

• Local/Embedded Metastore Server 
• Remote Metastore Server

Basic Configuration Parameters

The relevant configuration parameters are shown here. (Non-metastore parameters are described in Configuring Hive. Also see the Language Manual's Hive Configuration Properties, including Metastore and Hive Metastore Security.)

Also see hivemetastore-site.xml documentation under Configuring Hive.

The Hive metastore is stateless and thus there can be multiple instances to achieve High Availability. Using hive.metastore.uris it is possible to specify multiple remote metastores. Hive will use the first one from the list by default but will pick a random one on connection failure and will try to reconnect.

Additional Configuration Parameters

The following metastore configuration parameters were carried over from old documentation without a guarantee that they all still exist. See the HiveConf Java class for current Hive configuration options, and see the Metastore and Hive Metastore Security sections of the Language Manual's Hive Configuration Properties for user-friendly descriptions of the metastore parameters.

Data Nucleus Auto Start

 <property>
    <name>datanucleus.autoStartMechanism</name>
    <value>SchemaTable</value>
  </property>

Default Configuration

The default configuration sets up an embedded metastore which is used in unit tests and is described in the next section. More practical options are described in the subsequent sections.

Local/Embedded Metastore Database (Derby)

An embedded metastore database is mainly used for unit tests. Only one process can connect to the metastore database at a time, so it is not really a practical solution but works well for unit tests.

For unit tests Local/Embedded Metastore Server configuration for the metastore server is used in conjunction with embedded database.

Derby is the default database for the embedded metastore.

If you want to run Derby as a network server so the metastore can be accessed from multiple nodes, see Hive Using Derby in Server Mode.

Remote Metastore Database

In this configuration, you would use a traditional standalone RDBMS server. The following example configuration will set up a metastore in a MySQL server. This configuration of metastore database is recommended for any real use.

Local/Embedded Metastore Server

In local/embedded metastore setup, the metastore server component is used like a library within the Hive Client. Each Hive Client will open a connection to the database and make SQL queries against it. Make sure that the database is accessible from the machines where Hive queries are executed since this is a local store. Also make sure the JDBC client library is in the classpath of Hive Client. This configuration is often used with HiveServer2 (to use embedded metastore only with HiveServer2 add "--hiveconf hive.metastore.uris=' '" in command line parameters of the hiveserver2 start command or use hiveserver2-site.xml (available in Hive 0.14)).

Remote Metastore Server

In remote metastore setup, all Hive Clients will make a connection to a metastore server which in turn queries the datastore (MySQL in this example) for metadata. Metastore server and client communicate using Thrift Protocol. Starting with Hive 0.5.0, you can start a Thrift server by executing the following command:

hive --service metastore

In versions of Hive earlier than 0.5.0, it's instead necessary to run the Thrift server via direct execution of Java:

$JAVA_HOME/bin/java  -Xmx1024m -Dlog4j.configuration=file://$HIVE_HOME/conf/hms-log4j.properties -Djava.library.path=$HADOOP_HOME/lib/native/Linux-amd64-64/ -cp $CLASSPATH org.apache.hadoop.hive.metastore.HiveMetaStore

If you execute Java directly, then JAVA_HOME, HIVE_HOME, HADOOP_HOME must be correctly set; CLASSPATH should contain Hadoop, Hive (lib and auxlib), and Java jars.

Server Configuration Parameters

The following example uses a Remote Metastore Database.

From Hive 3.0.0 (HIVE-16452) onwards the metastore database stores a GUID which can be queried using the Thrift API get_metastore_db_uuid by metastore clients in order to identify the backend database instance. This API can be accessed by the HiveMetaStoreClient using the method getMetastoreDbUuid().

Client Configuration Parameters

Dynamic Service Discovery Configuration Parameters

From Hive 4.0.0 (HIVE-20794) onwards, similar to HiveServer2, a ZooKeeper service can be used for dynamic service discovery of a remote metastore server. Following parameters are used by both metastore server and client.

If you are using MySQL as the datastore for metadata, put MySQL jdbc libraries in HIVE_HOME/lib before starting Hive Client or HiveMetastore Server.

To change the metastore port, use this hive command:

hive --service metastore -p <port_num>

Supported Backend Databases for Metastore

postgres

Metastore Schema Consistency and Upgrades

Hive now records the schema version in the metastore database and verifies that the metastore schema version is compatible with Hive binaries that are going to accesss the metastore. Note that the Hive properties to implicitly create or alter the existing schema are disabled by default. Hive will not attempt to change the metastore schema implicitly. When you execute a Hive query against an old schema, it will fail to access the metastore.

To suppress the schema check and allow the metastore to implicitly modify the schema, you need to set a configuration property hive.metastore.schema.verification to false in hive-site.xml.

Starting in release 0.12, Hive also includes an off-line schema tool to initialize and upgrade the metastore schema. Please refer to the details here.

Save