Skip to main content

Hive Metastore

This document describes all supported parameters when connecting to and accessing Hive MetaStore services through the CREATE CATALOG statement.

Supported Catalog Types​

Catalog TypeType Identifier (type)Description
HivehmsCatalog for connecting to Hive Metastore
IcebergicebergCatalog for Iceberg table format
PaimonpaimonCatalog for Apache Paimon table format

Common Parameters Overview​

The following parameters are common to different Catalog types.

Parameter NameFormer NameRequiredDefaultDescription
hive.metastore.urisYesNoneURI address of Hive Metastore, supports multiple URIs separated by commas. Example: 'hive.metastore.uris' = 'thrift://127.0.0.1:9083','hive.metastore.uris' = 'thrift://127.0.0.1:9083,thrift://127.0.0.1:9084'
hive.metastore.authentication.typehadoop.security.authenticationNosimpleMetastore authentication method: supports simple (default) or kerberos. In versions 3.0 and earlier, authentication method was determined by hadoop.security.authentication property. Starting from version 3.1, Hive Metastore authentication method can be specified separately. Example: 'hive.metastore.authentication.type' = 'kerberos'
hive.metastore.service.principalhive.metastore.kerberos.principalNoEmptyHive server principal, supports _HOST placeholder. Example: 'hive.metastore.service.principal' = 'hive/_HOST@EXAMPLE.COM'
hive.metastore.client.principalhadoop.kerberos.principalNoEmptyKerberos principal used by Doris to connect to Hive MetaStore service.
hive.metastore.client.keytabhadoop.kerberos.keytabNoEmptyKerberos keytab file path
hive.metastore.usernamehadoop.usernameNohadoopHive Metastore username, used in non-Kerberos mode
hive.conf.resourcesNoEmptyhive-site.xml configuration file path, using relative path

Note:

For versions before 3.1.0, please use the former names.

Required Parameters​

  • hive.metastore.uris: Must specify the URI address of Hive Metastore

Optional Parameters​

  • hive.metastore.authentication.type: Authentication method, default is simple, optional kerberos

  • hive.metastore.service.principal: Kerberos principal of Hive MetaStore service, must be specified when using Kerberos authentication.

  • hive.metastore.client.principal: Kerberos principal used by Doris to connect to Hive MetaStore service, must be specified when using Kerberos authentication.

  • hive.metastore.client.keytab: Kerberos keytab file path, must be specified when using Kerberos authentication.

  • hive.metastore.username: Username for connecting to Hive MetaStore service, used in non-Kerberos mode, default is hadoop.

  • hive.conf.resources: hive-site.xml configuration file path, used when configuration for connecting to Hive Metastore service needs to be read from configuration files.

Authentication Methods​

Simple Authentication​

  • simple: Non-Kerberos mode, directly connects to Hive Metastore service.

Kerberos Authentication​

To use Kerberos authentication to connect to Hive Metastore service, configure the following parameters:

  • hive.metastore.authentication.type: Set to kerberos

  • hive.metastore.service.principal: Kerberos principal of Hive MetaStore service

  • hive.metastore.client.principal: Kerberos principal used by Doris to connect to Hive MetaStore service

  • hive.metastore.client.keytab: Kerberos keytab file path

'hive.metastore.authentication.type' = 'kerberos',
'hive.metastore.service.principal' = 'hive/_HOST@EXAMPLE.COM',
'hive.metastore.client.principal' = 'hive/doris.cluster@EXAMPLE.COM',
'hive.metastore.client.keytab' = '/etc/security/keytabs/hive.keytab'

When using Hive MetaStore service with Kerberos authentication enabled, ensure that the same keytab file exists on all FE nodes, the user running the Doris process has read permission to the keytab file, and the krb5 configuration file is properly configured.

For detailed Kerberos configuration, refer to Kerberos Authentication.

Configuration File Parameters​

hive.conf.resources​

If you need to read configuration for connecting to Hive Metastore service through configuration files, you can configure the hive.conf.resources parameter to set the conf file path.

Note: The hive.conf.resources parameter only supports relative paths, do not use absolute paths. The default path is under the ${DORIS_HOME}/plugins/hadoop_conf/ directory. You can specify other directories by modifying hadoop_config_dir in fe.conf.

Example: 'hive.conf.resources' = 'hms-1/hive-site.xml'

Catalog Type-Specific Data​

The following parameters are specific to each Catalog type, in addition to the common parameters.

Hive Catalog​

Parameter NameFormer NameRequiredDefaultDescription
typeYesNoneCatalog type, fixed as hms for Hive Catalog
hive.metastore.typeNo'hms'Metadata Catalog type, fixed as hms for Hive Metastore, must be hms when using HiveMetaStore

Examples​

  1. Create a Hive Catalog using unauthenticated Hive Metastore as metadata service, with S3 storage service.

    CREATE CATALOG hive_hms_s3_test_catalog PROPERTIES (
        'type' = 'hms',
        'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
        's3.access_key' = 'S3_ACCESS_KEY',
        's3.secret_key' = 'S3_SECRET_KEY',
        's3.region' = 's3.ap-east-1.amazonaws.com'
    );

  2. Create a Hive Catalog using Hive Metastore with Kerberos authentication enabled as metadata service, with S3 storage service.

     CREATE CATALOG hive_hms_on_oss_kerberos_new_catalog PROPERTIES (
        'type' = 'hms',
        'hive.metastore.uris' = 'thrift://127.0.0.1:9583',
        'hive.metastore.client.principal'='hive/presto-master.docker.cluster@LABS.TERADATA.COM',
        'hive.metastore.client.keytab' = '/mnt/keytabs/keytabs/hive-presto-master.keytab',
        'hive.metastore.service.principal' = 'hive/hadoop-master@LABS.TERADATA.COM',
        'hive.metastore.authentication.type'='kerberos',
        'hadoop.security.auth_to_local' = 'RULE:[2:\$1@\$0](.*@LABS.TERADATA.COM)s/@.*//
                           RULE:[2:\$1@\$0](.*@OTHERLABS.TERADATA.COM)s/@.*//
                           RULE:[2:\$1@\$0](.*@OTHERREALM.COM)s/@.*//
                           DEFAULT',
        'oss.access_key' = 'OSS_ACCESS_KEY',
        'oss.secret_key' = 'OSS_SECRET_KEY',
        'oss.endpoint' = 'oss-cn-beijing.aliyuncs.com'
    );

Iceberg Catalog​

Parameter NameFormer NameRequiredDefaultDescription
typeYesNoneCatalog type, fixed as iceberg for Iceberg
iceberg.catalog.typeNoNoneMetadata Catalog type, fixed as hms for Hive Metastore, must be hms when using HiveMetaStore
warehouseNoNoneIceberg warehouse path

Examples​

  1. Create an Iceberg Catalog using Hive Metastore as metadata service, with S3 storage service.

     CREATE CATALOG iceberg_hms_s3_test_catalog PROPERTIES (
        'type' = 'iceberg',
        'iceberg.catalog.type' = 'hms',
        'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
        'warehouse' = 's3://doris/iceberg_warehouse/',
        's3.access_key' = 'S3_ACCESS_KEY',
        's3.secret_key' = 'S3_SECRET_KEY',
        's3.region' = 's3.ap-east-1.amazonaws.com'
    );

  2. Create an Iceberg Catalog using Hive Metastore with Kerberos authentication enabled as metadata service in a multi-Kerberos environment, with S3 storage service.

    CREATE CATALOG IF NOT EXISTS iceberg_hms_on_oss_kerberos_new_catalog PROPERTIES (
        'type' = 'iceberg',
        'iceberg.catalog.type' = 'hms',
        'hive.metastore.uris' = 'thrift://127.0.0.1:9583',
        'warehouse' = 'oss://doris/iceberg_warehouse/',
        'hive.metastore.client.principal'='hive/presto-master.docker.cluster@LABS.TERADATA.COM',
        'hive.metastore.client.keytab' = '/mnt/keytabs/keytabs/hive-presto-master.keytab',
        'hive.metastore.service.principal' = 'hive/hadoop-master@LABS.TERADATA.COM',
        'hive.metastore.authentication.type'='kerberos',
        'hadoop.security.auth_to_local' = 'RULE:[2:\$1@\$0](.*@LABS.TERADATA.COM)s/@.*//
                           RULE:[2:\$1@\$0](.*@OTHERLABS.TERADATA.COM)s/@.*//
                           RULE:[2:\$1@\$0](.*@OTHERREALM.COM)s/@.*//
                           DEFAULT',
        'oss.access_key' = 'OSS_ACCESS_KEY',
        'oss.secret_key' = 'OSS_SECRET_KEY',
        'oss.endpoint' = 'oss-cn-beijing.aliyuncs.com'
    );

Paimon Catalog​

Parameter NameFormer NameRequiredDefaultDescription
typeYesNoneCatalog type, fixed as paimon for Paimon
paimon.catalog.typeNofilesystemMust be hms when using HiveMetaStore, default is filesystem for storing metadata in filesystem
warehouseYesNonePaimon warehouse path

Examples​

  1. Create a Paimon Catalog using Hive Metastore as metadata service, with S3 storage service.

     CREATE CATALOG IF NOT EXISTS paimon_hms_s3_test_catalog PROPERTIES (
        'type' = 'paimon',
        'paimon.catalog.type' = 'hms',
        'hive.metastore.uris' = 'thrift://127.0.0.1:9383',
        'warehouse' = 's3://doris/paimon_warehouse/',
        's3.access_key' = 'S3_ACCESS_KEY',
        's3.secret_key' = 'S3_SECRET_KEY',
        's3.region' = 's3.ap-east-1.amazonaws.com'
    );

  2. Create a Paimon Catalog using Hive Metastore with Kerberos authentication enabled as metadata service in a multi-Kerberos environment, with S3 storage service.

     CREATE CATALOG IF NOT EXISTS paimon_hms_on_oss_kerberos_new_catalog PROPERTIES (
        'type' = 'paimon',
        'paimon.catalog.type' = 'hms',
        'hive.metastore.uris' = 'thrift://127.0.0.1:9583',
        'warehouse' = 's3://doris/iceberg_warehouse/',
        'hive.metastore.client.principal'='hive/presto-master.docker.cluster@LABS.TERADATA.COM',
        'hive.metastore.client.keytab' = '/mnt/keytabs/keytabs/hive-presto-master.keytab',
        'hive.metastore.service.principal' = 'hive/hadoop-master@LABS.TERADATA.COM',
        'hive.metastore.authentication.type'='kerberos',
        'hadoop.security.auth_to_local' = 'RULE:[2:\$1@\$0](.*@LABS.TERADATA.COM)s/@.*//
                           RULE:[2:\$1@\$0](.*@OTHERLABS.TERADATA.COM)s/@.*//
                           RULE:[2:\$1@\$0](.*@OTHERREALM.COM)s/@.*//
                           DEFAULT',
        'oss.access_key' = 'OSS_ACCESS_KEY',
        'oss.secret_key' = 'OSS_SECRET_KEY',
        'oss.endpoint' = 'oss-cn-beijing.aliyuncs.com'
    );

Frequently Asked Questions (FAQ)​

  • Q1: Is hive-site.xml mandatory?

    No, it's only used when configuration needs to be read from it.

  • Q2: Must the keytab file exist on every node?

    Yes, all FE nodes must be able to access the specified path.

  • Q3: What should be noted when using write-back functionality, i.e., creating Hive/Iceberg databases/tables in Doris?

    Since creating tables involves metadata operations on the storage side, i.e., accessing the storage system, the Hive MetaStore service server side needs to configure corresponding storage parameters, such as access parameters for S3, OSS and other storage services. When using object storage as the underlying storage system, ensure that the bucket being written to matches the configured Region.

Report issue
ASF
Resources
Community
Join the community
Copyright © 2025 The Apache Software Foundation,Licensed under the Apache License, Version 2.0. Apache, Doris, Apache Doris, the Apache feather logo and the Apache Doris logo are trademarks of The Apache Software Foundation.