Here’s a breakdown of the Databricks Hive metastore, including how it works with Unity Catalog:

What is the Hive Metastore?

• • Centralized Repository: The Hive metastore serves as a core component in Databricks (and Apache Hive-based systems) for storing metadata about your tables. Metadata includes: Table names
  • Column names and data types
  • Table locations (where the data is stored)
  • Partitions
  • Other table properties.
• Enabling SQL-like Queries: This metadata is crucial for allowing SQL-like queries on data stored in various file formats within Databricks.

Databricks and the Hive Metastore: Two Options

Databricks offers two types of megastores:

1. Legacy Hive Metastore (Workspace-Level):
   • Each Databricks workspace has its own Hive metastore.
   • This is the traditional meta store familiar to experienced Databricks users.
   • Access is primarily workspace-bound.
2. Unity Catalog (Global-Level):
   • A more recent meta-store option provides greater governance and control.
   • Unity Catalog offers centralized metadata management across multiple Databricks workspaces.
   • It has enhanced security features, fine-grained access controls, and lineage tracking.

Working with both Metastores

• Coexistence: Both the Hive metastore and Unity Catalog can be used simultaneously in Databricks.
• Accessing Hive Metastore Data from Unity Catalog: The legacy Hive metastore appears as a top-level catalog named hive_metastore within the Unity Catalog. You can query tables in it using a three-level namespace (e.g., hive_metastore.database_name.table_name)

Key Considerations and Best Practices

• Use Cases: The legacy Hive metastore is suitable for existing workspaces, and migrations might be necessary to leverage Unity Catalog’s advantages.
• Migration to Unity Catalog: Databricks recommends migrating tables managed by the legacy Hive metastore to Unity Catalog for better governance and security.
• New Projects: Using Unity Catalog for new Databricks projects is generally preferable to benefit from its enhanced features.