In Databricks, tables are the primary way to organize and store structured data within the lakehouse architecture. They can be managed or unmanaged (external); the data is typically stored in the Delta Lake format.

Managed Tables

• Lifecycle Management: Databricks fully manages the lifecycle of these tables, including file layout and data storage.
• Data Deletion: When a managed table is dropped, the underlying data is deleted (usually within 30 days).
• Storage: Managed tables are stored in a designated managed storage location within your Databricks environment.
• Format: These tables always use the Delta Lake format, which provides ACID transactions, time travel, and other advanced features.

Unmanaged (External) Tables

• Metadata Only: Databricks manages the metadata (schema, table name, etc.) for these tables, not the underlying data.
• Data Location: The data for unmanaged tables is stored outside of Databricks’ managed storage, often in cloud object storage like S3 or ADLS.
• Flexibility: These tables offer more flexibility as you can directly access and manipulate the data files using external tools.
• Data Persistence: Dropping an unmanaged table only removes the metadata from Databricks; the data remains untouched.

Key Points

• Delta Lake: Delta Lake is the default and recommended format for tables in Databricks due to its performance, reliability, and advanced features.
• Unity Catalog: For organizations using Unity Catalog, tables are managed within a centralized governance framework with granular permissions and security controls.
• SQL and APIs: You can interact with tables using SQL queries, as well as various APIs in Python, R, Scala, and Java.