Databricks Query Federation, also known as Lakehouse Federation, is a feature that allows you to query data from multiple external data sources directly from Databricks without first ingesting or copying the data into your Databricks environment. It is a unified interface for accessing and analyzing data in different systems.

Key benefits of using Databricks Query Federation:

• Reduced data movement: Eliminates the need to copy data into Databricks, saving storage costs and reducing latency.
• Real-time insights: Query the latest data in external systems without waiting for ETL processes.
• Simplified architecture: Centralize your analytics in Databricks, reducing the complexity of managing multiple data systems.
• Improved data governance: Apply Databricks’ Unity Catalog’s security and governance controls to your federated queries.

How does Databricks Query Federation work?

1. Create connections: Use Databricks’ Unity Catalog to set up connections to your external data sources. This includes providing the necessary credentials and connection details.
2. Register foreign catalogs: Create foreign catalogs in Unity Catalog to represent the schemas and tables in your external data sources. This allows you to query them using familiar SQL syntax.
3. Execute federated queries: Write SQL queries referencing tables in your foreign catalogs. Databricks will execute these queries against the external data sources and return the results.

Supported data sources:

Databricks Query Federation currently supports a wide range of data sources, including:

• PostgreSQL
• MySQL
• Snowflake
• Redshift
• Azure Synapse Analytics
• And many more