Understanding the row limit in Databricks SQL is crucial. Currently, the maximum number of rows displayed in the query results UI is 64,000. This means that if your query returns more than 64,000 rows, you will only see the first 64,000 in the UI.

Why does this limit exist?

This limit is primarily to ensure optimal performance. Displaying large result sets can consume significant server and client resources, potentially leading to slowdowns and crashes. By setting a limit, we can maintain the system’s stability and prevent such issues.

What are the workarounds?

If you need to work with all the rows returned by your query, you have a few options:

1. Export the results: You can export the complete result set to a file (CSV, Parquet, etc.) and then work with it using other tools or libraries.
2. Use the API: Databricks provides APIs that allow you to fetch and process query results programmatically without the limitations of the UI.
3. Aggregate or sample: If you don’t need to see every row, consider aggregating or sampling your data to reduce the result set size.
4. Patiently wait for updates: Databricks is actively working on increasing this limit in the future.