The Spark configuration property spark.databricks.aggressiveWindowDown is used in Databricks clusters to control the frequency with which the cluster evaluates whether to scale down the number of workers.

Setting it to 600 (the maximum value) means:

• The cluster will check for potential downscaling opportunities every 600 seconds (10 minutes).
• This will slow down the downscaling process compared to lower values.
• It can be beneficial if your workloads have periods of high activity followed by lulls, as it prevents premature downscaling during brief quiet periods.

Considerations:

• Cost: Keeping workers running longer can increase costs, especially if the cluster remains underutilized for extended periods.
• Workload: The ideal value depends on the nature of your workloads. If you have consistent workloads, a lower value might be more efficient.

How to Set It:

You can set this configuration property in your Databricks cluster settings or through your job’s configuration.