The Databricks 2.1 API is a REST API that allows you to programmatically interact with Databricks Jobs, enabling greater automation and control over your data processing workflows.

Key features and improvements of the 2.1 API:

• Multi-task format: This format allows you to define complex workflows with multiple tasks, dependencies, and different configurations for each task.
• Task-level clusters: Optimize resource usage and reduce costs by creating and terminating clusters for each task individually.
• Shared job clusters: Improve efficiency when tasks have similar resource requirements by sharing the same cluster among multiple tasks.
• Update request: Modify existing jobs without resetting them, minimizing disruption to your workflows.

Available endpoints and actions:

The Databricks 2.1 API provides various endpoints for managing jobs, including:

• Create: Create new jobs with single or multiple tasks.
• List: Retrieve a list of jobs and their details.
• Get: Retrieve details of a specific job.
• Update: Modify existing jobs without resetting them.
• Reset: Update an existing job and reset its run history.
• Delete: Delete an existing job.
• Run Now: Trigger a job to run immediately.
• Submit Run: Submit a new run for a job.
• List Runs: Retrieve a list of runs for a job.
• Get Run: Retrieve details of a specific run.
• Cancel Run: Cancel a running job.
• Delete Run: Delete a job run.

Using the API:

To use the Databricks 2.1 API, you will need:

1. Databricks Personal Access Token: Generate a token from your Databricks account to authenticate your API requests.
2. API Client: You can use any HTTP client like curl, requests, or postman to interact with the API.

Here’s a simple example using curl to list all jobs:

curl -X GET \
  -H "Authorization: Bearer <your-personal-access-token>" \
  https://<your-databricks-workspace-url>/api/2.1/jobs/list