How to Use Databricks Jobs API for Effective Job Management

Building Machine Learning applications require companies to compile several tasks that are often cost-ineffective and consume more time. As a result, companies look for superior automation systems that can simplify the execution of mundane tasks. Databricks is a Data Analytics platform and enterprise software that provides companies with platform leverages with the pre-configured Machine Learning environment and helps them boos the Data Analysis process.

Companies use various platforms for their daily business activities and all these platforms store valuable business data. Also, to analyze the data, companies need to accomplish their tasks via Jobs that run code in Databricks clusters. To access Databricks Jobs from any 3rd party tools or external source, companies need to access Databricks Jobs API.

Databricks Jobs API allows businesses to do several tasks, including ETL tasks, on a given schedule, reducing the manual efforts required while working with data-related processes. In this article, you will learn about Databricks and the basic operations of Databricks Jobs API. It also introduces you to the fundamental elements included in the Databricks Workspace and some best practices you can follow while using Databricks Jobs API.

Prerequisites

• Understanding of Cloud Data Engineering.
• An active Databricks account.
• An idea of API requests.

Introduction to Databricks

Databricks is a Cloud-based Data Engineering and Big Data processing platform that unifies data to handle analytics and AI workloads. As companies collect large volumes of data for performing analytics tasks, data architects at Databricks designed a LakeHouse platform, which combines the reliability and governance of Data Warehouses with the flexibility of Data Lakes to perform SQL Analytics, BI, Data Science, and Machine Learning.

Introduction Databricks Workspace

Databricks Workspace is an environment for accessing all the Databricks assets. The workspace organizes objects into folders while providing access to data and computational resources. Below is the list of Databricks Workspace assets:

1) Clusters

A Cluster provides a set of computation resources and configurations for executing a particular process. It helps users run ETL pipelines, streaming analytics, Ad-hoc Analytics, Machine Learning, and other use cases.

2) Notebooks

Notebook is a web-based interface for developers to run codes, generate visualizations, and write narrative text. It allows users to import files and tables that run in sequential order to produce output from one or more previously run commands.

3) Jobs

Job is another programming approach that runs code in a Databricks cluster other than notebooks. A job consists of a single task that can be scheduled or run interactively using notebook UI (user interface). If you enable (recommended) ‘orchestration of multiple tasks in a workspace, Databricks runs multiple tasks like production pipelines to manage task orchestration. This ability simplifies the creation, management, monitoring, and error reporting for a job. 

Here are some important limitations while configuring a job:

• A job can be created only in a Data Science and Engineering workspace or a Machine Learning workspace.
• A workspace is limited to 1000 concurrent job runs.
• At any given hour, the number of jobs a workspace can create is limited to 5000. 

4) Libraries

Library makes third-party or locally built code available for notebooks and jobs running on a cluster. Databricks allows users to install libraries in three modes — workspace, cluster, and notebook that are scoped to their respective environments for a given session.

5) Data

Data is imported into Databricks workspace to perform desired operations in notebooks and clusters. It is brought into distributed file systems like Databricks File System (DBFS) to analyze small data files present in local machines. Whereas, for large data files, Databricks provides a wide variety of Apache Spark data sources like Avro, Hive Table, and many more.

6) Repos

Repos provide a repository-level integration with Git to support best practices for data science code development. Organizations sync algorithms created in Databricks notebooks with a remote Git repository to collaborate and version control. With Databricks Repos, developers can leverage Git functionality to clone repositories, manage branches, push or pull changes, and visually compare differences in a commit.

7) Models

MLflow in Databricks offers ML engineers an integrated experience to track and secure ML models. Model in Databricks refers to a model registered in MLflow Model Registry, which is a centralized model store to manage the end-to-end machine learning life cycle of models. 

Operations in Databricks Jobs API 

Job is one of the workspace assets that runs a task in a Databricks cluster. A job can be configured using UI, CLI (command line interface), and invoking the Databricks Jobs API. The Databricks Jobs API allows you to create, edit, and delete jobs with a maximum permitted request size of up to 10MB.

The Databricks Jobs API follows the guiding principles of representational state transfer (REST) architecture. Authentication and access to Databricks REST API can be processed either using Databricks personal access token or password. Databricks recommends using Databricks Jobs API 2.1 (recent release) that not only supports jobs with multiple tasks but also guides users to update their existing API clients. The steps that will help you understand basic operations in Databricks Jobs API are listed below:

• Creating a New Job
• Listing All the Jobs
• Updating and Resetting Jobs
• Deleting a Job and task

1) Creating a New Job

To create a new job, users can send a request to the server. Databricks Jobs APIs use HTTP POST requests methods that consist of request body schema as discussed below:

Schema	Data Type	Description
name	String	It helps developers to give an optional name for a job, and the default name is ‘untitled.’
tasks	Array	It comprises a list of task specifications
job_clusters	Array	It consists of a unique name for a job cluster
email_notification	Object	It consists of email addresses to be notified for a given event like job success or failure.
timeout_seconds	integer	It is an optional schema that can be applied to each run of the job. By default, there is no time limit.
schedule	Object	It helps to trigger a schedule of jobs for a given time zone.
max_concurrent_runs	integer	It is an optional schema that helps to execute multiple runs on the same job concurrently.
format	String	It helps to describe the type of job that usually takes two values, either ‘SINGLE_TASK’ or ‘MULTI_TASK.’
access_control_list	Array	It comprises a list of permissions to be set on a job.

A ‘post’ request from Databricks Jobs API results in either of the four responses having response schema as shown below:

1. 200: This response code explains that the job was successfully created.
2. 400: This response code shows that the request was malformed. It consists of an error code and a human-readable error message that displays the cause of the error.
3. 401: This response code denotes that the request was unauthorized.
4. 500: This response code reveals that the request was not handled correctly due to a server error.

For instance, if you want to create a job that runs a JAR task at 10:15 pm each night, you can follow the below-mentioned steps:

• For creating a job, use a ‘post’ HTTP method and an endpoint that can be the URL of the server as shown below:

• A ‘post‘ request consists of the following code that depends on the JSON parameters in the ‘create-job‘ file.

• Based on the requirement, desired parameters can be configured by using the ‘create-job.json’ file that contains the request schema as shown below:

• If the specified parameters are correct, response code ‘200’ will be shown with a result as below:

2) Listing All the Jobs

A list of all jobs can be found with the ‘get’ HTTP request method as shown below:

The ‘get‘ request uses a .netrc file for auto-login in a server and a jq (java query) file to extract data from JSON data as shown below:

If the ‘get‘ request is correctly queried, a response will be generated as shown below:

3) Updating and Resetting Jobs

Based on the requirements of organizations, the number of tasks in a job can change. A job can be changed partially or completely, which is called the ‘update‘ and ‘reset‘ operation in Databricks Jobs API, respectively.

A ‘reset‘ operation calls the ‘reset-job.json‘ file to perform a ‘post‘ HTTP request. For instance, if you want to make job 2 identical to job 1, use the below script:

Use the ‘update‘ command for events that require alteration in a specific set of a job, like add, change, or remove operations in a current job. For instance, if your task is to remove libraries and instead add email notification to job1, you can use the below script:

4) Deleting a Job and Task

Deleting a job is carried out by a ‘post‘ HTTP request that not only removes the entire details but also the historical run of a job in Databricks Jobs API. However, no action would occur if a delete operation is performed on a job that is already removed. For any other parametric values, an error message will be displayed as shown below:

Best Practices for Databricks Jobs API

A few best practices for Databricks Jobs API are listed below:

Cluster Configuration

The cluster configuration is an essential parameter while operationalizing a job. Databricks recommends developers use new clusters so that each task runs in a fully isolated environment. If you run a task in a new cluster, it is treated as Data Engineering workloads, whereas a task running on an existing all-purpose cluster is treated as analytics workload.

Notebook

Typically, the entire notebook cell and individual cell output are bound to 20MB and 8MB size limits. If you exceed these limits, a job run is canceled and marked as failed. To optimize the cell performance, Databricks Jobs API recommends using an all-purpose cluster along with the notebook autosave technique.

Streaming

The Streaming jobs should not have maximum concurrent runs greater than one. As the streaming task runs continuously, it should always be the final task, and no retries should be appreciated.

Library Dependencies

The Spark driver has certain library dependencies that cannot be overridden. Such libraries gain more priority compared to any of the other libraries that conflict with them. When dealing with library dependency, a good rule of thumb is to list Spark and Hadoop as provided dependencies while creating JARs for jobs.

Learn how Databricks Materialized Views can optimize query performance while managing workflows with the Databricks Jobs API.

Conclusion

In this article, you learned about Databricks and the basic operations of Databricks Jobs API. You also read about Databricks Workspace and how Databricks Jobs API helps companies automate some of their processes by accessing Databricks with code. Businesses are flooded with an enormous number of tasks while handling Data Science and Machine Learning processes. Combining such tasks can be efficient for enterprises to prohibit manual interventions. Databricks Jobs API stacks a large number of smaller tasks to provide seamless operation automatically. 

Companies need to analyze their business data stored in various sources. To gain a holistic view, this data must be loaded into a singular destination, such as a data warehouse. Hevo Data is a no-code data pipeline solution that helps you transfer data from various sources to your destination of choice. It fully automates the process of transforming and mapping data in the destination.Sign up for Hevo’s 14-day free trial and experience seamless data migration.

FAQs