Implement DevOps on DataBricks DBFS files

0
votes

I am trying to implement DevOps on Azure Data Factory and Azure Databricks.

I have completed the devops implementation for ADF DevOps and Databricks notebook files.

After deployment it seems like there are some issues with ADF pipelines which are fetching jar files stored in dbfs location.

One of the pipeline is shown below. enter image description here the path to the jar file is dbfs:/FileStore/jars/xxx............xxx1_0_SNAPSHOT-c073a.jar After deploying the ADF pipelines into the PROD environment, where it is pointing to the PROD databricks instance, the jar file with same name will not be available. This is causing the pipeline to fail in PROD.

How to fetch the jars from DEV dbfs location and deploy this into PROD dbfs location with same name in order to make the ADF pipeline run?

Below given is the method I am now following to implement DevOps.

  1. I have created a build pipeline which point to the git repository and build the jar files by executing pom.xml file
  2. Created a release pipeline which will copy the jar files from build artifact to FileStore/jars/
  3. Now the ADF pipeline will point to the jar available in FileStore/jars/

Is there any alternative method to resolve this or is this the proper approach?

1
azureazure-devopsazure-data-factoryazure-data-factory-2azure-databricks

1 Answers

0
votes

How do you deploy Azure Databricks and the JAR file right now?
For Databricks deployment you can use azure.databricks.cicd.tools
For ADF deployment you can use one of these tools: ADFTools
Sorry for the short answer, but I hope it leads you in the right direction.