Currently we are using ML pyspark jobs in Azure HDInsight cluster(version: HDI 3.6). Is it possible to use the same pyspark jobs in Azure Databricks without making much changes?
Thanks in advance!
1 Answers
0
votes
I would say yes, but take in consideration the following things :
- Make sure that the Python and Spark version of your HDInsight cluster are similar or compatible with the Databricks runtime choosen for the cluster.
- Are you using any external library? If so import it to your workspace and attach it to the cluster.
- Which data source are you using for your jobs? You may have to mount the data source into your databricks workspace, for example to mount a blob storage ( https://docs.databricks.com/spark/latest/data-sources/azure/azure-storage.html#mount-azure-blob-storage-containers-with-dbfs )
I hope this info helps you
Regards,
Juan
