Can pure python script (not pyspark) run in parallel in a cluster in Azure Databricks?

1
votes

I want to migrate my python scripts from local to run on cloud, specifically on a cluster created on Azure Databricks.

  1. Can pure python script run in parallel (using multiple nodes in a cluster at the same time) without having to be converted into pyspark?
  2. Is it possible to check whether the job is running in parallel?
1
pysparkazure-databricks

1 Answers

1
votes
  1. No
  2. If you have submitted a spark job, you can see logs and it will show you number of executors it is using. That's how you 'll know job is running in parallel