Databricks SparkSQL job

0
votes

Does Databricks support submitting a SparkSQL job similar to Google Cloud Dataproc?

The Databricks Job API, doesn't seem to have an option for submitting a Spark SQL job.

Reference: https://docs.databricks.com/dev-tools/api/latest/jobs.html https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.jobs

2
apache-spark-sqldatabricksgoogle-cloud-dataproc
what does it matter? you just do %sql <statement> ... - thebluephantom
inside a notebook - thebluephantom
So my use case is something like this. I have a multi-tenant Microservice which has to do some aggregations based on some changes to the data triggered by a tenant action. The aggregated data needs to be exposed via a REST API. The idea is to do the aggregations and be able to save them in my DB. With a DW like Google Big Query, we can get the result set back and store the data required in a synchronous call. I am looking for something similar - java_geek
mmm. I'm not fan of micro services. in fact on my last assignment the chief architect stated, don't unless you have to. In any event I am not sure. I looked at the REST stuff some time ago, but I am more from the BI and batch / engineering side. Answer still holds for your question afaics. - thebluephantom
did u find a solution otherwise? - thebluephantom

2 Answers

1
votes

No, you submit a notebook.

That notebook can be many things: python, spark script or with %sql Spark SQL.

0
votes

You can submit the spark job on databricks cluster just like the dataproc. Run your spark job in scala context and create a jar for the same. Submitting spark-sql directly is not supported. To create a job follow the official guide https://docs.databricks.com/jobs.html

Also, to trigger the job using REST API you can trigger the run-now request described https://docs.databricks.com/dev-tools/api/latest/jobs.html#runs-submit