How to register permanent spark udf?

1
votes

I'd like to use my udf in other sessions or in a view so is there any way to register permanent Spark UDF written in Python or scala?

1
apache-sparkviewuser-defined-functionspermanent

1 Answers

3
votes

you can't. Spark doesn't have a "server" it talks with a resource manager (Yarn/Kubernetes/Mesos) and deploys itself anew with each job. You either need to have the code as part of the package you submit or in the prepackage in jar/zip file which you include in the job. In both cases you need to register the udfs.