Unable to import uploaded Python wheel in Databricks

Question

Also posted on the official forum, reposting here for visibility.

Completed all the steps from here to upload a custom Python wheel to Databricks. I've built a simple test wheel called somelongexamplename and compiled it to a .whl using Pyhton 3.5.2 . It's very simple and only has one module with one method that prints "Hello world".

I've created a library in the Shared folder in the workspace in Databricks and then created a custom Cluster (v5.5) and installed the wheel on that one. Screenshots below show the library installed on the cluster and the cluster with the library installed.

It is similarly visible on the databricks-cli as shown below.

Running the below command in a notebook attached to the testing cluster also shows the wheel installed correctly.

%sh
/databricks/python/bin/pip freeze

Yet still when I run:

import somelongexamplename

I get:

ImportError                               Traceback (most recent call last)
<command-194690> in <module>()
----> 1 import somelongexamplename

ImportError: No module named 'somelongexamplename'

Does anyone have an idea of what may be causing this? I can't seem to find any other online resources that provide a solution.

philMarius philMarius · Accepted Answer · 2020-01-09T14:17:05

Turns out the problem was in my setup.py file all along. I had a misconfigured packages value in the setup method.

It was initially this:

setup(
    ...
    packages=find_packages(where='somelongexamplename'),
    package_dir={'': 'somelongexamplename'},
    ...
)

Changing it to this fixed it for me:

setup(
    ...
    packages=['somelongexamplename'],
    ...
)

If you're encountering this issue, please ensure you have your wheel configured correctly.

Unable to import uploaded Python wheel in Databricks

2 Answers