dask worker cannot import module

Question

I am running a dask cluster and a worker w. 16 cores using the CLI utilities. In general it seems to work very well. However, for some reason it will not import modules in the cwd. I try to run the following from my notebook instance:

def tstimp():
  import os
  return os.listdir()
c.run(tstimp)

And i get the following output:

{'tcp://192.168.1.90:35885': ['class_positions.csv',
'.gitignore',
'README.md',
'fullrun.ipynb',
'.git',
'rf.py',
'__pycache__',
'dask-worker-space',
'utils.py',
'.ipynb_checkpoints']}

Note that the module rf.py is listed here. Thus it should be possible to import it in the worker, but when i run the following code:

def tstimp():
  import rf
  return 42
c.run(tstimp)

I get this error: ModuleNotFoundError: No module named 'rf'

Why am I getting this error?

Ivo Merchiers Ivo Merchiers · Accepted Answer · 2019-08-19T12:12:01

It seems like the current directory is not added to the python path of the workers. You should be able to fix this by adding it to the path.

def tstimp():
  import sys
  sys.path.append('.')
  import rf
  return 42
c.run(tstimp)

dask worker cannot import module

1 Answers