Databricks PySpark Job keeps getting cancelled

Question

I'm using the Databricks notebook on Azure and I had a perfectly fine Pyspark notebook that had been running well all day yesterday. But then at the end of the day I noticed I was getting some strange error on code that I knew was previously working: org.apache.spark.SparkException: Job aborted due to stage failure: Task from application

But as it was late I left it till today. Today I tried creating a fresh cluster an running the code and this time it just keeps saying that my job was "cancelled"

In fact I just tried running 1 single line of code:

filePath = "/SalesData.csv"

and even that got cancelled.

Edit:

Here is the std error log from Azure:

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
/databricks/python/lib/python3.5/site-packages/IPython/config/loader.py:38: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
  from IPython.utils.traitlets import HasTraits, List, Any, TraitError
Fri Jan  4 16:51:08 2019 py4j imported
Fri Jan  4 16:51:08 2019 Python shell started with PID  2543  and guid  86405138b8744987a1df085e4454bb5d
Could not launch process The 'config' trait of an IPythonShell instance must be a Config, but a value of class 'IPython.config.loader.Config' (i.e. {'HistoryManager': {'hist_file': ':memory:'}, 'HistoryAccessor': {'hist_file': ':memory:'}}) was specified. Traceback (most recent call last):
  File "/tmp/1546620668035-0/PythonShell.py", line 1048, in <module>
    launch_process()
  File "/tmp/1546620668035-0/PythonShell.py", line 1036, in launch_process
    console_buffer, error_buffer)
  File "/tmp/1546620668035-0/PythonShell.py", line 508, in __init__
    self.shell = self.create_shell()
  File "/tmp/1546620668035-0/PythonShell.py", line 617, in create_shell
    ip_shell = IPythonShell.instance(config=config, user_ns=user_ns)
  File "/databricks/python/lib/python3.5/site-packages/traitlets/config/configurable.py", line 412, in instance
    inst = cls(*args, **kwargs)
  File "/databricks/python/lib/python3.5/site-packages/IPython/terminal/embed.py", line 159, in __init__
    super(InteractiveShellEmbed,self).__init__(**kw)
  File "/databricks/python/lib/python3.5/site-packages/IPython/terminal/interactiveshell.py", line 455, in __init__
    super(TerminalInteractiveShell, self).__init__(*args, **kwargs)
  File "/databricks/python/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 622, in __init__
    super(InteractiveShell, self).__init__(**kwargs)
  File "/databricks/python/lib/python3.5/site-packages/traitlets/config/configurable.py", line 84, in __init__
    self.config = config
  File "/databricks/python/lib/python3.5/site-packages/traitlets/traitlets.py", line 583, in __set__
    self.set(obj, value)
  File "/databricks/python/lib/python3.5/site-packages/traitlets/traitlets.py", line 557, in set
    new_value = self._validate(obj, value)
  File "/databricks/python/lib/python3.5/site-packages/traitlets/traitlets.py", line 589, in _validate
    value = self.validate(obj, value)
  File "/databricks/python/lib/python3.5/site-packages/traitlets/traitlets.py", line 1681, in validate
    self.error(obj, value)
  File "/databricks/python/lib/python3.5/site-packages/traitlets/traitlets.py", line 1528, in error
    raise TraitError(e)
traitlets.traitlets.TraitError: The 'config' trait of an IPythonShell instance must be a Config, but a value of class 'IPython.config.loader.Config' (i.e. {'HistoryManager': {'hist_file': ':memory:'}, 'HistoryAccessor': {'hist_file': ':memory:'}}) was specified.

Matt Matt · Accepted Answer · 2019-01-17T15:33:11

My team and I ran into this issue after installing the azureml['notebooks'] Python package into our cluster. The install seemed to work, but we received the 'Cancelled' message trying to run a code cell.

We also received an error in our log similar to the one in this post:

The 'config' trait of an IPythonShell instance must be a Config, 
  but a value of class 'IPython.config.loader.Config'...

It seems that some Python packages may conflict with this Config object, or be incompatible. We uninstalled the library, restarted the cluster, and everything worked. Hope this helps someone :)

Databricks PySpark Job keeps getting cancelled

4 Answers