pipenv + pyenv + jupyter

Question

I use pyenv to install and manage both different versions of python and virtual environments for various projects on my system. But pipenv is the recommended tool for managing virtual environments now, and I like a lot of its features. Everything works for the most part, but pipenv seems to have a leak or something when it comes to Jupyter notebook. Here are the steps which replicate the problem on my machine.

1. Install pipenv using brew install pipenv.
2. Install a new version of Python using, say, pyenv install 3.6.3, and activate it via pyenv global 3.6.3.
3. Install Jupyter notebook and another package, e.g. pip install jupyter requests.
4. Create a new directory and navigate into it; create a pipenv project using pipenv install
5. Activate the pipenv virtual environment using pipenv shell.
6. Do not install any packages, and confirm that none are installed in the current virtualenv using pip list. (I only get pip, setuptools, and wheel.)
7. Run jupyter notebook. My expectation: the console errors out with jupyter: command not found. Actual: the notebook server and browser UI launch as normal.
8. Create a new notebook and import the auxiliary package from step 3: import requests. My expectation: a ModuleNotFoundError is thrown. Actual: the package is imported and can be used as normal.

If I try to import the auxiliary package in the pipenv virtual environment but outside Jupyter notebook (in the repl or in a Python script) , I get a ModuleNotFoundError as expected. So somehow the problem is with jupyter specifically. Note that if I try to recreate this problem using a pyenv virtual environment instead of a pipenv virtual environment then the expected behavior occurs: jupyter notebook produces a jupyter: command not found error in the console.

My question is: am I using pipenv in a way that was not intended, or is there an alternative way to use pipenv that avoids this problem?

Thanks in advance!

Answer 1

The Problem

In step 1 you install pipenv with brew. The brew installation of pipenv also installs a python environment alongside it which is directly used. The command itself is on your path, so you can invoke it independent of the virtualenv you have loaded.

In step 2 you install and load a pyenv virtualenv. Loading the pyenv virtualenv changes your path. When you use pip now, it is the pyenv 3.6.3 pip and it will install into that pyenv. So when, in step 3, you run pip install jupyter requests they go into the pyenv virtualenv.

When you run pipenv, in step 4, it will use it's own python, so it is able to load itself. Your load of the pipenv virtualenv in step 5 then adds the bin folder for that to the your path, allowing you to run things you install. It does not undo the changes pyenv made. When you run pip in step 6 it runs inside this pipenv virtualenv and so prints the limited list of installed packages.

In step 7 you run jupyter. This is not inside the pipenv virtualenv, however it is inside the pyenv one. The pyenv path additions still exist so it is runnable. The jupyter command is a bash script and it does magic to reset the python path before running. That is how it can find the requests module in step 8.

You finally run some form of python repl. This presumably is in your pipenv virtualenv so it is loads modules which are in that environment.

The Solution

You need to unload the global pyenv. Since you have installed packages into your 3.6.3 python install I would also clear any virtualenvs and python versions you have installed. You can just reinstall them.

Your install of pipenv is fine, that can stay. It is also the primary tool that you should be using. You should not need to execute pip or pyenv directly.

You should install jupyter into the pipenv virtualenv and you should then have an install which only works when the pipenv virtualenv is loaded.