pytest cannot import module while python can

Question

I am working on a package in Python. I use virtualenv. I set the path to the root of the module in a .pth path in my virtualenv, so that I can import modules of the package while developing the code and do testing (Question 1: is it a good way to do?). This works fine (here is an example, this is the behavior I want):

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from rc import ns
>>> exit()
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python tests/test_ns.py 
issued command: echo hello
command output: hello

However, if I try to use PyTest, I get some import error messages:

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ pytest
=========================================== test session starts ============================================
platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /home/zz/Desktop/GitFolders/rc, inifile: 
collected 0 items / 1 errors 

================================================== ERRORS ==================================================
________________________________ ERROR collecting tests/test_ns.py ________________________________
ImportError while importing test module '/home/zz/Desktop/GitFolders/rc/tests/test_ns.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_ns.py:2: in <module>
    from rc import ns
E   ImportError: cannot import name ns
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================= 1 error in 0.09 seconds ==========================================
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ which pytest
/home/zz/Desktop/VirtualEnvs/VEnvTestRc/bin/pytest

I am a bit puzzled, it looks like this indicates an import error, but Python does it fine so why is there a problem specifically with PyTest? Any suggestion to the reason / remedy (Question 2)? I googled and stack-overflowed the 'ImportError: cannot import' error for PyTest, but the hits I got were related to missing python path and remedy to this, which does not seem to be the problem here. Any suggestions?

Answer 1

Found the answer:

DO NOT put a __init__.py file in a folder containing TESTS if you plan on using pytest. I had one such file, deleting it solved the problem.

This was actually buried in the comments to the second answer of PATH issue with pytest 'ImportError: No module named YadaYadaYada' so I did not see it, hope it gets more visibility here.

Answer 2

I can't say I understand why this works, but I had the same problem and the tests work fine if I run python -m pytest.

I'm in a virtualenv, with pytest also available globally:

(proj)tom@neon ~/dev/proj$ type -a python
python is /home/tom/.virtualenvs/proj/bin/python
python is /usr/bin/python

(proj)tom@neon ~/dev/proj$ python -V
Python 3.5.2

(proj)tom@neon ~/dev/proj$ type -a pytest
pytest is /home/tom/.virtualenvs/proj/bin/pytest
pytest is /usr/bin/pytest

(proj)tom@neon ~/dev/proj$ pytest --version
This is pytest version 3.5.0, imported from /home/tom/.virtualenvs/proj/lib/python3.5/site-packages/pytest.py

Answer 3

I just solved this by removing the __init__.py in my project root:

.
├── __init__.py <--- removed
├── models
│   ├── __init__.py
│   ├── address.py
│   ├── appointment.py
│   └── client.py
├── requirements.txt
├── setup.cfg
├── tests
│   ├── __init__.py
│   ├── models
│   │   ├── __init__.py
│   │   ├── appointment_test.py
│   │   └── client_test.py
│   └── other_test.py
└── script.py

Answer 4

I had the same problem but for another reason than the ones mentioned:

I had py.test installed globally, while the packages were installed in a virtual environment.

The solution was to install pytest in the virtual environment. (In case your shell hashes executables, as Bash does, use hash -r, or use the full path to py.test)

Answer 5

Had a similar issue and it worked when I added __init__.py file under tests directory.

Answer 6

In my case I am working in a container and unfortuantely pytest has the tendency to use python2.7 rather than my python3 interpreter of choice.

In my case this worked:

python3 -m pytest

My folder structure

/
app/
-module1.py
-module2.py
-tests/
--test_module1.py
--test_module2.py
requirements.txt
README.md

Answer 7

This problem will happen if you have a tests.py file and a tests folder with tests/__init__.py.

During the collection pytest finds the folder, but when it tries to import the test files from the folder, tests.py file will cause the import problem.

To fix simply remove the tests.py file and put all your tests inside the tests/ folder.

For your specific case the fix will be precisely:

• Remove the file /home/zz/Desktop/GitFolders/rc/tests.py
• Make sure /home/zz/Desktop/GitFolders/rc/tests/__init__.py is present

Answer 8

Simply put an empty conftest.py file in the project root directory, because when pytest discovers a conftest.py, it modifies sys.path so it can import stuff from the conftest module. A general directory structure can be:

Root
├── conftest.py
├── module1
│   ├── __init__.py
│   └── sample.py
└── tests
    └── test_sample.py

Answer 9

Install the packages into Your virtual environment.
Then start a new shell and source Your virtual environment again.

Answer 10

I had a similar issue, exact same error, but different cause. I was running the test code just fine, but against an old version of the module. In the previous version of my code one class existed, while the other did not. After updating my code, I should have run the following to install it.

sudo pip install ./ --upgrade

Upon installing the updated module running pytest produced the correct results (because i was using the correct code base).

Answer 11

In my case, the import error occurred because the package is pointing to another package/directory with the same name and its path is one level above the folder I actually wanted. I think this also explains why some people need to remove _ init _.py while others need to add back.

I just put print(the_root_package.__path__) (after import the_root_package) in both python console and pytest scripts to compare the difference

BOTTOM LINE: When you do python, the package you import may be different from the package when you run pytest.

Answer 12

Please check here: https://docs.pytest.org/en/documentation-restructure/background/pythonpath.html

I has an issue with pytest (that was solved using python -m pytest); the error was

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.9/site-packages/...

I found the problem was missing __init__.py in tests/ and tests/subfolders.

Answer 13

I was experiencing this issue today and solved it by calling python -m pytest from the root of my project directory.

Calling pytest from the same location still caused issues.

My Project dir is organized as:

api/
 - server/
  - tests/
      - test_routes.py
  - routes/
      - routes.py
 - app.py

The module routes was imported in my test_routes.py as: from server.routes.routes import Routes

Hope that helps!

Answer 14

Another special case:

I had the problem using tox. So my program ran fine, but unittests via tox kept complaining. After installing packages (needed for the program) you need to additionally specify the packages used in the unittests in the tox.ini

[testenv]
deps =
    package1
    package2 
...

Answer 15

The answer above not work for me. I just solved it by appending the absolute path of the module which not found to the sys.path at top of the test_xxx.py (your test module), like:

import sys
sys.path.append('path')

Answer 16

If it is related to python code that was originally developed in python 2.7 and now migrated into python 3.x than the problem is probably related to an import issue.

e.g. when importing an object from a file: base that is located in the same directory this will work in python 2.x:

from base import MyClass

in python 3.x you should replace with base full path or .base not doing so will cause the above problem. so try:

from .base import MyClass

Answer 17

I was getting this using VSCode. I have a conda environment. I don't think the VScode python extension could see the updates I was making.

python c:\Users\brig\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear test
Test Discovery failed:

I had to run pip install ./ --upgrade

Answer 18

I disagree with the posts saying that you must remove any __init__.py files. What you must instead do is alter the sys.path.

Run an experiment where you print sys.path when running the code normally. Then print sys.path while running the code via pytest. I think you will find that there is a difference between these two paths, hence why pytest breaks.

To fix this, insert the path from the first experiment at the 0th index of the second.

Let '/usr/exampleUser/Documents/foo' be the first element of print(sys.path) for experiment 1.

Below is code that should fix your issue:

import sys sys.path[0] = '/usr/exampleUser/Documents/foo'

Put this at the top of your file, before your actual import statement.

Source: I was dealing with this myself and the above process solved it.

Answer 19

Kept everything same and just added a blank test file at the root folder .. Solved it

Here are the findings, this problem really bugged me for a while. My folder structure was

mathapp/
    - server.py  
    - configuration.py 
    - __init__.py 
    - static/ 
       - home.html  
tests/            
    - functional 
       - test_errors.py 
    - unit  
       - test_add.py

and pytest would complain with the ModuleNotFoundError and gives the HINT - make sure your test modules/packages have valid Python names.

I introduced a mock test file at the same level as mathsapp and tests directory. The file contained nothing. Now pytest does not complain.

Result without the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 1 item / 1 error

=================================== ERRORS ====================================
_______________ ERROR collecting tests/functional/test_func.py ________________
ImportError while importing test module 'C:\mainak\workspace\0github\python-rest-app-cont\tests\functional\test_func.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\functional\test_func.py:4: in <module>
    from mathapp.service import sum
E   ModuleNotFoundError: No module named 'mathapp'
=========================== short test summary info ===========================
ERROR tests/functional/test_func.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
============================== 1 error in 0.24s ===============================

Results with the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 2 items

tests\functional\test_func.py .                                          [ 50%]
tests\unit\test_unit.py .                                                [100%]

============================== 2 passed in 0.11s ==============================

Answer 20

I had placed all my tests in a tests folder and was getting the same error. I solved this by adding an init.py in that folder like so:

.
|-- Pipfile
|-- Pipfile.lock
|-- README.md
|-- api
|-- app.py
|-- config.py
|-- migrations
|-- pull_request_template.md
|-- settings.py
`-- tests
    |-- __init__.py <------
    |-- conftest.py
    `-- test_sample.py

Answer 21

For anyone who tried everything and still getting error,I have a work around.

In the folder where pytest is installed,go to pytest-env folder.

Open pyvenv.cfg file.

In the file change include-system-site-packages from false to true.

home = /usr/bin
include-system-site-packages = true
version = 3.6.6

Hope it works .Don't forget to up vote.

Answer 22

Yet another massive win for Python's import system. I think the reason there is no consensus is that what works probably depends on your environment and the tools you are using on top of it.

I'm using this from VS Code, in the test explorer under Windows in a conda environment, Python 3.8.

The setup I have got to work is:

mypkg/
    __init__.py
    app.py
    view.py
tests/
    test_app.py
    test_view.py

Under this setup intellisense works and so does test discovery.

Note that I originally tried the following, as recommended here.

src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    test_app.py
    test_view.py

I could find no way of getting this to work from VS Code because the src folder just blew the mind of the import system. I can imagine there is a way of getting this to work from the command line. As a relatively new convert to Python programming it gives me a nostalgic feeling of working with COM, but being slightly less fun.

Answer 23

My 2 cents on this: pytest will fail at chance if you are not using virtual environments. Sometimes it will just work, sometimes not.

Therefore, the solution is:

• remove pytest with pip uninstall
• create your venv
• activate your venv
• pip install your project path in editable mode, so it will be treated by pytest as a module (otherwise, pytest wont find your internal imports). You will need a setup.py file for that
• install your packages, including pytest
• finally, run your tests

The code, using windows PowerShell:

pip uninstall pytest
python.exe -m venv my_env
.\my_env\Scripts\activate
(my_env) pip install -e .
(my_env) pip install pytest pytest-html pandas numpy

Then finally

(my_env) pytest --html="my_testing_report.html"

An example of setup.py, for pip install -e:

import setuptools

setuptools.setup(
    name='my_package',
    version='devel',
    author='erickfis',
    author_email='[email protected]',
    description='My package',
    long_description='My gooood package',
    packages=setuptools.find_packages(),
    classifiers=[
        'Programming Language :: Python :: 3',
        'Operating System :: OS Independent',
    ],
    include_package_data=True
)

Answer 24

I solved my problem by setting the PYTHONPATH in Environment Variables for the specific configuration I'm running my tests with.

While you're viewing the test file on PyCharm:

1. Ctrl + Shift + A
2. Type Edit Configurations
3. Set your PYTHONPATH under Environment > Environment variables.

Answer 25

Here's a medium article! describing the problem!

The issue is which pytest you are using and your use of a virtual environment. If you have installed pytest system-wide, in other words, outside of a virtual environment, pytest has a nasty habit of only looking outside your virtual environment for modules! If your project is using a module that is only installed in your virtual environment, and you’re using a system-wide pytest, it won’t find the module, even if you’ve activated the virtual environment.1

Here’s the step-by-step:1

1. Exit any virtual environment
2. Use Pip to uninstall pytest
3. Activate your project’s virtual environment
4. Install pytest inside the virtual environment
5. pytest will now find your virtual-environment-only packages!

Answer 26

It could be that Pytest is not reading the package as a Python module while Python is (likely due to path issues). Try changing the directory of the pytest script or adding the module explicitly to your PYTHONPATH.

Or it could be that you have two versions of Python installed on your machine. Check your Python source for pytest and for the python shell that you run. If they are different (i.e. Python 2 vs 3), use source activate to make sure that you are running the pytest installed for the same python that the module is installed in.

Answer 27

if you need a init.py file in your folder make a copy of the folder and delete init.py in that one to run your tests it works for local projects. If you need to run test regularly see if you can move your init.py to a separate file.

Answer 28

[Resolved] Before directly jumping into the solution of removing/ adding __init__.py, we might also want to look at how the imports are been done in your classes. Actually, I lost a day playing around with just __init__.py thinking that might be the problem :) However, that was quite informative.

In my case, it was the wrong way of calling classes from one python class to another python class which was throwing ImportError. Fixed the way classes/ modules to be called and it worked like charm. Hope, this helps others as well.

And yes, for similar error, we might have different solutions depending on how the code is written. Better to spend more time on self debugging. Lesson Learnt :) Happy coding!!!

Answer 29

If you run Pytest from a terminal:

Run pytest with the --import-mode=append command-line flag.

Argument description in the official documentation: https://docs.pytest.org/en/stable/pythonpath.html

---

UPD: Before I also wrote how to do the same if you use PyCharm, but community does not like extendend answers, so I removed additional information that probably was helpful to someone who have a similar issue.

Answer 30

I had a similar problem just recently. The way it worked for me it was realizing that "setup.py" was wrong

Previously I deleted my previous src folder, and added a new one with other name, but I didn't change anything on the setup.py (newbie mistake I guess).

So pointing setup.py to the right packages folder did the trick for me

from setuptools import find_packages, setup

setup(
    name="-> YOUR SERVICE NAME <-",
    extras_Require=dict(test=["pytest"]),
    packages=find_packages(where="->CORRECT FOLDER<-"),
    package_dir={"": "->CORRECT FOLDER<-"},
)

Also, not init.py in test folder nor in the root one.

Hope it helps someone =)

Best!