Python Run Unittest as Package Import Error

Question

I. Preface: The application directory structure and modules are listed at the end of the post.

II. Statement of the Problem: If the PYTHONPATH is not set the application runs, but the unittest fails with the ImportError: No module named models.transactions. This happens when trying to import Transactions in app.py. If the PYTHONPATH is set to /sandbox/app both the application and unittest run with no errors. The constraints for a solution are that the PYTHONPATH should not have to be set, and the sys.path should not have to be modified programmatically.

III. Details: Consider the case when the PYTHONPATH is set and the test_app.py is run as a package /sandbox$ python -m unittest tests.test_app. Looking at the print statements for __main__ sprinkled throughout the code:

 models   :  app.models.transactions
 models   :  models.transactions
 resources:  app.resources.transactions
 app      :  app.app
 test     :  tests.test_app

The unittest imports the app first, and so there is app.models.transactions. The next import that the app attempts is resources.transactions. When it is imported it makes its own import of models.transactions, and then we see the __name__ for app.resources.transactions. This is followed by the app.app import, and then finally the unittest module tests.test.app. Setting the PYTHONPATH allows the application to resolve models.transactions!

One solution is to put the models.transactions inside resources.transaction. But is there another way to handle the problem?

For completeness, when the application is run the print statements for __main__ are:

 models   :  models.transactions
 resources:  resources.transactions
 app      :  __main__

This is expected, and no imports are being attempted which are above /sandbox/app or sideways.

IV. Appendix

A.1 Directory Structure:

|-- sandbox
    |-- app
        |-- models
            |-- __init__.py
            |-- transactions.py 
        |-- resources
            |-- __init__.py
            |-- transactions.py        
        |-- __init__.py
        |-- app.py
    |-- tests
        |-- __init__.py
        |-- test_app.py

A.2 Modules:

(1) app:

from flask import Flask
from models.transactions import TransactionsModel
from resources.transactions import Transactions
print '     app      : ', __name__
def create_app():
    app = Flask(__name__)
    return app
app = create_app()
if __name__ == '__main__':
    app.run(host='127.0.0.1', port=5000, debug=True)

(2) models.transactions

print '     model    : ', __name__
class TransactionsModel:
    pass

(3) resources.transactions:

from models.transactions import TransactionsModel
print '     resources: ', __name__ 
class Transactions:
    pass

(4) tests.test_app

import unittest 
from app.app import create_app
from app.resources.transactions import Transactions   
print '     test     : ', __name__ 
class DonationTestCase(unittest.TestCase):
    def setUp(self):
        pass
    def tearDown(self):
        pass
    def test_transactions_get_with_none_ids(self):
        self.assertEqual(0, 0) 
if __name__ == '__main__':
    unittest.main()

Unknown Unknown · Accepted Answer · 2017-12-22T16:18:29

It is worthwhile mentioning up front that the Flask documents say to run the application as a package, and set the environment variable: FLASK_APP. The application then runs from the project root: $ python -m flask run. Now the imports will include the application root, such as app.models.transactions. Since the unittest is being run in the same way, as a package from the project root, all imports are resolved there as well.

The crux of the problem can be described in the following way. The test_app.py needs access to sideways imports, but if it runs as a script like:

/sandbox/test$ python test_app.py

it has __name__==__main__. This means that imports, such as from models.transactions import TransactionsModel, will not be resolved because they are sideways and not lower in the hierarchy. To work around this the test_app.py can be run as a package:

/sandbox$ python unittest -m test.test_app

The -m switch is what tells Python to do this. Now the package has access to app.model because it is running in /sandbox. The imports in test_app.py must reflect this change and become something like:

from app.models.transactions import TransactionsModel

To make the test run, the imports in the application must now be relative. For example, in app.resources:

from ..models.transactions import TransactionsModel

So the tests run successfully, but if the application is run it fails! This is the crux to the problem. When the application is run as a script from /sandbox/app$ python app.py it hits this relative import ..models.transactions and returns an error that the program is trying to import above the top level. Fix one, and break the other.

How can one work around this without having to set the PYTHONPATH? A possible solution is to use a conditional in the package __init__.py to do conditional imports. An example of what this looks like for the resources package is:

if __name__ == 'resources':
    from models.transactions import TransactionsModel
    from controllers.transactions import get_transactions
elif __name__ == 'app.resources':
    from ..models.transactions import TransactionsModel
    from ..controllers.transactions import get_transactions

The last obstacle to overcome is how do we get this pulled into the resources.py. The imports done in the __init__.py are bound to that file and are not available to resources.py. Normally one would include the following import in resources.py:

import resources

But, again, is it resources or app.resources? It would seem like the difficulty has just moved further down the road for us. The tools offered by importlib can help here, for example the following will make the correct import:

from importlib import import_module
import_module(__name__)

There are other methods that can be used. For example,

TransactionsModel = getattr(import_module(__name__), 'TransactionsModel')

This fixed the error in the present context.

Another, more direct solution is using absolute imports in the modules themselves. For example, in resources:

models_root = os.path.join(os.path.dirname(__file__), '..', 'models')
fp, file_path, desc = imp.find_module(module_name, [models_root])
TransactionsModel = imp.load_module(module_name, fp, file_path, 
    desc).TransactionsModel
TransactionType = imp.load_module(module_name, fp, file_path, 
    desc).TransactionType

Just a note concerning changing the PYTHONPATH with sys.path.append(app_root) in resources.py. This works well and is a few lines of code located where it needs to be. Additionally, it only changes the path for the executing file and reverts back when finished. Seems like a good use case for unittest. One concern might be when the application is moved to different environments.

Python Run Unittest as Package Import Error

2 Answers