calling pytest from inside python code

8
votes

I am writing a Python script for collecting data from running tests under different conditions. At the moment, I am interested in adding support for Py.Test.

The Py.Test documentation clearly states that running pytest inside Python code is supported:

You can invoke pytest from Python code directly... acts as if you would call “pytest” from the command line...

However, the documentation does not describe in detail return value of calling pytest.main() as prescribed. The documentation only seems to indicate how to read the exit code of calling the tests.

What are the limits of data resolution available through this interface? Does this method simply return a string indicating the results of the test? Is support more friendly data structures supported (e.g., outcome of each test case assigned to key, value pair)?

Update: Examining the return data structure in the REPL reveals that calling pytest.main yeilds an integer return type indicating system exit code and directs a side-effect (stream of text detailing test result) to standard out. Considering this is the case, does Py.Test provide an alternate interface for accessing the result of tests run from within python code through some native data structure (e.g., dictionary)? I would like to avoid catching and parsing the std.out result because that approach seems error prone.

1
pythonpytest
For those interested, I wrote some code to extract a dictionary from the test results: github.com/ds2643/runnerpy – David Shaked
There is a pytest plugin that instruments pytest execution. Its intended purpose is testing pytest plugins, though you may be able to adapt it for what you need. doc.pytest.org/en/latest/writing_plugins.html#testing-plugins – Frank T

1 Answers

2
votes

I don`t think so, the official documentation tells us that pytest.main returns an os error code like is described in the example. here

You can use the pytest flags if you want to, even the traceback (--tb) option to see if some of those marks helps you.

In your other point about parsing the std.out result because that approach seems error prone. It really depends on what you are doing. Python has a lot of packages to do it like subprocess for example.