ISO ert inline test idiom (appending TEST_OUTPUT from multiple ert calls)

Question

---+ BRIEF

Q: if I have multiple, separate, independent calls to ert, can I arrange to concatenate the output of these tests in the same buffer? With the calls to ert being made at different times?

(Without a centralized thingamajig that knows about these independent tests.)

E.g. if I have two (or more) tests, e.g. in a .emacs file

 (ert-deftest test1 ()(should t))
 (ert 'test1)

 ... many miles away, quite likely in a separate file ...

 (ert-deftest test2 ()(should t))
 (ert 'test2)

Both will run - I can tell this by looking at *Messages*

...
Running 1 tests (2017-07-07 12:10:39-0700)
   passed  1/1  test_ag_non-null_and_true
...
Running 1 tests (2017-07-07 12:10:39-0700)
   passed  1/1  test-ag-code-disabling
...

But the *ert* buffer will only contain output for the last test run.

---+ DETAIL

---++ MOTIVATION

Q: Why do I want to do this?

A: "inline testing". In particular, inline testing of stuff in my .emacs file (and in files it loads and/or requires).

Tests that are run every time .emacs or similar file is loaded. (There may be other tests that I run in batch mode - but here I am talking about tests that I want to run right away, every time .emacs is loaded (and convenient for edit/debug while modifying .emacs).

I.e. I am looking for an idiom for inline tests.

I have been putting code in my .emacs and ~/lib/gnu-emacs/**.el files that looks like this:

;; some blob of code
(defun foo (arg) ... )

(ert-deftest test-foo () ... )
(ert 'test-foo)

i.e. I define the function, the test, and run the test, right next to each other.

... I might as well admit that I often wrap a `progn' around,

(progn     
   ;; some blob of code
   (defun foo (arg) ... ) 
   (ert-deftest test-foo () ... )
   (ert 'test-foo)
)

because this makes it easy to make an edit to function foo or the test, and then jump to the end of the progn and do eval-last-sexp (which I have bound to C-j). And I often also wrap an if or and and around, to make it easy to disable a feature that is in development. Sometimes with let* in then mix, if the feature has variables.

(I have used other ert- family runners, like ert-run-tests-batch and ert-run-tests-interactively. But plain old ert seems enough.)

(Putting defun and deftest inside a progn or other breaks some tools that expect defun to always be at top level. And it makes multiline docstrings even uglier. But it is darn convenient, and IMHO tools that operate on programming language files should parse the programming language.)

So, later, I will do the same thing for another module. Far away. Possibly in a separate file, possibly a library file.

(progn     
   ;; some blob of code
   (defun bar (arg) ... ) 
   (ert-deftest test-bar () ... )
   (ert 'test-bar)
)

Neither module should know about the other.

Nor should there be a centralized module that has a list of all such tests, that get run at the end.

If only because there may be a failure partway through loading the file. And it is convenient to know what tests up to that point passed or failed.

(Whereas collecting and running all tests at the end loses that incremental test running ability.)

---++ Should be a macro

I hate repetition. E.g. in the stuff below, I hate having to type BAR three times.

(progn     
   ;; some blob of code
   (defun BAR (arg) ... ) 
   (ert-deftest test-BAR () ... )
   (ert 'test-BAR)
)

Therefore, of course I will create a macro that reduces this repetition. Although then folks looking for the ert-deftest will get confused...

---++ MEDIUM LENGTH SIMPLE EXAMPLE

If I have

  (ert-deftest test1 ()(should t))
  (ert 'test1)

I will get a nice *ert* buffer

Selector: test1
Passed:  1
Failed:  0
Skipped: 0
Total:   1/1

Started at:   2017-07-07 14:16:06-0700
Finished.
Finished at:  2017-07-07 14:16:07-0700

.

and if later I have

(ert-deftest test2 () (should t))
(ert 'test2)

It will erase the output of test1 in the *ert* buffer, and replace it with

Selector: test2
Passed:  1
Failed:  0
Skipped: 0
Total:   1/1

Started at:   2017-07-07 14:15:47-0700
Finished.
Finished at:  2017-07-07 14:15:47-0700

.

What I would like is to have the output of both tests concatenated:

Selector: test1
Passed:  1
Failed:  0
Skipped: 0
Total:   1/1

Started at:   2017-07-07 14:16:06-0700
Finished.
Finished at:  2017-07-07 14:16:07-0700

.

Selector: test2
Passed:  1
Failed:  0
Skipped: 0
Total:   1/1

Started at:   2017-07-07 14:15:47-0700
Finished.
Finished at:  2017-07-07 14:15:47-0700

.

Answer 1

As so often happens, the act of posting the question suggests the answer.

Although it is somewhat ugly, and certainly non-standard. So if anyone posts a concise standard-ish way, I will accept such an answer.

The IMHO ugly way:

When I do

(progn     
   ;; some blob of code
   (defun foo (arg) ... ) 
   (ert-deftest test-foo () ... )
   (ert 'test-foo)
)

I can add more code to manipulate the ert output buffer:

(progn     
   ;; some blob of code
   (defun foo (arg) ... ) 
   (ert-deftest test-foo () ... )
   (ert 'test-foo)
   ...append `*ert*' contents to `*ert-multiple-independent-tests*'
)

(Less to type if macro-fied)

This is obvious. I was hoping that the "automated self-test" arguments of `ert' might accomplish something similar.

ert is an alias for ‘ert-run-tests-interactively’ in ‘ert.el’.

(ert SELECTOR &optional OUTPUT-BUFFER-NAME MESSAGE-FN)

Run the tests specified by SELECTOR and display the results in a buffer.

SELECTOR works as described in ‘ert-select-tests’.
OUTPUT-BUFFER-NAME and MESSAGE-FN should normally be nil; they
are used for automated self-tests and specify which buffer to use
and how to display message.

Perfunctory investigation has so far yielded little.

I would love a better way, if only because it might have worked out the details of whether to do this in batch mode, etc.

Or if somebody has managed not just to concatenate the ert buffers from independent tests, but has managed to create a fused buffer that reports all concisely.