Always regenerate Sphinx documents containing a specific directive

Question

Sphinx usually incrementally builds the documentation which means that only files that have been changed will be regenerated. I am wondering if there is a way to tell Sphinx to always regenerate certain files which may not directly have been changed but are influenced by changes in other files. More specific: Is there a way to tell Sphinx to always regenerate files that contain a certain directive? The documentation I am working on relies on the possibility to collect and reformat information from other pages with the help of directives quite frequently. A clean (make clean && make [html]) and/or full (sphinx-build -a) build takes significantly longer than an incremental build. Additionally, manually keeping track of files which contain the directive might be complicated. The documentation is written by 10+ authors with limited experience in writing Sphinx documentation.

But even in less complex scenarios you might face this 'issue': For instance sphinx.ext.todo contains a directive called todolist which collects todos from the whole documentation. If I create a file containing all the todos from my documentation (basically an empty document just containing the todolist directive) the list is not updated until I make a clean build or alter the file.

If you want to test it yourself: Create a documentation with sphinx-quickstart and stick to the default values except for

'> todo: write "todo" entries that can be shown or hidden on build (y/n) [n]: y'

Add a file in source called todos.rst and reference this file from index.rst.

Content of the index.rst:

Welcome to sphinx-todo's documentation!
=======================================

.. toctree::
   :maxdepth: 2

   todos


.. todo::
 I have to do this


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

Content of todos.rst:

.. _`todos`:

List of ToDos
=============

.. todolist::

Assuming you use the html output you will notice that todos.html will not change when you add todos to index.html.

tl;dr: How -- if possible -- do I include files containing a specific directive (e.g. todolist) into an incremental build of Sphinx without the need of manually keeping track of them?

Answer 1

By default Sphinx will only update output for new or changed files. This is buried under sphinx-build -a.

At the end of the documentation of command options for sphinx-build:

You can also give one or more filenames on the command line after the source and build directories. Sphinx will then try to build only these output files (and their dependencies).

You could either invoke sphinx-build directly or through your makefile, depending on the makefile that shipped with your version of Sphinx (you can customize the makefile, too).

Answer 2

Just for the record: I benchmarked several solutions.

I created a function called touch_files in my conf.py. It searches for strings in files and -- if found -- touches the file to trigger a rebuilt:

def touch_files(*args):
    # recursively search the 'source' directory
    for root, dirnames, filenames in os.walk('.'):
        # check all rst files
        for filename in fnmatch.filter(filenames, '*.rst'):
            cur = os.path.join(root, filename)
            f = open(cur)
            # access content directly from disk
            s = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
            if any(s.find(d) != -1 for d in args):
                # if a string pattern has been found alter the edit 
                # time of the file
                os.utime(cur, None)
            f.close()

# actually call the function
touch_files('.. todolist::')

touch_files can be called with a variable amount of arguments and will edit a file when ONE of the arguments has been found. I tried to optimize the function with regular expression but this did not achieve much. Reading the file content directly from the disk with mmap seemed to have a minor impact.

This is the result of 78 files total from which 36 contain one of two directives.

Command                                 Time     Comment
time make html                          2.3 s    No changes
time sh -c 'make clean && make html'   13.3 s
time make htmlfull                      9.4 s    sphinx-build -a    
time make html                          8.4 s    with 'touch_files'
'touch_files'                           0.2 s    tested with testit

Result: Every command has been called just a few times (except 'touch_files') and therefore lack statistical reliability. Sphinx requires roughly 2.3 seconds to check the documentation for changes without doing anything. A clean build requires 13.3 seconds which is much longer than a build with sphinx-build -a. If we just rebuilt 36 out of 78 files, the built process is slightly faster, although I doubt a significant difference could be found here. The overhead of 'touch_files' is rather low. Finding the strings is quite cheap compared to editing the timestamps.

Conclusion: As Steve Piercy pointed out, using sphinx-build -a seems to be the most reasonable approach. At least for my use case. Should a file which does not contain a directive in question result in long building times touch_files might be useful though.