How to force make to always update and re-read an included makefile?

Question

I have a Makefile that generates a makefile.mk, which is subsequently included in the main makefile. Because the set of input files (*.in in this example) can change at any time, I need the makefile.mk to be regenerated once for each make run.

The include documentation states:

Once it has finished reading makefiles, make will try to remake any [included makefiles] that are out of date or don’t exist.

And on How Makefiles Are Remade:

[A]fter reading in all makefiles, make will consider each as a goal target and attempt to update it. If a makefile has a rule which says how to update it [...], it will be updated if necessary. After all makefiles have been checked, if any have actually been changed, make starts with a clean slate and reads all the makefiles over again.

Based on that, I thought the following should work:

include makefile.mk

makefile.mk:
    echo "--- Begin makefile.mk ---"
    @( \
        shopt -q -s nullglob; \
        INPUTS=(*.in); \
        echo "all:" $${INPUTS//.in/.out}; \
        for f in $$INPUTS; do \
            echo "$${f/.in/.out}: $$f"; \
            echo -e '\tcp $$< $$@'; \
        done \
    ) | tee $@
    echo "--- End makefile.mk ---"

(I know I can use pattern rules and wildcards in this simple example. In the real thing, makefile.mk has more complicated dependencies and is generated by a Python script.)

So we drop this into an empty directory and run it:

$ ls
Makefile
$ make
--- Begin makefile.mk ---
all:
--- End makefile.mk ---
make: Nothing to be done for 'all'.

So far so good. What happens if we create an input file?

$ touch test.in
$ ls
Makefile  makefile.mk  test.in
$ make
make: Nothing to be done for 'all'.

Because makefile.mk has no dependencies, it doesn't get updated. Bad! How about forcing it to always be considered out of date, by marking it phony:

.PHONY: makefile.mk

After adding this line, indeed makefile.mk is being regenerated:

$ make
--- Begin makefile.mk ---
all: test.out
test.out: test.in
    cp $< $@
--- End makefile.mk ---
make: Nothing to be done for 'all'.

However, it seems it's not being re-read, because make claims there's nothing to be done for all even though its prerequisite, test.out, doesn't exist.

Running make a second time does the right thing:

$ make
--- Begin makefile.mk ---
all: test.out
test.out: test.in
    cp $< $@
--- End makefile.mk ---
cp test.in test.out

Why doesn't make restart and read the updated makefile.mk, as the documentation seems to promise?

I found this related answer, which suggests that .PHONY is to blame, and offers a solution like this:

makefile.mk: force
    ...

.PHONY: force
force:

However, if I try that, make gets into an infinite loop, remaking makefile.mk over and over.

Out of pure curiosity: what is it that makes generation by a Python script more advantageous than from inside make? Arithmetic? String processing? — Vroomfondel
@Vroomfondel Generating several targets from each file, depending on the name of the directory it's in, and with command line arguments in the recipe that depend on a part of the filename. It can all be done in make, which indeed I did at first, but it became an opaque mess. It's a lot more readable and maintainable in Python. — Thomas
I checked the code for GNU make. It seems intentional that if an included makefile is marked as PHONY then it's not going to be considered updated for the purposes of re-exec. This is a good thing in general, because a file marked PHONY is always out of date which means the makefile would perform an infinite recursion of remaking. However, it probably should be explicitly documented this way, which I don't believe that it is. — MadScientist
@MadScientist Thanks for looking this up! There is something in the docs about double-colon rules, but not about PHONY. It's a bit strange though: when marked PHONY, it does remake the included file, once, but then fails to re-read it and continues with the old version. — Thomas

Thomas Thomas · Accepted Answer · 2017-10-24T09:10:24

I see two ways to solve this:

Use a unique suffix for generated makefile, so that it never exists when make is run, like:

UNIQUE := $(shell date +%s.%N)
include makefile.mk.$(UNIQUE)
makefile.mk.$(UNIQUE): […]

Generate makefile.mk as before with force), but make its inclusion (and other parts of the makefile) conditional on some make variable, so that it is not performed by default. Guard the recipe for makefile.mk by the opposite condition. Then invoke make recursively with this variable to activate the inclusion and disable the rebuild rule.

Edit by Thomas: Using the built-in MAKE_RESTARTS variable solves the infinite loop quite effectively.

include makefile.mk

ifndef MAKE_RESTARTS
makefile.mk: .FORCE
    @( \
        shopt -q -s nullglob; \
        INPUTS=(*.in); \
        echo "all:" $${INPUTS//.in/.out}; \
        for f in $$INPUTS; do \
            echo "$${f/.in/.out}: $$f"; \
            echo -e '\tcp $$< $$@'; \
        done \
    ) | tee $@

.PHONY: .FORCE
.FORCE:
endif

How to force make to always update and re-read an included makefile?

1 Answers