Makefile with dependency graph not known in advance

Question

I'm trying to create a makefile (GNU make) that does the following:

1. A script generates a bunch of files--filenames not known in advance.
2. Each one of these files is converted to a different file.
3. After all are converted, all of these files are combined into a single output file.

How do I create a makefile with a "bellcurve"-patterned dependency graph, where the intermediate source and target files are not known in advance?

Conceptually I'm doing the following:

combined.pdf: $(filter combined.pdf, $(wildcard *.pdf))
    cat *.pdf > combined.pdf

%.pdf: %.svg
    cp $^ $@

$(wildcard *.svg):
    # recipe is for simple example
    # actually the *.svg files are not known in advance
    echo a > a.svg
    echo b > b.svg
    echo c > c.svg

.PHONY: clean
clean:
    ${RM} *.svg *.pdf *.d

Of course this doesn't work: Make evaluates the targets and sources before it runs the target that actually creates the svg. Also, there's no way to make sure all svgs are converted before they are combined.

I realized I could create dependencies and include them into the makefile, but I had trouble getting this to work too:

.PHONY: clean

include deps.d

combined.pdf: deps.d
    cat *.pdf > combined.pdf

%.pdf: %.svg
    cp $^ $@


deps.d:
    @# recipe is for simple example
    @# actually the *.svg files are not known in advance
    echo a > a.svg
    echo b > b.svg
    echo c > c.svg
    @# we know what files exist now, so we can establish dependencies
    @# "a.pdf : a.svg"
    echo *.svg : > deps.d
    @# combined.pdf: a.pdf b.pdf c.pdf
    ls *.svg \
        | awk '{targetfn=$$0; sub(/\.svg$$/, ".pdf", targetfn); print targetfn, ":", $$0;}' \
        >> deps.d
    @# combined.pdf: a.pdf b.pdf c.pdf
    echo combined.pdf : $$(echo *.svg | sed -e 's/\.svg/\.pdf/g') >> deps.d

clean:
    ${RM} *.pdf *.svg *.d

However this still isn't connecting the dependency graph properly. When I run this, make quits as follows:

Makefile:3: deps.d: No such file or directory
echo a > a.svg
echo b > b.svg
echo c > c.svg
echo *.svg : > deps.d
ls *.svg \
        | awk '{targetfn=$0; sub(/\.svg$/, ".pdf", targetfn); print targetfn, ":", $0;}' \
        >> deps.d
echo combined.pdf : $(echo *.svg | sed -e 's/\.svg/\.pdf/g') >> deps.d
make: Nothing to be done for `a.svg'.

I still seem to have the problem that the make doesn't know about the rules in deps.d.

Also, this still doesn't solve the problem of building all the dependencies. I thought of using a marker file like this:

%.pdf: %.svg
    cp $^ $@
    @# if all svgs are converted, touch a target allpdfs
    if [ $(ls -1 *.svg | wc -l) -eq $(ls -1 *.pdf | grep -v combined\.pdf | wc -l) ]; touch allpdfs; fi

But there's no way to inform make that "allpdfs" may be created by this rule.

Answer 1

I'm surprised that moving the include directive makes a difference (what version of Make are you using?), but there is a simpler way. Your use of deps.d is in effect a recursive use of Make -- Make is arranging to execute itself a second time -- so we might as well make it official:

combined.pdf: ALL_SVGS
    $(MAKE) ALL_PDFS
    rm -f $@ # just in case it exists already
    cat *.pdf > $@

.PHONY: ALL_SVGS
ALL_SVGS:
    # recipe is for simple example                                              
    # actually the *.svg files are not known in advance                         
    echo a > a.svg
    echo b > b.svg
    echo c > c.svg

# These variables will be empty in the first execution of Make
SVGS = $(wildcard *.svg)
PDFS = $(patsubst %.svg,%.pdf,$(SVGS))

.PHONY: ALL_PDFS
ALL_PDFS: $(PDFS))

%.pdf: %.svg
    cp $^ $@

Answer 2

This isn't an answer exactly, because I don't know why this works, but I discovered that if I move the include directive after the target that creates the included file, everything works.

I.e. do this:

deps.d:
    ....

include deps.d

Because my deps.d includes enough dependency information, there's no need to have an intermediate target allpdfs file. Everything Just Works, even with make -j.

However, I don't know why this works. The include documentation isn't enlightening me.

UPDATE

I noticed the following note at the very bottom of the make manual discussing Automatic Prerequisites:

Note that the ‘.d’ files contain target definitions; you should be sure to place the include directive after the first, default goal in your makefiles or run the risk of having a random object file become the default goal. See How Make Works.

So what happened is that the first rule inside the generated deps.d became the default target, causing the mysterious premature completion of the build. So the solution is just to make sure include directives are not before your intended default target.

Answer 3

I was just working on this exact problem in a slightly different setting. Here is a clean solution - no need for recursion and such (and you can tweak the sed if you like):

include deps.d

combined.pdf:
    cat *.pdf > combined.pdf

%.pdf: %.svg
    cp $^ $@

deps.d:
    echo a > a.svg
    echo b > b.svg
    echo c > c.svg
    echo 'combined.pdf:' *.svg | sed 's/\.svg/\.pdf/g' > deps.d

Enjoy!