GNU Make: Batching multiple targets in single command invocation

1
votes

Consider the following setup:

$ touch 1.src 2.src 3.src
$ cat Makefile
%.dst: %.src
    @convert -o "$@" "$<"

We can compile our .src files into .dst files by running make 1.dst 2.dst 3.dst which calls the convert (just a placeholder) tool three times.

This setup is fine if there is little overhead in calling convert. However, in my case, it has a startup penalty of a few seconds for every single call. Luckily, the tool can convert multiple files in a single call while paying the startup penalty only once, i.e. convert -o '{}.dst' 1.src 2.src 3.src.

Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?
Edit: To be more precise, what feature I am looking for: Say that 1.dst is already newer than 1.src so it doesn't need to be recompiled. If I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.

A quick and dirty way would be creating a .PHONY rule that simply converts all src files to dst files but that way I would convert every src file each and every time. Further more, specifying dst files as prerequisites in other rules would also no longer be possible.

Thanks in advance!

2
makefilebuildgnu
I wouldn't do it this way. When a single file comes out from another single file, it can be run effectively in parallel when using make -j. Making multiple targets from multiple files in one step is clunky (as you can see from proposed answers), Makefile gets more complex and harder to maintain and probably you will hit different corner cases where it will become a bottleneck in a different way. I would consider whether reducing startup overhead is worth future maintenance. Maybe a simple parallel make would suffice instead? - raspy
I'm actually using make -j wherever I can. The issue with the convert tool is that it takes like 3 seconds to start up and less than a second to compile the source. Even if I run it in parallel, it spends a lot of time on the same startup process over and over again. I'm not sure if it's possible to reduce startup overhead. The tool is written in python and probably spends that startup time in the interpreter. - shilch

2 Answers

2
votes

If you have GNU make 4.3 or above, you can use grouped targets like this:

DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(_DST_FILES:.dst=.src)

all: $(DST_FILES)

$(DST_FILES) &: $(SRC_FILES)
        convert -o '{}.dst' $?
        @touch $(DST_FILES)

If your convert is only updating some of the targets then you need the explicit touch to update the rest.

Here's a way to do it with passing a goal on the command line that might work; change DST_FILES to:

DST_FILES := $(or $(filter %.dst,$(MAKECMDGOALS)),1.dst 2.dst 3.dst)
1
votes

Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?

It is possible, but messy, to write make rules for build steps that produce multiple targets with a single run of the recipe, such that the recipe is executed just once if any of the targets needs to be updated. However, you clarify that

[if] 1.dst is already newer than 1.src [, and] I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.

. That's a slightly different problem. You can use the $? automatic variable in a recipe to get the prerequisites that are newer than the rule's target, but for that to serve the purpose, you need a rule with a single target.

Here's one slightly convoluted way to make it work:

DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(DST_FILES:.dst=.src)

$(DST_FILES): dst.a
        ar x $< $@

dst.a: $(SRC_FILES)
        convert -o '{}.dst' $?
        x='$?'; ar cr $@ $${x//src/dst}

The dst.a archive serves as the one target with all the .src files as prerequisites, so as to provide a basis for use of $?. Additionally, it provides a workaround for the problem that whenever that target is updated, it becomes newer than all the then-existing .dst files: .dst files that are out of date with respect to the archive but not with respect to the corresponding .src file are extracted from the archive instead of being rebuilt from scratch.