GNU Parallel running Makefiles with different optimization levels

Question

There is not a optimal -O level. My approach in order to find the fastest execution for my particular code is to compile the same code with usual optimization levels (i.e. -O0, -Ofast, -O1, -O2, -O3,-march=native) and check which flags produce me the fastest execution (with time).

So, there is a way to check all optimization levels (listed before) running a Makefile with each optimization (-O level)?

I think that Gnu Parallel could run the Makefile changing the -O level but I don't know how figure it out?

Thanks in advance.

Could be an option but how edit CCFLAGS recursively to test all optimization levels using a unique Makefile? GnuParallel could be helpful but I dont know how achieve this task efficiently. — Bruce_Warrior
for flag in -O0 -Ofast -O1 -O2 -O3 -march=native; do make CFLAGS+="$flag"; done? — Etan Reisner
The time command is not a very good indicator of program's speed. ncsu.edu/hpc/Documents/sprofile.php — Jeremiah Dicharry
-Ofast -march=native should give the fastest program eventhough you're disregarding "strict standard compliance" i.e. -ffast-math. gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/… — Jeremiah Dicharry

Peter Cordes Peter Cordes · Accepted Answer · 2015-07-31T21:54:46

You want to use GNU Parallel to do multiple builds in parallel? You'd need separate build directories at least, and a more complex build setup if you want to avoid copying the whole directory of source code. If you try to do multiple separate builds in the same directory at the same time, some object files will be built with one set of CFLAGS, and others with others.

Use @Etan's loop suggestion:

NJOBS=$(getconf _NPROCESSORS_ONLN)  # adjust as desired
for flag in -O{0..3} -O{3,fast}" -march=native"; do
    make clean
    make -j"$NJOBS" CFLAGS+="$flag -fprofile-generate"
    ./a.out  # feed it some input that exercises different options and code paths
    make clean
    make -j"$NJOBS" CFLAGS+="$flag -fprofile-use"
    perf stat ./a.out | tee "perfstat$flag.txt"
done

Note the use of make -j for parallelism, rather than GNU parallel. Also note the use of profile-guided optimization. x264 has a build system with a make fprofiled target for building a PGO executable, which takes care of the build / run / rebuild cycle. So it's possible, but IDK if it makes their Makefile messy.

You could use GNU parallel for the timing run of your code, but you'll get more consistent results if you do the timing run on an idle machine.

If you want to test how your code does when there are multiple copies of it running at the same time, competing for cache space and memory bandwidth (or even execution resources with hyperthreading), then test that with multiple copies of the same code, not having some runs competing with gcc, some with -O0, and some with -O3.

As far as optimization options, you will normally get the best results from gcc with -fprofile-generate and -fprofile-use options. Clang can also do profile-guided optimization, using the same options, or using data from CPU perf counters. (The manual describes using a tool to convert Linux perf record data into something Clang can use.)

Some gcc optimizations are only enabled with -fprofile-use (or manually, not with just -O3). e.g. -funroll-loops can help in some tight loops. Don't use for everything, because larger code size can lead to I-cache misses overall in the whole program, which outweigh gains from reducing loop overhead in some hot loops.

GNU Parallel running Makefiles with different optimization levels

1 Answers