gnu-make finds the wrong object file during compilation

Question

First let me describe my folder structure

| SourceCode folder
| | Build folder
| | | makefile 
| | | buildrules.mak

| | Integration_Tests folder
| | | Subsystem_1_Test folder
| | | | buildfiles folder
| | | | test.cpp
| | | | makefile

| | Subsystem_1 folder
| | | buildfiles folder
| | | source1.cpp
| | | source2.cpp
| | | source3.cpp
| | | makefile

| | Subsystem_2 folder
| | | buildfiles folder
| | | source4.cpp
| | | source5.cpp
| | | source6.cpp
| | | makefile

So the makefile in the build folder will build all the subsystems and integration testing by calling the lower makefile. Each subsystem creates an executable and each subsystem tests creates an executable. The buildfiles folder stores the dependency files, object files and the final executable code. This is all populated by the makefiles as they are built.

The Integration tests are built using the Subsystem code under tests and builds the subsystem as well, but locally in the subsystem folder. The makefile for each subsystem integration test uses vpath to set the path of the subsystem folder. ie, for subsystem 1 test the vpath is:

vpath = ../../Subsystem_1

I don't have the exact code in front of me but the compile line is basically defined as the following, which lives in buildrules.mak and used by all the makefile

./buildfiles/%.o : %.cpp
     compile the code

So the issue I am seeing is when I build my integration test, while it should technically build and link fine, it seems to be finding object files in Subsystem_1/buildfiles folder and saying that the subsystem code is built and only builds test.cpp.

This is an error because testing has build defines defined and it needs to build the subsystem again for proper testing.

Now if subsystem 1 had not been built, ie no object files, then my integration tests will build the files as expected in put the object files in the local integration test buildfiles folder.

I believe this is due to the vpath being set in the integration test make file. It seems to search the subsystem_1 folder for object files and unluckily finds them.

If I'm building the Subsystem_1 folder and the test had already been built I do not see any issues, this is presumably because the vpath in Subsystem_1 does not have the path to the test, which it shouldn't.

I can't seem to figure out how to resolve this issue which in essence is a file path prob. It seems I need to be more specific on which buildfiles folder I need to match in the compile line, but I do not have the path.

I cannot user $(CURDIR) because that is an absolute path and I had been told that I have to use relative paths. I guess there are issues with paths having spaces and parenthesis. I could probably escape them before using it, but have been told that I must do everything with relative paths.

I have a variable in buildrules.mak that is the relative path from the directory that executes make to buildrules.mak. This was accomplished using:

BUILD_FOLDER_PATH := $(dir $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST)))

Is there a way to parse $(CURDIR) to get me to a common folder like sourcecod? Then could do something like:

$(BUILD_FOLDER_PATH)/../[path extracted from $(CURDIR)]/buildfiles

So if $(CURDIR) was c:/a/b/sourcecode/Integration_Tests/Subsystem_Test then I could extract "/Integration_Tests/Subsystem_Test" out of it to put into the above path.

If there is another way to do this I am open to suggestions as well.

Thanks!

EDIT: Test Makefile

# used when make is executed without any arguments, must be first thing in the file
all: default

##########################################################################
#                                                                        #
# Source Files                                                           #
#                                                                        #
##########################################################################

# These files are not shared with other processes
CPP_SRC := \
cmdclientmq.cpp \
cmdservermq.cpp \
ConvertUTF.cpp \
crcutil.cpp \
deka_cpu.cpp \
deka_debug.cpp \
deka_file.cpp \
deka_mutex.cpp \
deka_sem.cpp \
deka_serial.cpp \
deka_sharedMem.cpp \
deka_thread.cpp \
deka_time.cpp \
deka_timer.cpp \
encryptionutils.cpp \
exec_client.cpp \
logclient.cpp \
procstatus.cpp \
syserror.cpp

# These files are shared with other processes
EXPORTED_SRC :=

# These files are analyzed by the manifest tool
# Not all header files are listed because the manifest tool automatically checks if there is a .h file
# with the same name as a .cpp file
MANIFEST_SRC := \
$(CPP_SRC) \
$(EXPORTED_SRC)

##########################################################################
#                                                                        #
# makefile includes                                                      #
#                                                                        #
##########################################################################

# Basic compiler definitions, must be listed after source files
include ../../build/buildrules.mak

##########################################################################
#                                                                        #
# build flags                                                            #
#                                                                        #
##########################################################################

# Position Independent Code flag
GCC_FLAGS += -fPIC

##########################################################################
#                                                                        #
# Subsystem build paths                                                  #
#                                                                        #
##########################################################################

# directory include paths need for files external to the current subsystem
INCDIRS := \
$(EXECUTIVE_INC_PATH) \
$(LOGGER_INC_PATH) \
$(OS_INTERFACE_INC_PATH) \
$(SYSERROR_INC_PATH) \
$(SYSTEMDEFS_INC_PATH) \
$(UTILITIES_INC_PATH)

# list of directories in the current subsystem
VPATH := \
$(EXECUTIVE_PATH)/exec_client \
$(EXECUTIVE_PATH)/exec_core \
$(LOGGER_PATH)/main \
$(OS_INTERFACE_PATH) \
$(SYSERROR_PATH) \
$(UTILITIES_PATH)

##########################################################################
#                                                                        #
# Targets                                                                #
#                                                                        #
##########################################################################

# default target to build and link the subsystem code
default: $(LIB_DIR)/libpycommon.a
     @echo 
     @echo Common Library default: Done.
     @echo ----------------------------------------------------------------------------
     @echo

# remove the subsystem build folders and create the neccessary directories
clean:  REMOVE_INTEGRATION_DIRS CREATE_INTEGRATION_OUTPUT_DIRS
    @echo
    @echo Common Library clean: Done.
    @echo ----------------------------------------------------------------------------
    @echo

##########################################################################
#                                                                        #
# Build Rules                                                            #
#                                                                        #
##########################################################################

# Do not build archives or call ranlib in parallel:
#  http://www.gnu.org/software/make/manual/make.html#Archive-Pitfalls
.NOTPARALLEL:
$(LIB_DIR)/libpycommon.a: $(LIB_DIR)/libpycommon.a($(CPP_OBJS))
              $(RANLIB) $(LIB_DIR)/libpycommon.a

buildrules.mak

# get relative path to the current makefile by examining the $(MAKEFILE_LIST), which is
# dynamically updated.  Determine the number of words in $(MAKEFILE_LIST) and grab the
# last word.  Then extract the directory path up to the found last word.
# the include order is important, do not adjust
include $(dir $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST)))toolset.mak
include $(dir $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST)))common.mak

##########################################################################
#                                                                        #
# Build rules for creating subsystem and project folder structure        #
#                                                                        #
##########################################################################
CREATE_SUBSYSTEM_OUTPUT_DIRS :
    -@echo Creating directory: $(BUILD_PATH)
    -@mkdir -p $(BUILD_PATH)
    -@echo Creating directory: $(DEP_DIR)
    -@mkdir -p $(DEP_DIR)
    -@echo Creating directory: $(OBJ_DIR)
    -@mkdir -p $(OBJ_DIR)
    -@echo Creating directory: $(LINT_DIR)
    -@mkdir -p $(LINT_DIR)
    -@echo Creating directory: $(CCK_DIR)
    -@mkdir -p $(CCK_DIR)
    -@echo Creating directory: $(UNCRUST_DIR)
    -@mkdir -p $(UNCRUST_DIR)
    -@echo Creating directory: $(MET_DIR)
    -@mkdir -p $(MET_DIR)

REMOVE_SUBSYSTEM_DIRS :
    -@echo ----------------------------------------------------------------------------
    -@echo Removing directory: $(BUILD_PATH)
    -@echo ----------------------------------------------------------------------------
    -@rm -rf $(BUILD_PATH)
    -@echo

CREATE_PROJECT_OUTPUT_DIRS :
    -@echo Creating directory: $(IMAGE_DIR)
    -@mkdir -p $(IMAGE_DIR)
    -@echo Creating directory: $(PROJECT_OBJ_DIR)
    -@mkdir -p $(PROJECT_OBJ_DIR)

REMOVE_PROJECT_OUTPUT_DIRS :
    -@echo Removing directory: $(DIALYSIS_DIR)
    -@rm -rf $(DIALYSIS_DIR)
    -@echo Removing directory: $(ETC_DIR)
    -@rm -rf $(ETC_DIR)
    -@echo Removing directory: $(IMAGE_DIR)
    -@rm -rf $(IMAGE_DIR)
    -@echo Removing directory: $(PROJECT_OBJ_DIR)
    -@rm -rf $(PROJECT_OBJ_DIR)
    -@echo Removing directory: $(TAR_DIR)
    -@rm -rf $(TAR_DIR)
    -@echo Removing directory: $(QUAL_DIR)
    -@rm -rf $(QUAL_DIR)
    -@echo Removing directory: $(PYTHON_OUT)
    -@rm -rf $(PYTHON_OUT)

##########################################################################
#                                                                        #
# Integration Build Rules                                                #
#                                                                        #
##########################################################################
CREATE_INTEGRATION_OUTPUT_DIRS :
    -@echo Creating directory: $(BUILD_PATH)
    -@mkdir -p $(BUILD_PATH)
    -@echo Creating directory: $(DEP_DIR)
    -@mkdir -p $(DEP_DIR)
    -@echo Creating directory: $(OBJ_DIR)
    -@mkdir -p $(OBJ_DIR)
    -@echo Creating directory: $(LIB_DIR)
    -@mkdir -p $(LIB_DIR)
    -@echo Creating directory: $(PYTHON_OUT)
    -@mkdir -p $(PYTHON_OUT)

REMOVE_INTEGRATION_DIRS :
    @echo Removing directory: $(BUILD_PATH)
    @rm -rf $(BUILD_PATH)
    @echo

##########################################################################
#                                                                        #
# Unit Test Build Rules                                                  #
#                                                                        #
##########################################################################
CREATE_UNIT_TEST_OUTPUT_DIRS :
    -@echo Creating directory: $(BUILD_PATH)
    -@mkdir -p $(BUILD_PATH)
    -@echo Creating directory: $(DEP_DIR)
    -@mkdir -p $(DEP_DIR)
    -@echo Creating directory: $(OBJ_DIR)
    -@mkdir -p $(OBJ_DIR)

REMOVE_UNIT_TEST_DIRS :
    @echo Removing directory: $(BUILD_PATH)
    -@rm -rf $(BUILD_PATH)
    @echo

##########################################################################
#                                                                        #
# Create dependancy files from all cpp files                             #
#                                                                        #
##########################################################################

# convert the file extensions from .o to .d and strip the relative path and add the dependancy path
DEP_OBJS = $(patsubst %.o, $(DEP_DIR)/%.d, $(Call_OBJ))

##########################################################################
#                                                                        #
# Create object files from all cpp files                                 #
#                                                                        #
##########################################################################

# convert the file extensions from .cpp to .o
Cxx_OBJS = $(CPP_SRC:.cpp=.o)

# strip the relative path and add the object path
CPP_OBJS = $(patsubst %.o, $(OBJ_DIR)/%.o, $(Cxx_OBJS))

##########################################################################
#                                                                        #
# Create object files from all exported files                            #
#                                                                        #
##########################################################################

# Files needed by other subsystems
# convert the file extensions from .cpp to .o
Exx_OBJS = $(EXPORTED_SRC:.cpp=.o)

# strip the relative path and add the path to the subsystem object folder
EXPORTED_OBJ = $(patsubst %.o, $(OBJ_DIR)/%.o, $(Exx_OBJS))

# strip the relative path and add the project object folder
EXPORTED_OUT = $(patsubst %.o, $(PROJECT_OBJ_DIR)/%.o, $(Exx_OBJS))

##########################################################################
#                                                                        #
# List of files within the subsystem                                     #
#                                                                        #
##########################################################################

# create the list of all files associated with this subsystem
Call_OBJ = $(Cxx_OBJS) $(Exx_OBJS)

##########################################################################
#                                                                        #
# Files needed by Lint                                                   #
#                                                                        #
##########################################################################

# convert the file extensions from .o to .lob and strip the relative path and add the lint path
LINT_OBJS = $(patsubst %.o, $(LINT_DIR)/%.lob, $(Call_OBJ))

##########################################################################
#                                                                        #
# Interface Rules to create dependancy files from project source code    #
#                                                                        #
##########################################################################

# function used to create dependancy files
define DEPENDS_COMMAND
@echo
@echo ----------------------------------------------------------------------------
@echo -Depend: $@
@echo ----------------------------------------------------------------------------
@$(CC_DEP) -MM -MT '$(OBJ_DIR)/$*'.o $(INCDIRS) $< > $ [email protected]
@$(SED) 's,\($*\)\.o[ :]*,\1.o $(LINT_DIR)/\1.lob $@ : ,g' < [email protected] > $@;
@rm -f [email protected]
endef

# rule to create dependency for C++ code
$(DEP_DIR)/%.d: %.cpp
    $(DEPENDS_COMMAND)

##########################################################################
#                                                                        #
# Interface Rules to compile project source code                         #
#                                                                        #
##########################################################################

# function used to compile code
define COMPILE_COMMAND
@echo
@echo ----------------------------------------------------------------------------
@echo -Compiling: $@
@echo ----------------------------------------------------------------------------
@$(CC) $(CPU_GCC_FLAGS) $(GCC_FLAGS) $(INCDIRS) $< -o $@
endef

# build object files in subsytem ojbect folder for all cpp files
$(OBJ_DIR)/%.o: %.cpp
    @$(COMPILE_COMMAND)

##########################################################################
#                                                                        #
# Build exported object files and copy them to the build folder          #
#                                                                        #
##########################################################################

# move shared object files to the proper directory
$(EXPORTED_OUT): $(PROJECT_OBJ_DIR)/%.o : $(OBJ_DIR)/%.o
         @echo
         @echo Copying Shared File $< to $@ ...
         @cp  $<  $@

##########################################################################
#                                                                        #
# Build Rules to link files to create the final executable               #
#                                                                        #
##########################################################################

# Commands for linking final executable [LINK_DB_LIBS vs LINK_LIBS]
define LINK_EXECUTABLE
@echo
@echo "**********************************"
@echo "****  Linux Linking $@"
@echo "**********************************"
$(CC) $^ $(LINK_LIBS) $(LIB_FLAGS) -o $(IMAGE_DIR)/$@
@$(DEKA_TOOLS)/imagecrc.exe $(IMAGE_DIR)/$@ 2>&1
@echo
@echo LINK_EXECUTABLE Completed
@echo ----------------------------------------------------------------------------
endef

# Commands for linking final executable for subsystems using the database [LINK_DB_LIBS vs LINK_LIBS]
define LINK_DB_EXECUTABLE
@echo
@echo "********************************************"
@echo "****  Linux Linking $@"
@echo "********************************************"
@$(CC) $^ $(LINK_DB_LIBS) $(LIB_FLAGS) -o $(IMAGE_DIR)/$@
@$(DEKA_TOOLS)/imagecrc.exe $(IMAGE_DIR)/$@ 2>&1
@echo
@echo LINK_DB_EXECUTABLE Completed
@echo ----------------------------------------------------------------------------
endef

define LINK_EXECUTABLE_INTEGRATION
@echo
@echo "**********************************"
@echo "****  Linux Linking $@"
@echo "**********************************"
$(CC) $^ $(LINK_LIBS) $(LIB_FLAGS) -o $(PYTHON_OUT)/$@
@$(DEKA_TOOLS)/imagecrc.exe $(PYTHON_OUT)/$@ 2>&1
@echo ----------------------------------------------------------------------------
endef

define LINK_EXECUTABLE_UNIT_TEST
@echo
@echo "**********************************"
@echo "****  Linux Linking $@"
@echo "**********************************"
$(CC) $^ $(LINK_LIBS) $(LIB_FLAGS) $(CANTPP_LIB) -L$(GCC_LIB_PATH) -o $(OBJ_DIR_UT)/$@
@echo ----------------------------------------------------------------------------
endef

##########################################################################
#                                                                        #
# Build rules for Lint of C++ files                                      #
#                                                                        #
##########################################################################

# function used to lint C++ files
define LINT_COMMAND
@echo
@echo ----------------------------------------------------------------------------
@echo -Lint: $<
@echo ----------------------------------------------------------------------------
-@$(LINT) -dRELEASE_BUILD -u -b $(LINT_SYS_DIRS) $(INCDIRS) $(LINT_RULES) '-passes(2)' '-oo($@)' '-os($(@D)/$(*F).txt)' $<
@$(PYTHON) $(LINT_PYTHON_SCRIPT) --out $(@D)/$(*F).txt --src $(CURDIR)/$<
endef

# rule to create lint output for C++ code
$(LINT_DIR)/%.lob: %.cpp
    @$(LINT_COMMAND)

lint:   $(LINT_OBJS)
    @echo
    @echo lint: Done.
    @echo ----------------------------------------------------------------------------

lintclean:
    @rm -rf $(LINT_DIR)
    @mkdir $(LINT_DIR)
    @echo
    @echo lintclean: Done
    @echo ----------------------------------------------------------------------------

##########################################################################
#                                                                        #
# Build rules for Uncrustify of C++ files                                #
#                                                                        #
##########################################################################
uncrustify: $(UNCRUST_DIR)
    @python $(UNCRUST_PYTHON_FILE) $(UNCRUST_EXECUTABLE) -nooverwrite
    @echo
    @echo uncrustify: Done
    @echo ----------------------------------------------------------------------------

uncrustifyclean:
    @rm -rf $(UNCRUST_DIR)
    @mkdir $(UNCRUST_DIR)
    @echo
    @echo uncrustifyclean: Done
    @echo ----------------------------------------------------------------------------

##########################################################################
#                                                                        #
# Build rules for Understand CodeCheck of C++ files                      #
#                                                                        #
##########################################################################
codecheck.udb: $(CCK_DIR)
    @echo
    @cp -rf $(UND_RULES_ARCHIVE) $(UND_RULES)
    @$(UND) create -db $@ -languages c++
    @$(UND) -db $@ settings -c++MacrosAdd RELEASE_BUILD=1
    @$(UND) add $(CURDIR) -db $@
    -@$(UND) remove unittest -db $@
    @$(UND) analyze -all -db $(CURDIR)/$@ > $(CCK_DIR)/analyze.log 2>&1
    @$(UND) -db $(CURDIR)/$@ codecheck -html $(CCK_RULES_FILE) $(CCK_DIR) > $(CCK_DIR)/codecheck.log 2>&1
    @$(UPERL) $(UND_PERL_SCRIPT) $(CURDIR) -db $@
    @rm $@
    @echo
    @echo codecheck: Done
    @echo ----------------------------------------------------------------------------

codecheckclean:
    @rm -rf $(CCK_DIR)
    @mkdir $(CCK_DIR)
    @echo
    @echo codecheckclean: Done
    @echo ----------------------------------------------------------------------------

##########################################################################
#                                                                        #
# Build rules for Metrics/Complexity of C++ files using Understand       #
#                                                                        #
##########################################################################
metric.udb: $(MET_DIR)
    @echo
    @$(UND) create -db $@ -languages c++
    @$(UND) -db $@ settings -c++MacrosAdd RELEASE_BUILD=1
    @$(UND) settings -metricsShowDeclaredInFile on -db $@
    @$(UND) settings -metrics MaxNesting CyclomaticModified -db $@
    @$(UND) add $(CURDIR) -db $@
    -@$(UND) remove unittest -db $@
    @$(UND) analyze -all -db $@ > $(UND_METRICS_LOG) 2>&1
    @$(UND) metrics -db $@
    -@mv -f metric.csv $(MET_DIR)/
    @rm $@
    @echo
    @echo metrics: Done
    @echo ----------------------------------------------------------------------------

metricclean:
    @rm -rf $(MET_DIR)
    @mkdir $(MET_DIR)
    @echo
    @echo metricclean: Done
    @echo ----------------------------------------------------------------------------

##########################################################################
#                                                                        #
# Build Rules to create a manifest of static analysis tool results       #
#                                                                        #
##########################################################################
manifestall: manifestclean uncrustifyclean lintclean codecheckclean metricclean uncrustify codecheck.udb metric.udb
    @$(MAKE) -j $(NUMBER_OF_PROCESSORS) lint
    @python $(MANIFEST_SCRIPT_LOCATION) --cpp $(MANIFEST_SRC) --searchpath $(VPATH)
    @echo
    @echo manifestall: Done
    @echo ----------------------------------------------------------------------------

manifest:
    @python $(MANIFEST_SCRIPT_LOCATION) --cpp $(MANIFEST_SRC) --searchpath $(VPATH)
    @echo manifest: Done
    @echo ----------------------------------------------------------------------------

manifestclean:
    @rm -f $(CURDIR)/manifest.csv
    @echo manifestclean: Done
    @echo ----------------------------------------------------------------------------


##########################################################################
#                                                                        #
# Build rules for PyLint of Python files                                 #
#                                                                        #
##########################################################################

# TODO in the future

# rule to create lint output for Python code
#$(PYLINT_DIR)/%.pylint: %.py
#   @echo ----------------------------------------------
#   @echo -PyLint: $<
#   @echo ----------------------------------------------
#   @$(PYTHON) $(PYLINT) --include-ids=y --rcfile=$(PYLINT_CONFIG) $< > $@

##########################################################################
#                                                                        #
# Build rules for PEP8 (CodeCheck) of Python files                       #
#                                                                        #
##########################################################################

# TODO in the future

# rule to create codecheck for Python code
#$(CCK_DIR)/%.pep8: %.py
#   @echo ----------------------------------------------
#   @echo PEP8: $<
#   @echo ----------------------------------------------
#   @$(PYTHON) $(PEP) --statistics --show-source --repeat $< > $@


##########################################################################
#                                                                        #
# Dependancy File Creation                                               #
#                                                                        #
##########################################################################

# alwawys try to create a dependancy file unless $(MAKECMDGOALS) is one of the listed targets
ifneq ($(MAKECMDGOALS),clean)
ifneq ($(MAKECMDGOALS),lint)
ifneq ($(MAKECMDGOALS),lintclean)
ifneq ($(MAKECMDGOALS),uncrustify)
ifneq ($(MAKECMDGOALS),uncrustifyclean)
ifneq ($(MAKECMDGOALS),codecheck.udb)
ifneq ($(MAKECMDGOALS),codecheckclean)
ifneq ($(MAKECMDGOALS),metric.udb)
ifneq ($(MAKECMDGOALS),metricclean)
ifneq ($(MAKECMDGOALS),manifestall)
ifneq ($(MAKECMDGOALS),manifest)
ifneq ($(MAKECMDGOALS),manifestclean)
include $(DEP_OBJS)
endif
endif
endif
endif
endif
endif
endif
endif
endif
endif
endif
endif

your stated vpath doesn't explain the error. It would help if you showed us Subsystem_1_Test/makefile and buildrules.mak. — Beta
Your buildrules.mak includes a toolset.mak and a common.mak; how many files are in this tree, and how big are they in total? (Judging by the mixture of typos and superfluous lines, I'd say this is an amalgam of hand-written Make code and either generated or copied-and-pasted Make code, built up over a long period of time -- the technical term is "Big Ball of Mud" -- so fixing it may be difficult.) — Beta
there are hundreds of source files in probably 50ish various folders. getting the source code out of version controls is about 1.5GB. toolset.mak only has paths to all of our tools and common.mak is just pathing around the source tree to get to various folders. — Brian

Beta Beta · Accepted Answer · 2012-11-01T16:45:56

"Pathing around the source tree to get to various folders" is exactly the problem.

I suggest you try adding a line to buildrules.mak. Near the top, right after the two include statements, put in

OBJ_DIR := buildfiles

This is a hack. If it works, don't leave it at that, this makefile system needs a lot of cleaning up. If it doesn't work, revert, post common.mak and we'll try a gentler approach.

gnu-make finds the wrong object file during compilation

1 Answers