Trying to compile Detours.lib as 32bit on a 64bit machine: LNK1112: module machine type 'x86' conflicts with target machine type 'x64'

Question

I'm trying to compile Detours per the instructions. I'd like to compile 32 bit versions of the library, since I plan to hook a 32-bit application. I've googled for answers but they seem to explain the obvious, like switch to 64-bit, but I intend on compiling for 32-bit.

I've used a mix following commands, trying to get it to work:

SET DETOURS_TARGET_PROCESSOR=X86
CALL "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvarsall.bat"  x86
CALL "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars32.bat"
NMAKE

However this results in the following errors:

1) During VCVARSALL:

[ERROR:team_explorer.bat] Directory not found : "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer"
[vcvarsall.bat] Environment initialized for: 'x86'

2) During NMAKE

LIBCMT.lib(chkstk.obj) : fatal error LNK1112: module machine type 'x86' conflicts with target machine type 'x64'

As you can see, despite my best efforts, it is complaining about the target machine type.

I have also tried vcvarsall x64, paired with SET DETOURS_TARGET_PROCESSOR=X86. (Not sure if this is a valid test) This resulted in

iping_d.obj : error LNK2001: unresolved external symbol iping_ProxyFileInfo

Please help me understand what I am doing wrong.

When I choose DETOURS_TARGET_PROCESSOR=X64, it compiles successfully, but it creates a 64-bit lib which I cannot use in my 32bit dll.

I've also tried all this from the VC x86 Native Tools Command Prompt, with the same results.

Answer 1

I've written up an answer to this question on the detours issue tracker, here: https://github.com/microsoft/Detours/issues/98.

Reposting the content here for convenience:

I think the issue here is that you are using the wrong configuration for cross compilation. To cross compile x86 binaries on a x64 host, you need to:

• Launch a "x64_x86 Cross Tools Command Prompt for VS" command prompt.
• set DETOURS_TARGET_PROCESSOR=X86

I think you might have accidentally launched a "x86_x64 Cross Tools Command Prompt for VS" command prompt. If I do this, and then set DETOURS_TARGET_PROCESSOR=X86, I can reproduce the same error. I imagine the same thing would happen if I did the inverse to cross compile for X64.

The naming is very confusing, but you can easily verify by checking the environment variables that are set in the command prompt.

x64_x86 Cross Tools Command Prompt:

C:\>set | findstr VSCMD_ARG
VSCMD_ARG_app_plat=Desktop
VSCMD_ARG_HOST_ARCH=x64
VSCMD_ARG_TGT_ARCH=x86

Note: VSCMD_ARGT_TGT_ARCH is equal to the architecture we are targeting, and should be equal to DETOURS_TARGET_PROCESSOR.

x86_x64 Cross Tools Command Prompt:

C:\>set | findstr VSCMD_ARG
VSCMD_ARG_app_plat=Desktop
VSCMD_ARG_HOST_ARCH=x86
VSCMD_ARG_TGT_ARCH=x64

We can see here that the the values are inverted from what we want.