I was having a similar problem, seeing the SGEN "incorrect format" error when building in VS or MSBuild from command line. My project is x64, but MSBuild insisted on using the 32-bit version of the tool. (Some of my peers work around this by building in VS 2015, but I have only VS 2017 installed and want to keep it that way.)
Looking at the diagnostic build output, it looks like SGEN is running from the directory named by its SdkToolsPath parameter (for me: C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\
). This is assigned from TargetFrameworkSDKToolsDirectory. Looking at the targets files, this comes from SDK40ToolsPath. And that is set from MSBuild's .config file.
I resolved this by editing C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\MSBuild.exe.config
(requires Admin privilege), setting the SDK40ToolsPath property using
<property name="SDK40ToolsPath" value="$([MSBuild]::GetRegistryValueFromView('HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\NETFXSDK\4.6.2\WinSDK-NetFx40Tools-x64', 'InstallationFolder', null, RegistryView.Registry32))" />
(Note: If you're looking for this path in the registry on a 64-bit OS, go to HKLM\SOFTWARE\WOW6432Node\Microsoft...)
The main change is, of course, x86 to x64 to use the 64-bit tools. I also changed the framework to be what we use (4.6.2). This may mean we can reliably only use tools for 64-bit projects and for this framework, with this change in place. Still, I hope this might help someone running into this issue. (I'm shocked and dismayed MSBuild doesn't automatically change the tools path based on Framework & Architecture.)