npm install using node-gyp works from CLI, fails from TeamCity

Question

I'm stuck with an unreliable build in TeamCity.

When executing an npm install for my project that indirectly uses node-gyp to compile a solution, crucially it only fails with when executed from the TeamCity agent.
[EDIT AFTER ANSWER] The context of execution is from Build Step that executes a custom build.ps1 which automates the build process - covering c#, npm projects, executes unit tests etc. This is to say it isn't a bunch of discrete steps in TeamCity but a single PowerShell command that can execute from any environment.
If I execute from the command line on the same server, same user that the TeamCity Agent runs under, it works fine.

These are the errors from the build agent...

MSBuild: C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe
Executing Build
configuration: debug
Restoring npm for src\my-corp-app

> [email protected] preinstall D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\scrypt
> node node-scrypt-preinstall.js


> [email protected] install D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak
> npm run rebuild || echo "Keccak bindings compilation fail. Pure JS implementation will be used."


> [email protected] rebuild D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak
> node-gyp rebuild


D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild ) 
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak\build\keccak.vcxproj(21,3): error MSB4019: The imported project "D:\Microsoft.Cpp.Default.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
gyp ERR! build error 
gyp ERR! stack Error: `msbuild` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:262:23)
gyp ERR! stack     at emitTwo (events.js:126:13)
gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Windows_NT 10.0.14393
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd D:\IR\my-corp-app\TcBuild\src\my-corp-app\node_modules\keccak
gyp ERR! node -v v8.12.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] rebuild: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] rebuild script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\teamcity_user\AppData\Roaming\npm-cache\_logs\2019-02-13T11_38_48_872Z-debug.log
"Keccak bindings compilation fail. Pure JS implementation will be used."

The work around is to execute the command line, it succeeds, then execute in TeamCity again and the recent artifacts from the command line execution make it skip this node-gyp process and the build succeeds. This workaround is getting tiresome.

I've unsucessfully followed the various googled suggestions for fixing "C:\Microsoft.Cpp.Default.props" was not found errors:

npm install --global --production windows-build-tools
Set / Clear VCTargetsPath

Have also compared the environment variables between my CLI prompt and TeamCity Agent when executing the build but the only differences are the expected additions for TeamCity JRE and build numbers.

Am currently stuck trying to determine what magic makes the install work from an interactive CLI as opposed to TeamCity agent running as a Windows Service

The complete log referred to in the above log has details like

0 info it worked if it ends with ok
1 verbose cli [ 'C:\\Program Files\\nodejs\\node.exe',
1 verbose cli   'C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli   'run',
1 verbose cli   'rebuild' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'prerebuild', 'rebuild', 'postrebuild' ]
5 info lifecycle [email protected]~prerebuild: [email protected]
6 info lifecycle [email protected]~rebuild: [email protected]
7 verbose lifecycle [email protected]~rebuild: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]~rebuild: PATH: C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\keccak\node_modules\.bin;D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\.bin;C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\keccak\node_modules\.bin;D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\.bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319\;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Amazon\cfn-bootstrap\;C:\ProgramData\chocolatey\bin;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\ManagementStudio\;C:\Program Files\dotnet\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn\;D:\apps\jdbc;C:\Program Files\Git\cmd;D:\apps\OpenSSL-Win64\bin;C:\Program Files\nodejs\;C:\Users\teamcity_user\AppData\Local\Microsoft\WindowsApps;C:\Users\teamcity_user\AppData\Roaming\npm;C:\Users\teamcity_user\.dotnet\tools
9 verbose lifecycle [email protected]~rebuild: CWD: D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\keccak
10 silly lifecycle [email protected]~rebuild: Args: [ '/d /s /c', 'node-gyp rebuild' ]
11 silly lifecycle [email protected]~rebuild: Returned: code: 1  signal: null
12 info lifecycle [email protected]~rebuild: Failed to exec rebuild script
13 verbose stack Error: [email protected] rebuild: `node-gyp rebuild`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\index.js:301:16)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at EventEmitter.emit (events.js:214:7)
13 verbose stack     at ChildProcess.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at ChildProcess.emit (events.js:214:7)
13 verbose stack     at maybeClose (internal/child_process.js:915:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
14 verbose pkgid [email protected]
15 verbose cwd D:\IR\my-corp-app\TcBuild\src\my-corp-app-node\node_modules\keccak
16 verbose Windows_NT 10.0.14393
17 verbose argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "rebuild"
18 verbose node v8.12.0
19 verbose npm  v6.4.1
20 error code ELIFECYCLE
21 error errno 1
22 error [email protected] rebuild: `node-gyp rebuild`
22 error Exit status 1
23 error Failed at the [email protected] rebuild script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

Does npm drop a builderror.log on the file system? It does for me, which lead me to find the error "error C2039: 'ForceSet': is not a member of 'v8::Object'" — Sam Rueby
I edited question to show the complete log - but there is no file with that exact name builderror.log. Half wondered if my %PATH% is too long, but its only 1700 chars and the limit is 2048 — fiat

fiat fiat · Accepted Answer · 2019-02-25T00:52:51

Finally solved this by adding an inserting extra (and redundant) PowerShell build step:

Runner Type: PowerShell

Script: SourceCode

Working directory: <the obvious folder>

Script Source: npm install

and it magically worked.

Now, of course I already was executing npm install from deep down inside my build automation build.ps1.

For some reason, the above works and the buried one doesn't. I can't explain why. Confirmed that the execution folder was correct and also used get-command npm from both contexts and the result was the same for both:

CommandType     Name     Version    Source
-----------     ----     -------    ------
Application     npm.cmd  0.0.0.0    C:\Program Files\nodejs\npm.cmd

npm install using node-gyp works from CLI, fails from TeamCity

1 Answers