Using NSIS to run 32 bit java keytool on 64 bit server

Question

Our installer generates a keystore using the jdk keytool. We recently upgraded to jdk 7 and since then have had issues with 32/64 bit integration, specifically with attempting to install the 32 bit version on a 64 bit server (don't ask why we're doing this; I already lost that battle). The problem is that when attempting to run keytool from the installer, it fails to generate a keystore, and throws the error

java.lang.InternalError: Should not get here.

First few lines of stack trace:

     at sun.nio.fs.WindowsNativeDispatcher.GetFinalPathNameByHandle(Native Method)
     at sun.nio.fs.WindowsLinkSupport.getFinalPath(WindowsLinkSupport.java:77)
     at sun.nio.fs.WindowsLinkSupport.getRealPath(WindowsLinkSupport.java:242)
     at sun.nio.fs.WindowsPath.toRealPath(WindowsPath.java:836)
     at sun.nio.fs.WindowsPath.toRealPath(WindowsPath.java:44)

Running this line:

nsExec::ExecToLog '"$InstallationRoot\$ConfigName\jdk\bin\keytool.exe" -genkey -keystore "$InstallationRoot\$ConfigName\tomcat\tomcat.jks" -alias tomcat -keyalg RSA -validity 109500 -storepass password -keypass password -dname CN=$server,OU=Syseng,O=Company,L=City,ST=State,C=US'

The variables are all valid and used throughout the installer. I tried a DetailPrint to output the exact line it's running, then copied and pasted the output to the command line and it worked.

"C:\TempInstallDir\Inst1\jdk\bin\keytool.exe" -genkey -keystore "C:\TempInstallDir\Inst1\tomcat\tomcat.jks" -alias tomcat -keyalg RSA -validity 109500 -storepass password -keypass password -dname CN=SANDBOX,OU=Syseng,O=Company,L=City,ST=State,C=US

I also tried using Exec and ExecWait, in case there was some weirdness with nsExec.
I tried writing the keytool command to a batch file then calling that from the installer. Again, that gave the same exception no matter how called from the installer, but if I just went to the command line and ran that same batch file, it worked.
I tried forcing it to run in a 32-bit command prompt using %WINDIR%\SysWOW64\cmd.exe /C.

I'm basically out of ideas here - I can reliably make it fail by running it in the installer and reliably make it work by running the exact same command directly from the command line. It also works fine running from a 32-bit server or running the 64-bit installer on a 64-bit server.

Also, I'm running the installer as an admin; otherwise there are other steps that would be failing. This is occurring on both Windows Server 2008 R2 and Windows 7.

Has anyone seen this? If so, what's the workaround (aside from just using the 64-bit installation)? On the other hand, if there really is no workaround and we have to use 64 bit, then that would give me a little more ammunition to bring this up again.

Slappy Slappy · Accepted Answer · 2013-06-27T09:06:38

Hmm there may be various reasons. Difficult to say without debugging. Few tips:

try running the installer in admin (elevated) / user (non-elevated) mode
try running the CMD (command line) in admin (elevated) / user (non-elevated) mode
try setting the working directory before calling nsExec::ExecToLog
try setting SetShellVarContext current|all
try changing SetRegView 32|64|lastused

My first bet is the cmd.exe (command line) launched from NSIS differs from cmd launched directly from Windows.

When you run cmd manually (succesfully) it is launched from which dir? And is it a 32bit process (can be seen in task manager - http://www.heckler.com.br/blog/2010/03/16/how-to-open-a-64bit-command-prompt-from-a-32bit-application/) or 64bit?

Using NSIS to run 32 bit java keytool on 64 bit server

1 Answers