Clickonce program will not start when launched from shell_execute

Question

I have a very old program that I have no control over. It launches a filetype with its default application like this(I cannot modify this code):

LET Err (SHELL_EXECUTE 'open' (FIX_MESG '"{1}"' File_name) '' '')

^^The above code works, so long as that filetype isn't associated with a ClickOnce program.

The old program is 32 bit, the OS is Windows 7 64 bit. I can compile my clickonce program as anything, but none seem to work. (I've tried x86, x64 and anyCPU)

How can I make a 32 bit program use shell execute to launch a ClickOnce program on a 64bit OS?

More details: This is a reproducible error. Build 2 programs. Program 1 is a Clickonce program. Associate it with any filetype. Program 2 will do a shell execute command to open whatever filetype the clickonce program is associated with. If you compile program 2 as x86, it will give you a success response, but do nothing.

Test code for console shell execute program:

private static void Main()
{
    int value = ShellExecuteA(IntPtr.Zero, "open", @"C:\Users\bsee\Desktop\testfile.ecn2", "", "", 0);
        if (value > 32)
            MessageBox.Show("Clickonce reports success. But did it actually start?");
}

[DllImport("Shell32.dll")]
public static extern int ShellExecuteA(IntPtr hwnd, string lpOperation, string lpFile, string lpParameters, string lpDirecotry, int nShowCmd);

'none seem to work' - is there an error message? can you see anything starting in Task mgr? — Rikalous
@Rikalous There is no error message. In fact shell_execute when tested returns a value over 32 which technically means successful. — Brandon
@OndrejJanacek It probably isn't. The clickonce application is C#. If there was any chance it was a language specific bug I thought I'd explore the possibility. — Brandon
the accepted answer in @Rikalous's thread suggested a batch file that calls the actual file. This doesn't modify the existing program. Was there something that made this solution not successful for you? — Claies

Hans Passant Hans Passant · Accepted Answer · 2013-11-11T16:09:42

This is a pretty common problem, lots of google hits but few solutions. Your code snippet is flawed and cannot diagnose the issue, surely also the problem in that app you have to interop with. It will start the file association just fine, the issue is that it then fails to do the job. Your code snippet cannot diagnose this, you must use the Process class instead and obtain the ExitCode when the process completes. It will not be 0.

Let's talk about what goes wrong. ClickOnce creates the file association for you and will write the "open" verb for the file extension like this:

 rundll32.exe dfshim.dll, ShOpenVerbExtension {guid} %1

And also writes another registry key for the {guid}, written to HKCU\Software\Classes\CLSID\{guid}. Where dfshim.dll's ShOpenVerbExtension() function will look for the {guid}. This key is special on a 64-bit operating system, there are two versions of it. The registry redirector will remap the request to open the key for a 32-bit app, it goes to Software\Wow6432Node instead. This is a basic mechanism in the 64-bit version of Windows that prevents trouble when 32-bit apps accidentally load 64-bit components. Particularly important for COM.

There are also two versions of rundll32.exe. One in c:\windows\system32, the 64-bit version and another in c:\windows\syswow64, the 32-bit version. That 32-bit version will see the redirected view of the registry keys. Which one is used is determined by another redirection, implemented by the File Redirector. Which automatically remaps requests that 32-bit apps make to files in c:\windows\system32 to c:\windows\syswow64. Yet another important counter-measure to stop 32-bit apps from failing when they accidentally load a 64-bit Windows DLL.

Perhaps you see the trouble here, at issue is that this app you need to interop with is a 32-bit app. It will therefore always end up running the 32-bit version of rundll32.exe from c:\windows\syswow64. dfshim.dll now is going to end up looking in the System\Wow6432Node part of the registry and will not find the {guid} there. Major fail whale, it doesn't display any kind of error message and if the app doesn't check the ExitCode either then there's no diagnostic at all.

Notable is that you can easily make your test program succeed. Project + Properties, Build tab, change the Platform Target setting to AnyCPU. Turn off the "Prefer 32-bit" option off on VS2012 and up. Your ShellExecute() call will now use c:\windows\system32\rundll32.exe program and dfshim has no trouble retrieving the {guid}.

You can also easily fix the 32-bit version of it. Just use Regedit.exe to edit the open verb of the file association and change it to %windir%\sysnative\rundll32.exe. The "sysnative" part of the name is a special name that the file redirector understands, mapping the request to system32 so you'll always get the 64-bit version of program.

The ClickOnce team can fix it too, and has done so according to the connect article, they probably ended up writing both CLSID\{guid} keys. That fix is not included with .NET 4.0, not sure if it made it into 4.5. Edit: it did.

Plenty of possible fixes, the core problem is that you just don't have enough control over it. You of course can't get that app to run in 64-bit mode, you cannot typically make sure that the target machine has the updated version of .NET, you cannot change what the ClickOnce installer writes to the registry keys.

Just two alternatives left: don't use ClickOnce or ask the customer to edit the registry key himself. You can craft a .reg file to minimize the odds for mistakes.

Clickonce program will not start when launched from shell_execute

1 Answers