This is a pretty common problem, lots of google hits but few solutions. Your code snippet is flawed and cannot diagnose the issue, surely also the problem in that app you have to interop with. It will start the file association just fine, the issue is that it then fails to do the job. Your code snippet cannot diagnose this, you must use the Process class instead and obtain the ExitCode when the process completes. It will not be 0.
Let's talk about what goes wrong. ClickOnce creates the file association for you and will write the "open" verb for the file extension like this:
rundll32.exe dfshim.dll, ShOpenVerbExtension {guid} %1
And also writes another registry key for the {guid}, written to HKCU\Software\Classes\CLSID\{guid}
. Where dfshim.dll's ShOpenVerbExtension() function will look for the {guid}. This key is special on a 64-bit operating system, there are two versions of it. The registry redirector will remap the request to open the key for a 32-bit app, it goes to Software\Wow6432Node instead. This is a basic mechanism in the 64-bit version of Windows that prevents trouble when 32-bit apps accidentally load 64-bit components. Particularly important for COM.
There are also two versions of rundll32.exe. One in c:\windows\system32, the 64-bit version and another in c:\windows\syswow64, the 32-bit version. That 32-bit version will see the redirected view of the registry keys. Which one is used is determined by another redirection, implemented by the File Redirector. Which automatically remaps requests that 32-bit apps make to files in c:\windows\system32 to c:\windows\syswow64. Yet another important counter-measure to stop 32-bit apps from failing when they accidentally load a 64-bit Windows DLL.
Perhaps you see the trouble here, at issue is that this app you need to interop with is a 32-bit app. It will therefore always end up running the 32-bit version of rundll32.exe from c:\windows\syswow64. dfshim.dll now is going to end up looking in the System\Wow6432Node part of the registry and will not find the {guid} there. Major fail whale, it doesn't display any kind of error message and if the app doesn't check the ExitCode either then there's no diagnostic at all.
Notable is that you can easily make your test program succeed. Project + Properties, Build tab, change the Platform Target setting to AnyCPU. Turn off the "Prefer 32-bit" option off on VS2012 and up. Your ShellExecute() call will now use c:\windows\system32\rundll32.exe program and dfshim has no trouble retrieving the {guid}.
You can also easily fix the 32-bit version of it. Just use Regedit.exe to edit the open verb of the file association and change it to %windir%\sysnative\rundll32.exe
. The "sysnative" part of the name is a special name that the file redirector understands, mapping the request to system32 so you'll always get the 64-bit version of program.
The ClickOnce team can fix it too, and has done so according to the connect article, they probably ended up writing both CLSID\{guid}
keys. That fix is not included with .NET 4.0, not sure if it made it into 4.5. Edit: it did.
Plenty of possible fixes, the core problem is that you just don't have enough control over it. You of course can't get that app to run in 64-bit mode, you cannot typically make sure that the target machine has the updated version of .NET, you cannot change what the ClickOnce installer writes to the registry keys.
Just two alternatives left: don't use ClickOnce or ask the customer to edit the registry key himself. You can craft a .reg file to minimize the odds for mistakes.