How to unload C++ Shell Extension DLL properly

7
votes

I have shell extension dll written in C++ and COM The dll is registered and loaded into memory. My upgrade setup program will do the following:

  • Unregister the shell extension dll, kill the explorer.exe
  • Copy the later version of the shell extension dll (Step 2)
  • Start t he explorer.exe

It works fine. But the issue is:

If the user opened any other applications ( Internet explorer, sometimes windows task manager, notepad etc), the step Step 2 is failing.

Is there any way to close all shell extension dll hooks while upgrading the dll.

In the dll I am using GetOverlayInfo, context menu, database connection etc

2
visual-studio-2010winapivisual-c++shell-extensions
It's a fallacy thinking that killing explorer.exe will help anything. First of all it is a very invasive thing to do (Windows Installer will instead move the file in use into another place on the same volume to install the new file in the original location) and secondly it may not work at all because the user could be using a completely different file manager that makes use of shell extensions. I do that myself and installers making those assumptions are just annoying. You could close the references remotely, causing all kinds of havoc. Don't do it! – 0xC0000022L
I'm with @0xC0000022L. I hate restarts just like everyone else, but I think this calls for one. You can complete your installation when the system is started again, before explorer is launched. – eran
@eran: just requires a restart of the affected applications (or a logoff), actually. – 0xC0000022L
Your shell extension DLL gets loaded into other processes through the shell dialogs. Like File + Open in Notepad. You can rename the original DLL and put your new one into place but a logout is required to make it effective. – Hans Passant
@HansPassant: just wondering, do you know the definitive answer to this one: will the new shell extension be used by new application instances right away or not? I'm not 100% sure. – 0xC0000022L

2 Answers

8
votes

Simply put: don't do it (i.e. do not unload it forcibly). You would have to enumerate all processes that have your shell extension loaded and then "restart" those. This is very invasive and quite frankly: bad behavior (for an installer). This also requires particular privileges which your installer may not otherwise need.

What most people still don't seem to be aware of is that MoveFile (and MoveFileEx) can be used to move DLLs or EXE files that are currently in use by a running application.

This is the approach Windows Installer takes. Have you ever noticed the folder \Config.msi in the root of a given drive after installing some .msi? This folder actually contains (moved and usually renamed to some unique "temporary" name) the original files that have been moved out but were still in use at the time. They are then usually scheduled for deletion upon boot (MoveFileEx with MOVEFILE_DELAY_UNTIL_REBOOT).

You can use the same mechanism in your homebrew installer and move the old file that is in use out of its original location and move the other one in right away. Any new application instance will then make use of the new shell extension (*), while the old one will continue to be used by any of the running applications that loaded it at one point or another.

(*) I'm not 100% sure about this because of the rules that apply to DLL loading and prevent modules with the same name to be loaded again under some circumstances.

1
votes

Depending on your installer, you may be able to make use of the Restart Manager support in Windows Vista+. This will allow your setup to query for every application that is using your DLL, and try and gracefully shut them down. If they can't be, then you will need to register it for replacement on a restart. Once the setup has finished, the Restart Manager will try and restart any programs it shut down (that support being restarted).