ShellExecute returns success, but does not execute file

Question

I have a weird problem in one of my projects lately regarding the Win32 API ShellExecute() function.

The program is compiled using Visual C++ 2015, in ANSI mode.

int ret = (int)ShellExecuteA(0, "open", "C:\\Users\\Maverick\\stratum.jpg", NULL, NULL, SW_SHOWNORMAL);
printf("ShellExecute return value: %i\n", ret);

In the above code, ShellExecute() returns 42, so it should be successful. However, it doesn't actually open the file.

I don't have privilege problems, and it fails with the same problem even when running the program as an administrator.

In fact, I can successfully run the file this way:

system("C:\\Users\\Maverick\\stratum.jpg");

I don't want to be forced to use system(), though.

Also, before I migrated the project to a newer Visual Studio, I was using Visual C++ 6.0, and the code worked fine.

Any clues what may be the problem?

EDIT: ShellExecuteEx() also returns successful (1), but does not open the file.

SHELLEXECUTEINFO ShExecInfo;
ShExecInfo.cbSize = sizeof(SHELLEXECUTEINFO);
ShExecInfo.fMask = NULL;
ShExecInfo.hwnd = NULL;
ShExecInfo.lpVerb = NULL;
ShExecInfo.lpFile = "C:\\Users\\Maverick\\stratum.jpg";
ShExecInfo.lpParameters = NULL;
ShExecInfo.lpDirectory = NULL;
ShExecInfo.nShow = SW_SHOWNORMAL;
ShExecInfo.hInstApp = NULL;

int ret = (int)ShellExecuteExA(&ShExecInfo);

The IDE/compiler version has no effect on how ShellExecute() behaves, since it is a Win32 API function implemented at the OS layer. The same function gets called in both versions. In any case, try using ShellExecuteEx() instead, it provides more accurate error reporting than ShellExecute() does. Also consider replacing "open" with NULL to invoke the default action, which might not be "open" to begin with, depending on the imaging software you have installed. — Remy Lebeau
Thank you for your comment. Tried to replace "open" with NULL but result is the same. Will try with ShellExecuteEx — Flavio
tried with ShellExecuteEx, same problem, returns successful but it does not execute file :( — Flavio
@Flavio: it is quite possible that ShellExecute/Ex() is successfully launching a new process, but that process is then failing to load the file. From ShellExecute/Ex's perspective, its portion of work was successful. Unfortunately, there is no way to detect this kind of failure when using ShellExecute/Ex(). You would have to invoke the target app manually, such as by using FindExecutable() and CreateProcess() directly (which does not account for all the possible ways a file can be opened by the Shell, but should suffice for your particular example). — Remy Lebeau
@Flavio: also note that ShellExecuteEx() returns a BOOL, not an int. If it returns FALSE, use GetLastError() to get the actual error code. — Remy Lebeau

Flavio Flavio · Accepted Answer · 2017-04-05T17:50:35

Well, looks like I found the problem and was weird indeed. Maybe a memory leak. Calling the function CreatePopupMenu() many times in a loop (like, tens of thousands) before a ShellExecute/ShellExecuteEx call, will make this problem happen. Unless we free the HMENUs using DestroyMenu BEFORE ShellExecute call. Looks like was a memory leak of some sort maybe, well thank you anyways for your answers.

ShellExecute returns success, but does not execute file

1 Answers