Unhooking API Hook using APIHijack

Question

I've built an application that hooks into the Win32 TextOut function via APIHiJack. When applications are started, the DLL is injected as expected and my new TextOut function is called successfully.

Currently, there are two issues I'm struggling with regarding remove the hook and would like some guidance.

1) If some applications are closed, they do not send a FreeLibrary(?) call to the DLL to unhook and cleanup. Is this normal and if so, how is this usually dealt with?

2) More importantly, if my application crashes for any reason, the applications still have the hook and crash as expected as my new TextOut function no longer exists.

I have tried to enumerate all processes and find which ones contain the DLL (pretty much the reverse process of the hooking technique) but it never seems to find as many injection points as when hooking and therefore the hooked application(s) crash.

Many thanks in advance.

Answer 1

Applications have two major ways to terminate:

• ExitProcess(): When they use ExitProcess() then FreeLibrary() is called (and THREAD_DETACH in DLLMAIN).

• TerminateProcess(): But when the application uses TerminateProcess() it terminates immediately without any cleanups.

My Solution ist to hook TerminateProcess() an make the cleanups I need - specially flushing the buffers of my log files and closing them.

but: A termination with TerminateProcess() my indicate a serious error and the application may be instable.

Answer 2

1) No, but you will get a THREAD_DETACH in the DllMain.

2) If your hook code verifies if your server is running and if not it should do the default behavior of the hooked function. If your handler crashes it will crash the host application. If your server application crashes you can avoid host application crashing verifying if the server is running. You can verify it using events or with the Pid.