Is there any way to synchronize processes taking crashes into account?

Question

I'm working on a project which involves two processes, let them be A and B. A has got two threads: T1 and T2.

A and B are run separately.

At a certain point T1 sends a message (it's a COM call, a search request) to B. When B has processed some data it returns by sending the result (again, it's a COM call, the search results) directly to T2. I need T1 to block until T2 has processed B's data. So far I used the following solution:

Initialization: searchSyncSempahore = ::CreateSemaphore(NULL, 0, 1, NULL);

T1:

B_handle->search(searchString);
::WaitForSingleObject(searchSyncSempahore, INFINITE);
// Use searchResult variable

T2:

searchResult = _some_data_from_B;
::ReleaseSemaphore(searchSyncSempahore, 1, NULL);

This works so far. It works even if B is so fast that T2 calls ReleaseSemaphore before T1 reaches the WaitForSingleObject line.

My problem is: what happens if B crashes? T1 will just wait for ever. I have to specify the timeout but if it's too low when T2 receives results it will just screw up the semaphore, messing up the following search.

So how can I correctly synchronize these two threads taking into account that T2 could never be called (i.e. How can I implement "abort search")?

This is really complicated. Since you're waiting on the result anyway, why not use a synchronous call in T1 and pass the result to T2 when the call completes? Let COM take care of the housekeeping. — arx
I don't have access to B code, so I can't modify it. B has been designed in this way: it exposes an asynchronous interfaces and uses a dispinterface for callbacks. T2 is actually the ATL Sink thread. — Emiliano

gahcep gahcep · Accepted Answer · 2012-06-14T10:14:35

The possible solution is to create a watchdog thread which will handling such situation using Manual-Reset events. So the logic will change a small: instead of waiting the mutex, you can wait for event. The watchdog thread than could check if the thread stuck for a long time, just set event by itself and could set an additional event saying that critical situation is appeared. I mean this third thread can act as a watchdog timer.

The other variant is to use a Condition Variables in that third thread. So, you can simply kick one of the condition variables and it will wake up the thread waiting for it. I must admit, I'm not very familiar with this approach.

UPDATE: Consider using a SignalObjectAndWait function. The function signals one object and waits on another object as a single operation. Also, you can set a wait timeout, and the function returns if the interval elapses, even if the object's state is nonsignaled.

Another benefit of using thread watchdog, that you can do additional stuff in it: logging to file or even in MS EventLog or can provide logic to change an application behaviour. One more example is that a watchdog thread can implement 'thread redundancy' capability.

Is there any way to synchronize processes taking crashes into account?

2 Answers