stopping consumer when producer can't provide anymore

Question

Here is my consumer code, i removed the computation part since it's (i think) irrelevant :

void *compute()
{
    struct document** document;
    document = (struct document**)malloc( sizeof(struct document*) );
    while( ( hasNextData || counter > 0 ) ) {
        sem_wait( &full );
        pthread_mutex_lock( &buffer_access) ;
        if( remove_document( document ) ) {
            fprintf( stderr, "Consumer report error condition\n");
            pthread_mutex_unlock(&buffer_access);
        }
        else {
//          pthread_mutex_unlock( &buffer_access );
//          Compute the document
        }
        sem_post( &empty );
    }
    free( document );
    pthread_exit( 0 );
}

I also have a producer which will produce documents and put them in a buffer. If the buffer is full, it waits for the documents to be processed (they are removed by remove_document). When the input file is entirely read, it stops and the thread exits.

This code works well with one consumer thread or two, but when there are more, it deadlocks. I think the problem is with the waiting semaphore.

Let's pretend we have 6 input lines (they will create documents in the buffer) and 10 consumer threads. Ten threads will be created to process only 6 inputs. So at best there will be 4 locking threads if each thread processes one data ? And at worst 9 locking threads ?

That's my understanding of the problem. But I don't know how to prevent that since the tutorials I read were using while(1) as a condition in both the producers and the consumers.

Is there a standard/common way to solve the consumer-producer problem when they don't produce/consume while(true) ? Or should i tweak with counters ? But i don't see where to put the condition. If i put it outside the wile, i'll just kill threads before they work on something, if i put it inside they will be locked by the sem_wait()

One way: Send a signal to each thread. If a thread is blocked on sem_wait it will unblock and returnEINTR which the code should check for. If it is not at sem_wait it will just continue executing. The signal handler should set a flag to say that the thread should stop processing and the while loop should be checking this flag. — kaylum
The standard way to stop a consumer is to send it a sentinel value which is not a job to process but rather a signal that it should shut down. Still, your description is lacking, because you claim "it" works for two but not more threads, which simply indicates that something is rotten in your code. Clean it up and extract a minimal example for posting here if you then still need help fixing it. — Ulrich Eckhardt

Pemdas Pemdas · Accepted Answer · 2016-04-03T12:52:31

Your problem is not really related to the classic consumer/producer problem itself. Your problem is that you need a way execute a controlled shutdown of the "worker/consumer" threads.

In this case, I might accomplish this by sending a "empty/null" document to the consumers. You can define empty however you want.

The thread managing the consumers needs to send this signal once for ever consumer thread that was created.

The procedure might look something like this:

    while(1)
    {
        sem_wait( &full );
        pthread_mutex_lock( &buffer_access) ;
        if( remove_document( document ) ) {
            fprintf( stderr, "Consumer report error condition\n");
            pthread_mutex_unlock(&buffer_access);
            // I would probably exit here too, but thats up to you
        }
        else if (isEmptyDoc(document)
        {
          // Clean up and exit thread.
        }
        else
        {
          // Do stuff. 
        }
        sem_post( &empty );
    }

stopping consumer when producer can't provide anymore

1 Answers