Reading from blocking queue with multiple threads

Question

I have a producer-consumer model using a blocking queue where 4 threads read files from a directory puts it to the blocking queue and 4 threads(consumer) reads from blocking queue.

My problem is every time only one consumer reads from the Blockingqueue and the other 3 consumer threads are not reading:

        final BlockingQueue<byte[]> queue = new LinkedBlockingQueue<>(QUEUE_SIZE);

            CompletableFuture<Void> completableFutureProducer = produceUrls(files, queue, checker);
//not providing code for produceData , it is working file with all 4 //threads writing to Blocking queue. Here is the consumer code.

    private CompletableFuture<Validator> consumeData(
            final Response checker,
            final CompletableFuture<Void> urls
    ) {
        return CompletableFuture.supplyAsync(checker, 4)
                .whenComplete((result, err) -> {
                    if (err != null) {
                        LOG.error("consuming url worker failed!", err);
                        urls.cancel(true);
                    }
    });


    }
  completableFutureProducer.join();
            completableFutureConsumer.join();

This is my code. Can someone tell me what I am doing wrong? Or help with correct code. Why is one consumer reading from the Blocking queue.

Adding code for Response class reading from Blocking queue :

    @Slf4j
    public final class Response implements Supplier<Check> {
        private final BlockingQueue<byte[]> data;
        private final AtomicBoolean producersComplete;
        private final Calendar calendar = Calendar.getInstance();
    
        public ResponseCode(
                final BlockingQueue<byte[]> data
        ) {
            this.data = data;
            producersDone = new AtomicBoolean();
    
        }
public void notifyProducersDone() {
    producersComplete.set(true);
}

        @Override
        public Check get() {
            try {
                Check check = null;
                try {
                    while (!data.isEmpty() || !producersDone.get()) {
                        final byte[] item = data.poll(1, TimeUnit.SECONDS);
                        if (item != null) {
                           LOG.info("{}",new String(item));
// I see only one thread printing result here .
                            validator = validateData(item);
                        }
                    }
        
                } catch (InterruptedException | IOException e) {
                    Thread.currentThread().interrupt();
                    throw new WriteException("Exception occurred while data validation", e);
        
                } 
                return check;
            } finally {
                LOG.info("Done reading data from BlockingQueue");
            }
        }
    }

I dont see any code that would try to take from the queue nor put something into it. — Antoniossss

Enno Shioji Enno Shioji · Accepted Answer · 2020-08-09T16:53:17

It's hard to diagnose from this alone, but it's probably not correct to check for data.isEmpty() because the queue may happen to be temporarily empty (but later get items). So your threads might exit as soon as they encounter a temporarily empty queue.

Instead, you can exit if producers were done AND you got an empty result from the poll. That way the threads only exit when there are truly no more items to process.

It's a bit odd though that you are returning the result of the last item (alone). Are you sure this is what you want?

EDIT: I've done something very similar recently. Here is a class that reads from a file, transforms the lines in a multi-threaded way, then writes to a different file (the order of lines are preserved).
It also uses a BlockingQueue. It's very similar to your code, but it doesn't check for quue.isEmpty() for the aforementioned reason. It works fine for me.

Reading from blocking queue with multiple threads

3 Answers