Losing messages when using RabbitMQ with Logstash

Question

I have just installed RabbitMQ, I'm using it with logstash. I have been sending messages but I lose a lot of data. I have no idea how could I solve it. I'm using the default configuration in rabbitMQ.

These are the output in logstash and the input in logstash:

output {
 rabbitmq {
     host => "IPHOST"
     exchange => "logstash-rabbitmq"
     exchange_type => "direct"
     key => "logstash-key"
     workers =>4
     durable => true
     persistent => true
}
}




input
 {

rabbitmq {
    host => "IPHOST"
    queue => "logstash-queue"
    durable => true
    key => "logstash-key"
    threads => 8
    prefetch_count => 100
    port => 5672
    user => "test"
    password => "test"
     ack => false
     exchange => "logstash-rabbitmq"
}

}

I'm using RabbitMQ Management in order to see the evolution of my queue. When I send 10,000 messages I only receive less than a half. Is there any parameter I should change to improve the behaviour of rabbitMQ? I was going to use it in order so that I don't lose messages, but I'm losing even more than when I didn't have it.

I can't see any message in the queue

Change the ack parameter to true. This means that the consumer has to send back an acknowledgement of receiving the message before moving on to another message from the queue. — idbehold
I have tried it but I have the same problem. I receive even more messages without using rabbitMQ. It is a nonsense — billy6
Add the feature to re-push the messages to the queue if not delivered for the first time due to transaction failure or anything, like a dead letter exchange. Make sure the messages are persistent. Acknowledge is true, queue is durable. If the server goes down there should be no data loss. — underdog

underdog underdog · Accepted Answer · 2015-12-11T05:58:26

Possible steps to fix this

Add the feature to re-push the messages to the queue if not delivered for the first time due to transaction failure or anything, like a dead letter exchange.

Make sure the messages are persistent. Acknowledge is true, queue is durable. If the server goes down there should be no data loss.

Make sure queue is created before publishing.

You should use MessageProperties for making your messages persistence.

channel.basicPublish("", "task_queue", 
        MessageProperties.PERSISTENT_TEXT_PLAIN,
        message.getBytes());

Enable the firehose tracer to check the message in the queue.

Checkout the rabbitmq reliability guide.

Losing messages when using RabbitMQ with Logstash

1 Answers