rabbitmq client hangs while trying to declare queue

Question

I tried searching for solution of my problem but could not find it stack overflow.

Issue When a user tries to declare a queue or exchange, in a corner case where RabbitMQ server is having some issue, the client keeps waiting without any timeout which causes the thread calling the rabbitmq to always remain in waiting state (wait which never ends).

Below is stacktrace

java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:50)
    - locked <0x00000007bb0464c8> (a com.rabbitmq.utility.BlockingValueOrException)
    at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:89)
    - locked <0x00000007bb0464c8> (a com.rabbitmq.utility.BlockingValueOrException)
    at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
    at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
    at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
    at 
(AMQChannel.java:118)
    at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:833)
    at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:61)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:917)
    - locked <0x00000007bb555300> (a java.lang.Object)
    at com.sun.proxy.$Proxy293.queueDeclare(Unknown Source)
    at org.springframework.amqp.rabbit.core.RabbitAdmin.declareQueues(RabbitAdmin.java:575)
    at org.springframework.amqp.rabbit.core.RabbitAdmin.access$200(RabbitAdmin.java:66)
    at org.springframework.amqp.rabbit.core.RabbitAdmin$12.doInRabbit(RabbitAdmin.java:504)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1456)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1412)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1388)
    at org.springframework.amqp.rabbit.core.RabbitAdmin.initialize(RabbitAdmin.java:500)
    at org.springframework.amqp.rabbit.core.RabbitAdmin$11.onCreate(RabbitAdmin.java:419)
    at org.springframework.amqp.rabbit.connection.CompositeConnectionListener.onCreate(CompositeConnectionListener.java:33)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:553)
    - locked <0x00000007bb057828> (a java.lang.Object)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1412)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1388)
    at org.springframework.amqp.rabbit.core.RabbitAdmin.declareQueue(RabbitAdmin.java:207)

Any help will be highly appreciated. Declaration of queues is currently in my postconstruct of beans calling our component handling messaging, thus not letting any new bean create.

UPDATE The issue came again on our prod server. When trying to connect via amqp-client-3.4.2 directly it seems working. But from spring-rabbit-1.6.7.RELEASE, spring-amqp-1.6.7.RELEASE it is not working.

Via amqp-client-3.4.2

        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("<<HOST NAME>>");
        factory.setUsername("<<USERNAME>>");
        factory.setPassword("<<PASSWORD>>");
        factory.setVirtualHost("<<VIRTUAL HOST>>");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();
        channel.queueDeclare(QUEUE_NAME, true, false, false, null);

Code flow with rabbit-amqp client

Spring way which is not working

        CachingConnectionFactory factory = new CachingConnectionFactory();
        factory.setHost("<<HOST NAME>>");
        factory.setUsername("<<USERNAME>>");
        factory.setPassword("<<PASSWORD>>");
        factory.setVirtualHost("<<VIRTUAL HOST>>");

        RabbitAdmin admin = new RabbitAdmin(factory);
        Queue queue = new Queue(QUEUE_NAME);
        admin.declareQueue(queue);

Code flow with spring amqp

This issue occurs rarely and we are still trying to figure out the reason behind this behavior. We tried setting connection timeout but did not worked in our test program.

On debugging it further it looks like an exception is not letting notification sent back to our code. For client not found kind of issues, we are getting exception properly.

We are using RabbitMQ 3.6.10 and Erlang 19.3.4 on CentOS Linux 7 (Core)

Gary Russell Gary Russell · Accepted Answer · 2018-01-16T14:11:43

Declaration of queues is currently in my postconstruct of beans

I can't speak to the hang but you should NEVER interact with the broker from post construct, afterPropertiesSet() etc. It is too early in the application context lifecycle.

There are several work arounds - implement SmartLifecycle; return true from isAutoStartup() and put the bean in an early phase (see Phased). start() will be called after the application context is fully created.

However, it's generally better to just define the queues, bindings etc as beans and let the framework take care of doing all the declarations for you.

rabbitmq client hangs while trying to declare queue

1 Answers