MQRC Resource problem in WebSphere MQ

Question

This is in Cluster Environment. Queue Manager lost its identity in cluster and it is unable to connect to other servers. All channels to repository and others were retrying state.

CPU usage is optimal in this server. This is a UNIX box.

When I checked the logs below is it,

AMQ9532: Program cannot set queue attributes.

EXPLANATION: The attempt to set the attributes of queue 'SYSTEM.CLUSTER.TRANSMIT.QUEUE' on queue manager 'QMGR.SERVER6A' failed with reason code 2102.

ACTION: Ensure that the queue is available and retry the operation.

----- amqrmssa.c : 690 --------------------------------------------------------
AMQ9999: Channel program ended abnormally.

EXPLANATION: Channel program 'Channel.Coord00' ended abnormally.

ACTION: Look at previous error messages for channel program 'Channel.Coord00' in the error files to determine the cause of the failure.

----- amqrccca.c : 883 --------------------------------------------------------
03/06/11 08:24:26 AMQ9544: Messages not put to destination queue.

EXPLANATION: During the processing of channel 'Channel.Server6A' one or more messages could not be put to the destination queue and attempts were made to put them to a dead-letter queue. The location of the queue is 1, where 1 is the local dead-letter queue and 2 is the remote dead-letter queue.

ACTION: Examine the contents of the dead-letter queue. Each message is contained in a structure that describes why the message was put to the queue, and to where it was originally addressed. Also look at previous error messages to see if the attempt to put messages to a dead-letter queue failed. The program identifier (PID) of the processing program was '1372200'.

----- amqrmrca.c : 1318 -------------------------------------------------------

Then I did recycled the queue manager it is now fine?

My question here is how did the MQ resource problem occurr? CPU usage of this server is not more than 15%. Please advise.

T.Rob T.Rob · Accepted Answer · 2011-03-07T13:58:16

There are three different and unrelated problems shown in the log.

AMQ9532: Program cannot set queue attributes.

EXPLANATION: The attempt to set the attributes of queue 'SYSTEM.CLUSTER.TRANSMIT.QUEUE' on queue manager 'QMGR.SERVER6A' failed with reason code 2102.

The 2102 is MQRC_RESOURCE_PROBLEM and presumably the resource issue referred to in the posting. The 2102 can be any kind of scarce resource, including semaphores, user processes, queue handles, etc. Since the QMgr was attempting to set an attribute of the queue, it would have already had a thread instantiated but it would have required additional queue handles. When something like this occurs, use your admin tool (WMQ Explorer, mqmon or one of the many 3rd party tools) to look into the number of open queue handles, open channels, etc. Note that for a resource error, it will be necessary to maintain an open connection to the QMgr or else the tool will be unable to make a new connection when the resource shortage occurs.

AMQ9999: Channel program ended abnormally.

EXPLANATION: Channel program 'Channel.Coord00' ended abnormally.

ACTION: Look at previous error messages for channel program 'C00.US.MP00' in the error files to determine the cause of the failure.

This error appears to actually be two different errors since it references two different channels. One of these appears to be an outbound cluster channel and the other appears to be a point-to-point channel. Neither channel mentioned in this error are associated with the first and last error message.

03/06/11 08:24:26 AMQ9544: Messages not put to destination queue.

EXPLANATION: During the processing of channel 'Channel.Server6A' one or more messages could not be put to the destination queue and attempts were made to put them to a dead-letter queue. The location of the queue is 1, where 1 is the local dead-letter queue and 2 is the remote dead-letter queue.

ACTION: Examine the contents of the dead-letter queue. Each message is contained in a structure that describes why the message was put to the queue, and to where it was originally addressed. Also look at previous error messages to see if the attempt to put messages to a dead-letter queue failed. The program identifier (PID) of the processing program was '1372200'.

The last error appears to be an inbound cluster channel. Since the first error was trying to set attributes of the cluster transmit queue, it could only have been associated with an outbound channel. Therefore the first and last error messages are unrelated. This error message appears to show an inbound message that was destined for a queue and that queue was full, PUT-disabled, or otherwise unable to accept the message. The message was therefore routed to the dead letter queue.

For the resource error, I would suggest reviewing the performance report appropriate to your platform. Go to the SupportPacs page and look for those SupportPacs named MP* and then look for the one for your platform. The Performance Reports give you specific tuning advice.

You may also want to review the Problem Determination chapter in the System Administration manual for additional advice on how to identify resource issues.

The WebSphere MQ cluster design and operation article in the developerWorks Mission:Messaging series has specific advice about keeping clusters healthy.

Last but not least, the WebSphere MQ MustGather page has sections on troubleshooting for all major platforms and categorized by problem area.

MQRC Resource problem in WebSphere MQ

2 Answers