Websphere MQ intergration with flume

Question

I am trying to integrate Websphere (IBM) MQ with flume. I have few xml files which will be coming from MQ & I want get those messages in flume & do further processing in Spark.

I am doing this integration on AWS EC2 Instance where my Hadoop is also installed. Below are the steps which I followed for integration.

Create Queue Manager: https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.con.doc/q015210_.htm

./crtmqm -q -d MY.DEFAULT.BTS.QUEUE -u SYSTEM.DEAD.LETTER.QUEUE CSBTS.QUEUE.MANAGER

Start Queue Manager:

./strmqm CSBTS.QUEUE.MANAGER
./runmqsc CSBTS.QUEUE.MANAGER
Define Server Channel

DEFINE CHANNEL(CHAN2) CHLTYPE(SVRCONN) TRPTYPE(TCP) DESCR('Server-connection to Client_2')

Define Client Channel

DEFINE CHANNEL(CHAN2) CHLTYPE(CLNTCONN) TRPTYPE(TCP) CONNAME(10.0.0.238) QMNAME(CSBTS.QUEUE.MANAGER) DESCR('Client-connection to Server_2')

ALTER QMGR CHLAUTH(DISABLED)

Define local queue

DEFINE QLOCAL (CSBTS.DEAL)
Add messages

./amqsput CSBTS.DEAL CSBTS.QUEUE.MANAGER
Create connection factory

DEF CF(CF4) QMGR(CSBTS.QUEUE.MANAGER) TRANSPORT(CLIENT) HOSTNAME(10.0.0.238) CHANNEL(CHAN2)
Start listener

DEFINE LISTENER (CSBTS_LIST) TRPTYPE (TCP) PORT (1414) CONTROL(QMGR)

DISPLAY LISTENER (CSBTS_LIST)

DISPLAY LSSTATUS (CSBTS_LIST)

(RESTART qm)
Run flume

/usr/bin/flume-ng agent -c /etc/flume-ng/conf -f /etc/flume-ng/conf/flume.conf -n tier1

Flume is executed through linux user “hduser”.

And below is the authentication issue which we are getting.

20 Jul 2016 09:17:33,236 ERROR [PollableSourceRunner-JMSSource-source1] (org.apache.flume.source.PollableSourceRunner$PollingRunner.run:156)  - Unhandled exception, logging and sleeping for 5000ms
org.apache.flume.FlumeException: Source had error configuring or starting
        at org.apache.flume.source.AbstractPollableSource.process(AbstractPollableSource.java:48)
        at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.FlumeException: Could not create connection to broker
        at org.apache.flume.source.jms.JMSMessageConsumer.<init>(JMSMessageConsumer.java:76)
        at org.apache.flume.source.jms.JMSMessageConsumerFactory.create(JMSMessageConsumerFactory.java:32)
        at org.apache.flume.source.jms.JMSSource.createConsumer(JMSSource.java:317)
        at org.apache.flume.source.jms.JMSSource.doStart(JMSSource.java:299)
        at org.apache.flume.source.BasicSourceSemantics.start(BasicSourceSemantics.java:83)
        at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:74)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'CSBTS.QUEUE.MANAGER' with connection mode 'Client' and host name '10.0.0.238(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.
        at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:531)
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
        at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:413)
        at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8475)
        at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7814)
        at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:299)
        at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)
        at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6024)
        at com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6073)
        at org.apache.flume.source.jms.JMSMessageConsumer.<init>(JMSMessageConsumer.java:69)
        ... 13 more
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
        ... 21 more

I am stuck with this from past 3 weeks, if anyone has faced this befoew then please help us out.

Error log from AMQERR01.LOG

----- cmqxrsrv.c : 2321 -------------------------------------------------------
07/21/2016 10:08:17 AM - Process(16169.50) User(mqm) Program(amqzlaa0)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ5540: Application 'flume.node.Application' did not supply a user ID and
password

EXPLANATION:
The queue manager is configured to require a user ID and password, but none was
supplied.
ACTION:
Ensure that the application provides a valid user ID and password, or change
the queue manager configuration to OPTIONAL to allow applications to connect
which have not supplied a user ID and password.
----- amqzfuca.c : 4287 -------------------------------------------------------
07/21/2016 10:08:17 AM - Process(16169.50) User(mqm) Program(amqzlaa0)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ5541: The failed authentication check was caused by the queue manager
CONNAUTH CHCKCLNT(REQDADM) configuration.

EXPLANATION:
The user ID 'mqm' and its password were checked because the user ID is
privileged and the queue manager connection authority (CONNAUTH) configuration
refers to an authentication information (AUTHINFO) object named
'SYSTEM.DEFAULT.AUTHINFO.IDPWOS' with CHCKCLNT(REQDADM). 

This message accompanies a previous error to clarify the reason for the user ID
and password check.
ACTION:
Refer to the previous error for more information. 

Ensure that a password is specified by the client application and that the
password is correct for the user ID. The authentication configuration of the
queue manager connection determines the user ID repository. For example, the
local operating system user database or an LDAP server. 

To avoid the authentication check, you can either use an unprivileged user ID
or amend the authentication configuration of the queue manager. You can amend
the CHCKCLNT attribute in the CHLAUTH record, but you should generally not
allow unauthenticated remote access.
-------------------------------------------------------------------------------
07/21/2016 10:08:18 AM - Process(18280.13) User(mqm) Program(amqrmppa)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ9557: Queue Manager User ID initialization failed for 'mqm'.

EXPLANATION:
The call to initialize the User ID 'mqm' failed with CompCode 2 and Reason
2035.
ACTION:
Correct the error and try again.
----- cmqxrsrv.c : 2321 -------------------------------------------------------
07/21/2016 10:10:58 AM - Process(16169.51) User(mqm) Program(amqzlaa0)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ8077: Entity 'hduser' has insufficient authority to access object
'CSBTS.QUEUE.MANAGER'.

EXPLANATION:
The specified entity is not authorized to access the required object. The
following requested permissions are unauthorized: connect
ACTION:
Ensure that the correct level of authority has been set for this entity against
the required object, or ensure that the entity is a member of a privileged
group.
----- amqzfubx.c : 687 --------------------------------------------------------
07/21/2016 10:10:58 AM - Process(18280.14) User(mqm) Program(amqrmppa)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ9557: Queue Manager User ID initialization failed for 'hduser'.

EXPLANATION:
The call to initialize the User ID 'hduser' failed with CompCode 2 and Reason
2035.
ACTION:
Correct the error and try again.
----- cmqxrsrv.c : 2321 -------------------------------------------------------

Please update your question with the error message seen in the QMgr AMQERR01.LOG file from the same time as the 2035 error is seen. — Morag Hughson
I dont have to do that, I am already using Websphere 8.0.0 version. MQS_REPORT_NOAUTH is for versions below 7 — Harsha
Hi @MoragHughson I have updated my question with the AMQERR01.LOG error which i am getting for 2035 error. I am not passing any password from my flume.conf file. If you want to look into my flume.conf file, i will post it in my question. — Harsha

Harsha Harsha · Accepted Answer · 2016-07-21T15:42:26

Thanks @MoragHughson, it was based on your advice to look for the AMQERR01.LOG to find out more ablout the issue. It helped to overcome my issue of Interating MQ with Flume.

What i did was i changed the queue manager configuration to OPTIONAL to allow applications to connect which have not supplied a user ID and password. In my case I was not passing the password in my flume.conf file.

I followed this link - http://www-01.ibm.com/support/docview.wss?uid=swg21680930 and used the option B.

Below is the command which i used to make Authentication OPTIONAL -

ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) CHCKCLNT(OPTIONAL)

REFRESH SECURITY TYPE(CONNAUTH)

I hope this helps to anyone who is trying to integrate MQ with FLume/Kakfa.

Websphere MQ intergration with flume

1 Answers