Integrating Apache Hadoop and Flume with IBM MQ

Over the past 2 years of working with Apache Hadoop a few things have come up folks wanting to use Apache Kafka which definitely has it’s place in the Hadoop Big Data and Next Generation of Technology spheres. But there is also the need to integrate with what a lot of folks would consider Legacy Messaging (IBM MQ) with Apache Hadoop and Apache Flume. I personally don’t feel IBM MQ is a legacy technology it actually performs well and integrates with almost every platform out there. So this brings me to what I decided was a good idea after working with IBM MQ and WebSphere Application Server for a very long time 15 years.  I saw the need to integrate Apache Hadoop like technologies and IBM MQ specifically in regards to IBM MQ and Local Queue Managers that can utilize MQ Bindings to connect which will provide high performance/low latency messaging with Legacy Technologies and Hadoop.  Here is documentation from IBM stating MQ Bindings performs better than Clients.

The project is hosted at github. What this code enables is the ability to define a source or sink using IBM MQ Local QMGR and Queues as an endpoint.  Btw I provide no guarantees and no liability you are welcome to ask for bug fixes via github issues.

Most likely you will have a license to utilize IBM MQ from IBM or you can download the MQ Advanced Developer version which is free for developers.

After installing IBM MQ you must configure your local QMGR to allow the “flume” group or user-id you are using to run the local flume process on the same server as your QMGR.

How Configure IBM MQ to work with Flume in Bindings Mode

su – mqm
# Source Local QMGR Environment Variables
. /opt/ibm/mqm/bin/setmqenv
# Create a Local QMGR if it doesn’t already exist
crtmqm GSSMQP1
# Start the Local QMGR if it’s not already started
strmqm GSSMQP1
# Create Local Queues if none are defined for testing
runmqsc GSSMQP1
define qlocal(GSS.REQUEST.REPLY.QUEUE)
    end

#ACL the Local QMGR for Flume and the Queue(s) being used by Flume
setmqaut -m GSSMQP1 -t qmgr -g flume +connect +inq +dsp
setmqaut -m GSSMQP1 -t queue -n GSS.REQUEST.REPLY.QUEUE -g flume +connect +inq +browse +get +put +dsp

# Refresh Security on the Local QMGR
runmqsc GSSMQP1
    refresh security
end

Configure Flume as described in the github readme