MQ - Data Compression on queue

Question

Problem: MQ7 has a hard limit of a maximum 100MB jms message. For large payloads ( xml ) that are close that, can this be compressed on the queue to shorten the data length?

I tried compressing a 7MB jms string message using the MQ ZLIB compression on the svr.def.conn channel and it didn't make any difference to the data length of the jms message. I only set the one channel and expected the channel that is used would compress the data going into the queue.

MQ Server: 7.5 Client: JAVA Message Type: String

Answer 1

Channel level compression is used to compress the data in transit between the two ends of the channel, in your case between the JMS client and the MQ SVRCONN channel. The messages themselves will be compressed while going over the network but not while sitting on the queue.

Answer 2

I would recommend to commpress the payload and use a ByteMessage. Message properties can be used to qualify the payload type, similar to HTTP e.g. "Content-Encoding", "Content-Type"

String payload = ...; // the xml

Session session = ...;
BytesMessage bytesMessage = session.createBytesMessage();

bytesMessage.writeBytes(compressGZIP(payload, StandardCharsets.UTF_8));
bytesMessage.setStringProperty("Content-Encoding", "gzip");
bytesMessage.setStringProperty("Content-Type", "text/xml; charset=utf-8");

Here is the compressGZIP method:

private byte[] compressGZIP(String string, Charset charset) throws IOException {
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    try (GZIPOutputStream out = new GZIPOutputStream(byteArrayOutputStream)) {
        StringReader stringReader = new StringReader(string);
        // IOUtils from apache commons-io
        IOUtils.copy(stringReader, out, charset);
    }
    return byteArrayOutputStream.toByteArray();
}

The consumer can then ask for the message properties, decompress and re-create the xml based on the "Content-Encoding" and "Content-Type" message properties.

Something like this

public void onMessage(Message message) {
    BytesMessage bytesMessage = (BytesMessage) message;

    long bodyLength = bytesMessage.getBodyLength();
    byte[] rawPayload = new byte[(int) bodyLength];
    InputStream payloadInputStream = new ByteArrayInputStream(rawPayload);

    String contentEncoding = bytesMessage.getStringProperty("Content-Encoding");
    if("gzip".equals(contentEncoding)) {
        payloadInputStream = new GZIPInputStream(payloadInputStream);
    }

    String contentType = bytesMessage.getStringProperty("Content-Type");
    MimeType mimeType = new MimeType(contentType); // from javax.activation

    if("text".equals(mimeType.getPrimaryType())) {
        if("xml".equals(mimeType.getSubType())) {
            Charset charset;

            String charsetString = mimeType.getParameter("charset");
            if(charsetString != null) {
                charset = Charset.forName(charsetString);
            } else {
                charset = StandardCharsets.UTF_8; // default
            }

            Reader reader = new InputStreamReader(payloadInputStream, charset);

            String xml = IOUtils.toString(reader);
            IOUtils.closeQuietly(reader);
        }
    }
}

The advantage of this solution is that you stay on the standard JMS api instead of using a provider specific configuration.

The disadvantage is that the sender and receiver must implement content-type handling.

Thus you have to make a decision between portability and implementation effort.