Message attributes on spring-integration-aws flows cause SQS messages to loop, where's the best place to filter them?

Question

So, I created a couple of short flows, using SQS as the backing store between processing segments. The basic flow is:

RestController -> SQS Queue1 OCA
  SQS Queue1 MDCA -> Service-Adapter -> SQS Queue2 OCA
  SQS Queue2 MDCA -> Service Adapter

However, I ran into a couple of problems... The "SQS Queue1 MDCA" reads messages off the queue with AWS specific Message Headers that end up getting to the outbound adapter writing to Queue2. These message headers specify things like the AWS_queue name, aws message id, etc, which cause the message to be re-delivered to queue1.

Once I filtered these headers off (I used a simple custom transformer to test this), the attributes of the outbound-channel-adapter then work as expected and the QueueMessagingTemplate delivers the message to the proper queue.

So, my question for the spring-int and spring-aws experts is, "Where do you think the proper place to filter out any pre-existing SQS headers is so that they don't get picked up by any downstream SQS processing?" Seems to me like you'd want to do it in the SQSHandler, since they might be relevant in any base spring-aws-messaging calls.

As a side note, with a message being created from an SQS ICA, and an object-to-json transformer in the flow, it also caused 12 message headers to be created which is 2 more than is allowed by SQS (causing the message delivery to fail).

Sample message with headers slightly modified to protect the innocent queues... As you can see the aws_queue, destination, etc are still in the message from being read from "wrench_workflow-mjg" so that when trying to deliver to the next queue, those headers overwrote the configuration in the spring-int xml configuration and never delivered to the next queue. The message just kept cycling through SQS queue "wrench_workflow-mjg" (after fixing the more than 10 attributes problem).

GenericMessage [payload={"eventId":"event-1","eventStartDateTime":1476127827.201000000}, 
headers={aws_messageId=db9b6cc0-f133-4182-b79c-4d5d9717a3a9, ApproximateReceiveCount=1, 
SentTimestamp=1476127827803, id=0b662b72-f149-a970-5e63-64a1b28290fb, 
SenderId=AIDAJOYV7TECZCZCOK22C,
aws_receiptHandle=AQEBdaToWo9utbjBwADeSuChX/KrY3l8eoSBMZ0RMmbI8ewjJQw6tV74RwSYxWPKZBSzgJhCmfJ8AUX+
reOy2yGrmefULU7NS8nqYTdMW6BB4Ug2+mHIY+8Tze+2ws15FB5t96q3iJO8+tP5pl/xuo+CiTDv+
L1rlYkVODD0Yv1zosGKx48IhGOXdL8nJ4Im8bujcUNLJy/vEYD8Qcfsi6NHOF3Qn0A4Xw+Nva0wp86zfq,
aws_queue=wrench_workflow-mjg,
lookupDestination=wrench_workflow-mjg, 
ApproximateFirstReceiveTimestamp=1476127827886, timestamp=1476127836254}

I can probably pull together an example if necessary, but I'm hoping you understand what I'm getting at without.

Gary Russell Gary Russell · Accepted Answer · 2016-10-11T12:51:08

We should probably add header-mapping facilities to the adapters to enable selective mapping of headers like we have for other technologies in Spring Integration.

I have opened a GitHub Issue.

In the meantime, you could put an <int:header-filter.../> (or HeaderFilter transformer) immediately upstream of the outbound adapter to remove the unwanted headers.

You could also add a custom Request Handler Advice to the outbound adapter.

Message attributes on spring-integration-aws flows cause SQS messages to loop, where's the best place to filter them?

1 Answers