Buffering Spring integration publish-subscribe channel

Question

I have a spring integration application that has a pub-sub channel with two subscribers - one publishes the message to Kafka and the other writes the message to a file. The problem is that the service-activator that writes the message to the file is not able to keep up with the speed of the other service-activator that produces to Kafka. This causes a slow down in the overall message processing rate. To overcome this, I have added an extra layer in between the pub-sub channel and the service-activator that writes to file. A transformer, the does nothing but consumes the message and puts the messages into a direct channel which is consumed by the filewriter. This has improved performance in my case but I was wondering if this is the right way to do? Sample configuration below:

<int:publish-subscribe-channel id="pschannel"/>
<int:service-activator id="kafkaSA" ref="producer" input-  channel="pschannel" method="publish"/>
<int:transformer input-channel="pschannel" ref="dummytransformer" method="doNothing" output-channel="directChannel"/>
<bean id="dummytransformer" class="org.test.DummyTransformer"/>
<int:channel id="directChannel">
    <int:queue capacity="200000" />
<int:channel>
<int:service-activator id="fileSA" ref="filewriter" input-channel="directChannel" method="publish" >
    <int:poller max-messages-per-poll="10000" fixed-delay="100" />
</int:service-activator>

Artem Bilan Artem Bilan · Accepted Answer · 2017-05-23T13:22:24

First of all it isn't direct because it is really <queue> by your config.

Well, this is really one way to and you definitely don't block your Kafka producer (the first subscriber).

You should consider an infinite config for the queue and poller:

<int:channel id="directChannel">
    <int:queue/>
<int:channel>
...
<int:poller fixed-delay="100" />

This way the consumer on that directChannel will process messages in its best pace not causing lags in other places.

Another way to do distribution is a task-executor for the publish-subscribe-channel - all the subscribers will be executed in their one thread.

But yeah, any way you should bare in mind that you always will have a lag between Kafka and that file producer.

You don't need any DummyTransformer, BTW. There is a special <bridge> component on the matter: http://docs.spring.io/spring-integration/reference/html/messaging-channels-section.html#bridge

Buffering Spring integration publish-subscribe channel

1 Answers