hazelcast topic as a hazelcast-jet source

1
votes

I'm trying to get more up to speed with Hazelcast Jet as a processing pipeline and began digging into the Sources documentation a bit more

https://jet-start.sh/docs/api/sources-sinks#sources

I was very surprised to see that Hazelcast Topics and ReliableTopics aren't natively supported sources, however Kafka and JmsQueue are supported out of the box. I do see that a hazelcast reliabletopic is a supported sink... which makes me question whether or not I am misunderstanding a topic as a valid source for a pipeline.

Am I misunderstanding something?

1
hazelcasthazelcast-jet

1 Answers

2
votes

Hazelcast ITopic is a non-distributed in-memory facility, and doesn't support exactly-once processing. These are probably some of the reasons Hazelcast hasn't so far offered first-class support for it. Kafka is an example of a persistent, distributed and fault-tolerant facility, which makes it a good match for an architecture where you require the exactly-once consistency level.

On the other hand, you can use the SourceBuilder and SinkBuilder to implement your own connectors for it. Here's an example from our code samples repository.