0
votes

I am using elasticsearch connector (confluent) for kafka connect. I am getting messages from topic into elasticsearch index. I can see that offset is part of _id if no key is used. I want to browse the messages in elasticseach based on offsets and timestamp.

Is it possible to get the offset and timestamp of message as fields in the index?

Here is the elasticsearch document

{ "_index": "test-elasticsearch-sink", "_type": "kafka-connect", "_id": "test-elasticsearch-sink+0+0", "_score": 1, "_source": { "f1": "value1" } }

Thanks, Rajesh

2

2 Answers

0
votes

This is not currently possible, but we are thinking along the lines of a feature that will allow hoisting record metadata like timestamp, topic, partition, offset to fields.

0
votes

Using Single Message Transforms you can indeed add new fields including values such as offset and timestamp. Have a look at org.apache.kafka.connect.transforms.InsertField.