I'm using the Kafka connect HDFS sink connector to write to HDFS from kafka, it is working fine. My messages look like this:
key: my-key
value: {
"name": "helen"
}
My use case is that in need to append the keys of my message to the events i send to HDFS.
The problem is that the key doesn't appear in the value payload so that i cannot use:
"partitioner.class":
"io.confluent.connect.hdfs.partitioner.FieldPartitioner",
"partition.field.name": "key",
My question is how can I add the key to the message I send to the HDFS or how can i partioned based on the key?