Remove topic name from S3 paths when using Kafka Connect S3 Sink

0
votes

Using https://docs.confluent.io/current/connect/kafka-connect-s3/index.html

How would I set up the connector so that the Kafka topic names are not included in the output S3 paths?

I'm using the TimeBasedPartitioner and I'd like to have a fixed name in the path instead of the topic name. So this:

fixed_name/<encodedPartition>/<topic>+<kafkaPartition>+<startOffset>.<format>

Instead of the default:

topics/<topic>/<encodedPartition>/<topic>+<kafkaPartition>+<startOffset>.<format>

Thank you!

1
amazon-s3apache-kafkaapache-kafka-connect

1 Answers

1
votes

You cannot remove the topic name unless you write your own Partitioner and overwrite generatePartitionedPath