How to write to ORC files using BucketingSink in Apache Flink?

2
votes

I'm working on a Flink streaming program that reads kafka messages and dump the messages to ORC files on AWS s3. I found there is no document about the integration of Flink's BucketingSink and ORC file writer. and there is no such an ORC file writer implementation can be used in BucketingSink.

I'm stuck here, any ideas?

1
hdfsapache-flinkorc

1 Answers

2
votes

I agree, a BucketingSink writer for ORC files would be a great feature. However, it hasn't been contributed to Flink yet. You would have to implement such a writer yourself.

I'm sure the Flink community would help designing and reviewing the writer, if you would consider contributing it to Flink.