Is there a way to write data to parquet files if I don't know the schema before hand using Apache Flink?

Question

I have multiple event streams and I constantly keep integrating new kinds of events into my Kafka streaming layers. I currently have 3 different types of complex events coming in from my web streams and I am planning on integrating 6 more over this year. Is there a way to write all these data streams to parquet without me having to configure multiple flink datastreams for each topic with a predefined schema?

Answer 1

You can have multiple side outputs, each for a specific event type with it's own configured parquet writer. For each different event, you can provide a different schema.