Flink Custom serialization with registerTypeWithKryoSerializer

0
votes

I'd like to advice a quick possible improvement you could do on the DOC online wrt Serialization. As matter of fact you did an amazing job both on the implementation and on the documentation. The way flink automatically understands how to best serialize objects is very smart and powerful.

While developing a real-time analytics project that is going to leverage Flink I encountered an issue that is more related to missing doc than to flink. I'd like to suggest here to amend, as it could spare several hours of despare of other people in the future :)

I had a couple classes needed custom serializers. I created Kryo serializers and also plugged those with registerTypeWithKryoSerializer. What was not clear in the current doc is that since some of those are POJO, Flink prefers that over GenericType that then uses my kryo serializers. Once I understood, after several hours of deep debugging, I just made sure those were not POJO any more, then all of a sudden my serializers were used. So on one side you could think about always preferring custom serializers over POJO. But in the very short term I'd just suggest to amend the doc.

Let me know what you think and congrats for this amazing piece of work. For previous projects we did use storm or spark streaming, but Flink is miles ahead for real-time streaming analytics.

Thanks and keep up the good work!

1
apache-flinkflink-streaming

1 Answers

0
votes

So the current quick workaround is to make sure your objects are not POJO. In other case they are not serialized via GenericType that uses Kryo and sees your custom serializers.

Very useful to debug when you have this kind of issues is the: env.getConfig().disableGenericTypes();

That allows to stop task startup with exception allowing you to check what kind of serializers and HintTypes were used.