Flink Queryable State Error

Question

I am trying to use queryable state on Flink (version 1.4.2) but unfortunately I keep getting the following error:

INFO  my.test.flink.QueryableState  - Params are a96438fa12879b7598c9cf32684e2669, kafka-cluster_jobmanager_1, 6123
INFO  my.test.flink.QueryableState  - Before the call java.util.concurrent.CompletableFuture@26aa12dd[Not completed]
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: readerIndex(0) + length(4) exceeds writerIndex(0): PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 0)
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
        at my.test.flink.QueryableState.main(QueryableState.java:67)
Caused by: java.lang.IndexOutOfBoundsException: readerIndex(0) + length(4) exceeds writerIndex(0): PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 0)
        at org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1166)
        at org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf.readInt(AbstractByteBuf.java:619)
        at org.apache.flink.queryablestate.network.messages.MessageSerializer.deserializeHeader(MessageSerializer.java:231)
        at org.apache.flink.queryablestate.network.ClientHandler.channelRead(ClientHandler.java:76)
        at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
        at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
        at org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
        at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
        at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
        at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
        at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
        at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
        at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
        at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)

On the client side I am using flink-queryable-state-client-java_2_11.jar and the relevant part of code for the queryable client is

QueryableStateClient client = new QueryableStateClient(jobManagerHost, jobManagerPort);

TypeInformation<MyEvent> typeInformation = TypeInformation.of(new TypeHint<MyEvent>() {});
ListStateDescriptor<MyEvent> descriptor = new ListStateDescriptor<MyEvent>("myEvents",
                 typeInformation.createSerializer(new ExecutionConfig()));

CompletableFuture<ListState<MyEvent>> resultFuture =
                        client.getKvState(JobID.fromHexString(jobIdParam),"myEvents", "1", 
                        BasicTypeInfo.STRING_TYPE_INFO , descriptor );

logger.info("Before the call " + resultFuture);
try {
         logger.info("Finished"+ resultFuture.get());
 } catch(Exception ex) {
         ex.printStackTrace();
 }

Finally the job running on Flink has a ListState configured as it can been seen below. Note that data are keyed on ListState by String

        TypeInformation<MyEvent> typeInformation = TypeInformation.of(new TypeHint<MyEvent>() {});
        ListStateDescriptor<MyEvent> eventState = 
                new ListStateDescriptor<MyEvent>("myEvents",typeInformation);
        eventState.setQueryable("myEvents");
        eventListState = getRuntimeContext().getListState(eventState);

It seems to me like a serialization error but I do not know what I need to do to fix it. Does anybody have an idea what might be wrong with code above ? Am I missing something?

David Anderson David Anderson · Accepted Answer · 2018-05-18T17:07:21

I ran into that exact same problem when updating this queryable state demo for Flink 1.4. If I recall correctly, the important part is dealing with the CompletableFuture correctly -- you can't just call get() straightaway.

See the code for a working example, the key part of which looks something like this:

try {

    CompletableFuture<FoldingState<BumpEvent, Long>> resultFuture =
      client.getKvState(jobId, EventCountJob.ITEM_COUNTS, key, 
      BasicTypeInfo.STRING_TYPE_INFO, countingState);

    resultFuture.thenAccept(response -> {
      try {
        Long count = response.get();
        // now we could do something with the value
      } catch (Exception e) {
        e.printStackTrace();
      }
    });

    resultFuture.get(5, TimeUnit.SECONDS);

} catch (Exception e) {
  e.printStackTrace();
}

Flink Queryable State Error

1 Answers