Spark Structured Streaming- Unable to print data in logs

Question

I'm consuming data from Kafka topic with JSON data format using Spark Structured Streaming code in Scala and writing the data into another kafka sink.

We are submitting the job in cluster mode, and we are trying to check the no. of messages coming in one batch but we are unable to see the no. of messages coming in a batch being written into the logs.

Is it possible to find out the no, of messsages coming in a batch to be written into the log file.

mike mike · Accepted Answer · 2020-03-31T07:59:23

According to the Spark Structured Streaming + Kafka integration Guide you get the following data for each row of the source:

Column         Type
key            binary
value          binary
topic          string
partition      int
offset         long
timestamp      long
timestampType  int

Using the offset you should be able to log the amount of offsets you are processing.

Spark Structured Streaming- Unable to print data in logs

1 Answers