Kafka Streams: Daily & Monthly Transform

Question

The data being received from the stream should be aggregated at the Daily and Monthly level.

With the following version in consideration (but includes branch with return type of array):

https://kafka.apache.org/0101/javadoc/org/apache/kafka/streams/kstream/KStream.html

How can we use the transform method to perform both? Essentially, the Daily aggregation should first be complete prior to performing Month to date aggregation.

Hi Raman, this question needs some further details on the problem — Matthew Pigram
These details should include code that you've already tried which illustrates the problem. — Karl Richter
Hello Matthew, Karl, I haven't got to that point yet. The scenario is as below. I have a topic, which is at the detail level. I understand I can open a stream and then use transform (overriding the transform(key, val) method) to perform Daily aggregation, where the data will be written to the Store. I can then materialize it to another topic after the transform. How can the Daily Aggregated data be transferred to another stream on which another transformer can be applied, to perform Monthly aggregation. — Raman

Robin Moffatt Robin Moffatt · Accepted Answer · 2018-05-14T08:41:38

If you're using Kafka Streams, you might also look at KSQL, which is built on Kafka Streams. It enables you to simply use SQL to declare your stream processing, including aggregations:

CREATE TABLE DAILY_SALES_AGG AS \
SELECT STORE, PRODUCT, SUM(NETT_SALES) AS DAILY_SALES_TOTAL \
FROM SALES_STREAM WINDOW TUMBLING (SIZE 1 DAY) \
GROUP BY STORE, PRODUCT; 

CREATE TABLE MONTHLY_SALES_AGG AS \
SELECT STORE, PRODUCT, SUM(NETT_SALES) AS MONTHLY_SALES_TOTAL \
FROM SALES_STREAM WINDOW TUMBLING (SIZE 28 DAYS) \
GROUP BY STORE, PRODUCT;

For more information see the KSQL syntax reference, including aggregate functions.

More info on KSQL here:

Disclaimer: I work for Confluent, who lead the development of the open-source KSQL project

Kafka Streams: Daily & Monthly Transform

1 Answers