Spark Structured Streaming stream-stream join question

1
votes

Scenario is slightly different than the classic stream-stream join

streamA: stream of transactions: transTS, userid, productid,...

streamB: stream of new products created: productid, productname, createTS, ...)

I want to join transactions with productIds, but I can't find a combination of watermarks/join conditions to make that happen.

streamA_wm = streamA.withWatermark("transTS", "3 minutes")
streamB_wm = streamB.withWatermark("createTS", "1 day")

streamA_wm
   .join(streamB_wm, "productId AND transTS >= createTS", "leftOuter")

The result is empty.

What am I doing wrong?

1
apache-sparkspark-structured-streaming
Curious as to why the Products would be a Stream. - thebluephantom
Because new products are created continuously, and I need some attributes of products in the join with the transactions. - Benedetto

1 Answers

1
votes

I think you may have the wrong approach here. Whilst products are transactional when created and updated, they are metadata relative the other Transaction Stream.

I would suggest the following:

  1. Join the Transactions Stream to the reference data Products - which is not subjected to Stream processing.
  2. Do not cache the Products, this ensures you go to source.
  3. Use parquet, KUDU for the Products.

But may be there is a reason for a Stream for Products, but ... What happens if no more updates to Product made and you get data for that Product again via the Stream for Transactions?