Kafka Streams Split 1 Stream in Multiple Stream

Question

I got a Projekt to learn working with Kafka Streams, but I have really trouble with it. I am working on kafka-streams version 1.0.1. We have a Main Topic Stream with Messages with the following Style:

{
    "phenomenonTime" : "2017-04-03T16:08:19.000Z",
    "resultTime" : "2017-04-03T16:08:19.000Z",
    "result" : {
      "Temperature" : 0,
      "Pressure" : 0,
      "Humidity" : 0,
      "Mean altitude" : 0,
      "Mass PM2.5" : 7.4,
      "Mass Error PM2.5" : 1.5,
      "Mass PM10" : 12.3,
      "Mass Error PM10" : 1.5
    }
  }

It's a Json Format, there is my first Probleme, I dont know how exactly use the Json Deserializer or Serializer.

But my Main Target is to create the Topics Temperature,Pressure,Humidity,Mean altitude... out of the Main Topic his Result Field with the right values in the Temperature Topic.

How can I realise that with Kafka Streams? I hope you can help me, to get better started with Kafka Streams.

EDIT:

Hole Message + key (formatted)

    Key c45e9532-9810-11e8-8839-03e1e3365152
    Value { "phenomenonTime" : "2017-04-03T16:08:09.000Z",
 "resultTime" : "2017-04-03T16:08:09.000Z",
 "result" : { "Temperature" : 0,
 "Pressure" : 0, 
"Humidity" : 0, 
"Mean altitude" : 0, 
"Mass PM2.5" : 7.1,
 "Mass Error PM2.5" : 1.5,
 "Mass PM10" : 9.6, "Mass Error PM10" : 1.5 },
 "[email protected]" : "http://localhost:8080/FROST-Server/v1.0/Observations('c45e9532-9810-11e8-8839-03e1e3365152')/Datastream", 
"Datastream" : { "unitOfMeasurement" : { "name" : null, "symbol" : null, "definition" : null }, "@iot.id" : "geo.uni-augsburg.de/Fixed-Wing-UAV-1/Datastreams/LOAC_LOCAL_201704031605.mass" }, 
"[email protected]" : "http://localhost:8080/FROST-Server/v1.0/Observations('c45e9532-9810-11e8-8839-03e1e3365152')/FeatureOfInterest",
 "FeatureOfInterest" : { "@iot.id" : "c458a1a4-9810-11e8-8839-176a6dbe6951" }, "@iot.id" : "c45e9532-9810-11e8-8839-03e1e3365152", "@iot.selfLink" : "http://localhost:8080/FROST-Server/v1.0/Observations('c45e9532-9810-11e8-8839-03e1e3365152')" }

Unformated:

Key c45e9532-9810-11e8-8839-03e1e3365152
Value { "phenomenonTime" : "2017-04-03T16:08:09.000Z", "resultTime" : "2017-04-03T16:08:09.000Z", "result" : { "Temperature" : 0, "Pressure" : 0, "Humidity" : 0, "Mean altitude" : 0, "Mass PM2.5" : 7.1, "Mass Error PM2.5" : 1.5, "Mass PM10" : 9.6, "Mass Error PM10" : 1.5 }, "[email protected]" : "http://localhost:8080/FROST-Server/v1.0/Observations('c45e9532-9810-11e8-8839-03e1e3365152')/Datastream", "Datastream" : { "unitOfMeasurement" : { "name" : null, "symbol" : null, "definition" : null }, "@iot.id" : "geo.uni-augsburg.de/Fixed-Wing-UAV-1/Datastreams/LOAC_LOCAL_201704031605.mass" }, "[email protected]" : "http://localhost:8080/FROST-Server/v1.0/Observations('c45e9532-9810-11e8-8839-03e1e3365152')/FeatureOfInterest", "FeatureOfInterest" : { "@iot.id" : "c458a1a4-9810-11e8-8839-176a6dbe6951" }, "@iot.id" : "c45e9532-9810-11e8-8839-03e1e3365152", "@iot.selfLink" : "http://localhost:8080/FROST-Server/v1.0/Observations('c45e9532-9810-11e8-8839-03e1e3365152')" }

But the [email protected], Datastream .... are not the Important. But the Key must be the same.

Thats how it looks exactly (https://i.imgur.com/zvwf3g7.png)

Hole exported Stream:

https://pastebin.com/PUfhL8fK

Example Kafka Client :

https://pastebin.com/y4k7fQgz

Vasyl Sarzhynskyi Vasyl Sarzhynskyi · Accepted Answer · 2018-08-04T17:04:21

for that you need to create multiple KStream objects for each required destination topic. for extracting required fields from main json, use mapValues method on kStream. to simplify work with json values, you can use JsonSerde from spring-kafka library (groupId: org.springframework.kafka, artifactId: spring-kafka).

example for temperature & pressure topics (and do the same for each of required destination topics):

Map<String, String> streamProperties = new HashMap<>();
streamProperties.put("bootstrap.servers", "localhost:9092");
streamProperties.put("key.serde", "org.apache.kafka.common.serialization.Serdes$StringSerde");
streamProperties.put("value.serde", "org.apache.kafka.common.serialization.Serdes$StringSerde");

Map<String, String> streamProperties1 = new HashMap<>(streamProperties);
streamProperties1.put("application.id", "temperature");
Map<String, String> streamProperties2 = new HashMap<>(streamProperties);
streamProperties2.put("application.id", "pressure");

Class<Map<String, Object>> genericMapClass = (Class) Map.class;
Consumed<String, Map<String, Object>> consumed = Consumed.with(Serdes.String(), new JsonSerde<>(genericMapClass));
Produced<String, Map<String, Object>> produced = Produced.with(Serdes.String(), new JsonSerde<>(genericMapClass));

StreamsBuilder streamBuilder1 = new StreamsBuilder();
KStream<String, Map<String, Object>> temperatureKStream = streamBuilder1.stream("mainSourceTopic", consumed);
temperatureKStream.mapValues((generalDetails) -> {
    Object temperatureValue = ((Map) generalDetails.get("result")).get("Temperature");
    Map<String, Object> temperatureMessageDetails = new HashMap<>();
    temperatureMessageDetails.put("Temperature", temperatureValue);
    temperatureMessageDetails.put("phenomenonTime", generalDetails.get("phenomenonTime"));
    temperatureMessageDetails.put("resultTime", generalDetails.get("resultTime"));
    System.out.println("temperatureMessageDetails: " + temperatureMessageDetails);
    return temperatureMessageDetails;
}).to("temperatureTopic", produced);

StreamsBuilder streamBuilder2 = new StreamsBuilder();
KStream<String, Map<String, Object>> pressureKStream = streamBuilder2.stream("mainSourceTopic", consumed);
pressureKStream.mapValues((generalDetails) -> {
    Object pressureValue = ((Map) generalDetails.get("result")).get("Pressure");
    Map<String, Object> pressureMessageDetails = new HashMap<>();
    pressureMessageDetails.put("Pressure", pressureValue);
    pressureMessageDetails.put("phenomenonTime", generalDetails.get("phenomenonTime"));
    pressureMessageDetails.put("resultTime", generalDetails.get("resultTime"));
    System.out.println("pressureMessageDetails: " + pressureMessageDetails);
    return pressureMessageDetails;
}).to("pressureTopic", produced);

StreamsConfig streamsConfig1 = new StreamsConfig(streamProperties1);
KafkaStreams kafkaStreams1 = new KafkaStreams(streamBuilder1.build(), streamsConfig1);
kafkaStreams1.start();

StreamsConfig streamsConfig2 = new StreamsConfig(streamProperties2);
KafkaStreams kafkaStreams2 = new KafkaStreams(streamBuilder2.build(), streamsConfig2);
kafkaStreams2.start();

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    kafkaStreams1.close();
    kafkaStreams2.close();
}));

Kafka Streams Split 1 Stream in Multiple Stream

1 Answers