Hazelcast Jet 0.6.1 - Dag Definition

Question

The Hazelcast Jet prints the DAG definition on the console,once started

This converts the Pipeline definition to the DAG.

Here is a Pipeline definition.

    private Pipeline buildPipeline() {
 Pipeline p = Pipeline.create();
 p.drawFrom(Sources.<String, Record>remoteMapJournal("record", getClientConfig(), START_FROM_OLDEST))
          .addTimestamps((v) ->  getTimeStamp(v), 3000)
          .peek()
          .groupingKey((v) -> Tuple2.tuple2(getUserID(v),getTranType(v)))
    .window(WindowDefinition.sliding(SLIDING_WINDOW_LENGTH_MILLIS, SLIDE_STEP_MILLIS))
    .aggregate(counting())
    .map((v)-> getMapKey(v))
    .drainTo(Sinks.remoteMap("Test", getClientConfig()));
    return p;
  }

and here is a DAG definition printed on console.

 .vertex("remoteMapJournalSource(record)").localParallelism(1)
.vertex("sliding-window-step1").localParallelism(4)
.vertex("sliding-window-step2").localParallelism(4)
.vertex("map").localParallelism(4)
.vertex("remoteMapSink(Test)").localParallelism(1)
.edge(between("remoteMapJournalSource(record)", "sliding-window-step1").partitioned(?))
.edge(between("sliding-window-step1", "sliding-window-step2").partitioned(?).distributed())
.edge(between("sliding-window-step2", "map"))
.edge(between("map", "remoteMapSink(Test)"))

Is there any way to get the DAG definition with all the details like sliding window details, aggregation APIs etc ?

Oliv Oliv · Accepted Answer · 2018-06-19T06:08:44

No, it's technically impossible. If you write a lambda (for example for a key extractor), there's no way to display the code that defined the lambda. The only way for you to get more information is to embed that information into the vertex name.

In Jet 0.7, this printout will be changed to the graphviz format so that you can copy-paste it to a tool and see the DAG as an image.

Hazelcast Jet 0.6.1 - Dag Definition

1 Answers