Unable to create a google cloud dataflow template through CLI command using Java

Question

I am working on google app engine project, I have created one class for google cloud dataflow job and tried to follow the steps https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#creating-and-staging-templates? But when I try to execute the command provided in the link my dataflow job start execution immediately, I dont want to start execution I just want to create a template and execute it later on

Command that I tried to execute and create template:

mvn compile exec:java -Dexec.mainClass=com.testUtils.TimeSpentDataFlow -Dexec.args="--runner=DataflowRunner --project=my-project-id --stagingLocation=gs://staging --templateLocation=gs://MyTemplateData16Apr19"

I have also try with Custom Option Class but no luck. below is my code snippet

    public static void main(String[] args) {
        logger.info("main start");
        init();
        logger.info("main end");
    }
    private static void init() {
        DataflowPipelineOptions dataflowOptions = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
        dataflowOptions.setGcpTempLocation("gs://gcpTempLocation/");
        dataflowOptions.setProject("my-project-id");
        dataflowOptions.setStagingLocation("gs://stagingLocation/");
        dataflowOptions.setRunner(DataflowRunner.class);

        FileSystems.setDefaultPipelineOptions(dataflowOptions);
        Pipeline p = Pipeline.create(dataflowOptions);
    p.apply(Create.of("5435798895722496"))
         .apply(new PrintData());
         p.run().waitUntilFinish();
    }

When I try to execute the command for default wordcount program the template get created but if I try the same command with my custom Class it start executing

I have solved the problem by setting below option - ` dataflowOptions.setTemplateLocation("gs://MyTemplateData16Apr19"); ` — SylvesterTheKid

Andrew Pilloud Andrew Pilloud · Accepted Answer · 2019-04-16T20:12:23

Your command line args aren't being passed into the pipeline options. If you want to use the command line args, replace the block initializing 'dataflowOptions' with:

DataflowPipelineOptions dataflowOptions =
     PipelineOptionsFactory.fromArgs(args)
        .withValidation()
        .as(DataflowPipelineOptions.class);

The other option is to set the template location:

dataflowOptions.setTemplateLocation("gs://MyTemplateData16Apr19");

Unable to create a google cloud dataflow template through CLI command using Java

1 Answers