Read from PubSub in Dataflow: set the subscription name

0
votes

I'm my pipeline (Java) I have set the option of read from pubsub using the subscription or the topic:

    PCollection<PubsubMessage> messages = null;
    if (options.getUseSubscription()) {
        messages = pipeline.apply("ReadPubSubSubscription", PubsubIO.readMessagesWithAttributes()
                .fromSubscription(options.getInputSubscription()).withIdAttribute("messageId"));
    } else {
        messages = pipeline.apply("ReadPubSubTopic", PubsubIO.readMessagesWithAttributes()
                .fromTopic(options.getInputTopic()).withIdAttribute("messageId"));
    }

If you use the topic option, each time you run your pipeline, a new subscription is created.

Is there any way of avoiding that? Maybe set the name of the subscription that is created automatically from Dataflow?

2
javagoogle-cloud-dataflowapache-beamgoogle-cloud-pubsub

2 Answers

0
votes

I don't think Dataflow supports such a feature since this kind of information will not be shared across Dataflow pipelines. If you use the fromTopic() option, Dataflow will create a subscription when reading. But will it be possible for you to pre-create a subscription during pipeline submission if one has not been provided by the user and use that ?

0
votes

No, you can't do this. But before starting the pipeline, you can customize manually the options object, and for example, you can create, or recover, the subscription, add it to the options, and then perform your pipeline.