Google Cloud - How to get data from remote PubSub as input for Apache Beam Pipeline executing locally?

0
votes

I've configured a local (java) environment for executing Apache Beam Pipeline following the official doc.

The sample project (WordCount) works perfectly, but now I'd like to change it and get input data from PubSub topic and put output to BigQuery.

I've already created my PubSub topic and my BigQuery dataset, but my question is: how can I configure REMOTE input (pubsub topic) and output (BigQuery) for a locally running Pipeline (for debug purpose)?

1
google-cloud-platformapache-beamgoogle-cloud-pubsub
Do you mean pubsub topic from a different project when you say REMOTE? Us access pubsub toip from a different project you can give complete name of the pubsub topic including project name projects/[project name]/topics/[topic name] – Ankur

1 Answers

0
votes

See classes PubsubIO / PubsubClient (or com.google.cloud.pubsub.v1).

For remote access you'd most likely need to register an external IP.