how to post message from Google Cloud pubsub(gcp) to a remotely running Kafka topic

Question

Need help to get some help to publish all message from GCP(Google Cloud Pubsub) to a remotely running Kafka Topic

I've used the connector code from "https://powerspace.tech/how-to-stream-data-from-google-pubsub-to-kafka-with-kafka-connect-dbef1c340a76" and was able to move data from GCP to Kafka topic. The Kafka and the connector runs in the same machine, now I need to just have the connector code and add the messages from PubSub to remotely running Kafka Topic

Answer 1

Copying messages from Google Cloud Pub/Sub to a Kafka topic running remotely should not be much different than if everything were located in GCP. There are two options:

1. Run Kafka connect with your Kafka deployment. In this scenario, you have to ensure that the machines running Kafka have permission to make outgoing requests to Google Cloud on port 443.
2. Run Kafka on GCP (likely in a GCE instance). You will have to set the bootstrap.servers property in the Kafka connect configuration to a list of servers where Kafka is running. This means that those servers will have to be publicly addressable and allow incoming connections.