Post from GCP Pub Sub to on-prem Kafka

Question

I have a requirement to publish messages from Google Pub/Sub topic to a Kafka running on my on-prem infrastructure. I stumbled on this link.

https://docs.confluent.io/current/connect/kafka-connect-gcp-pubsub/index.html

This should work. Wanted to know if you've used any other alternative solution to achieve this?

what do you expect as alternative? There is no native connector between Kafka and Pubsub. So you need to have an intermediary process that link the both worlds. It can be a compute engine, a cloud functions, your on prem servers (...) you need a compute stuff to achieve this. — guillaume blaquiere

rmesteves rmesteves · Accepted Answer · 2020-08-24T14:49:34

If you need to integrate PubSub and Kafka I suggest that you create a script for this purpose. In Python for example we have libraries for both PubSub and Kafka

Based on that, you could create a script more or less like below and run it inside some processing resource like Compute Engine or in your on premises server:

from google.cloud import pubsub_v1
from kafka import KafkaProducer


def callback(message):
     print(message.data)
     producer.send('<your-topic>', message.data)
     message.ack()


producer = KafkaProducer(bootstrap_servers='localhost:1234')  //Change it for your real parameter

subscription_name = "projects/<your-project>/subscriptions/<your-subscription>"
subscriber = pubsub_v1.SubscriberClient()

future = subscriber.subscribe(subscription_name, callback)

Post from GCP Pub Sub to on-prem Kafka

1 Answers