Publish non-string message in Cloud PubSub

Question

Almost all of the examples of pub/sub available over web uses String messages.

How do I publish message other than text messages like Java Object, JSON or AVRO to topic and then consume it through subscription.

Many Thanks Pari

Answer 1

You cannot really do that directly, as you can see here and here the published message has to be a bytestring. What you can do instead is load your file, encode it to utf-8 and then publish it. Afterwards, when pulling, you can dump the message data to a file. For instance, using python for json, you would need to publish like so:

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project, topic_name)

with open(json_path) as f:
    data = str(json.load(f))

data = data.encode('utf-8')
publisher.publish(topic_path, data=data)

Then you can pull and dump like so:

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    project, subscription_name)

def callback(message):
    with open('received_message.json', 'w') as outfile:
        json.dump(message.data, outfile)

subscriber.subscribe(subscription_path, callback=callback)

Answer 2

publish JSON with Node - publishMessage:

const {PubSub} = require('@google-cloud/pubsub')
const pubsub = new PubSub()

const json = {
  foo: 'bar'
}

await pubsub.topic('my-topic').publishMessage({json})

Answer 3

As of this writing

The node.js @google-cloud/pubsub library only accepts buffer objects and you'll need to stringify.

https://github.com/googleapis/nodejs-pubsub/blob/master/samples/topics.js

async function publishMessage(topicName, data) {

  // Imports the Google Cloud client library
  const {PubSub} = require('@google-cloud/pubsub');

  // Creates a client
  const pubsub = new PubSub();

  // Publishes the message as a string, e.g. "Hello, world!" or JSON.stringify(someObject)
  const dataBuffer = Buffer.from(data);

  const messageId = await pubsub
    .topic(topicName)
    .publisher()
    .publish(dataBuffer);
  console.log(`Message ${messageId} published.`);
}

---

This might change soon!

Please follow this request/issue: https://github.com/googleapis/nodejs-pubsub/issues/121

The library may soon be modified to accept non buffer objects and buffer them for you.

Answer 4

The following code lets you publish a message in Pub/Sub using JSON:

topic_path = 'your-topic-id'

publisher = pubsub_v1.PublisherClient()

record = {
    'Key1': 'Value1',
    'Key2': 'Value2',
    'Key3': 'Value3',
    'Key4': 'Value4'
}

data = json.dumps(record).encode("utf-8")
future = publisher.publish(topic_path, data)
print(f'published message id {future.result()}')

However, you must encode the JSON to UTF-8.

I hope that It helps you.