Error in pubsub subscriber: Max message size exceeded

Question

I am using Google Cloud Pubsub for my application. The subscriber to the pubsub topic is written in Javascript and runs on Nodejs. I am using the official pubsub client provided by google.

The code looks like this:

var topic = gcloud.pubsub({projectId: 'myProjectId'}).topic('topicName');
var pubsub = gcloud.pubsub({projectId: 'myProjectId'});
var sub = pubsub.subscription('subName', {topic: topic});
sub.on('error', function(err) { console.error(err); });
sub.on('message', messageHandler);

My error handler keeps getting triggered with this error message:

Error: Max message size exceeded

First of all, this is a very strange message to be receiving on the subscriber. If the message is too large, it should have been rejected when it was published.

Much more importantly, is that my subscriber just seems to die at some point. It keeps processing these messages until about 80 errors and then just stops. The message handler never gets called again.

How can I fix this?

Updated SDK version to 0.46.1. New error message:

Received message larger than max (10406691 vs. 4194304)

Kamal Aboul-Hosn Kamal Aboul-Hosn · Accepted Answer · 2017-02-06T19:19:07

This is likely a bug in the Node.js Pub/Sub client library where the max_receive_message_length property is not being set on the gRPC channel. The "message size" in this case does not refer to the size of a message published to Pub/Sub (which can be up to ~10MB based on the maximum publish request size), but rather to the size of the message allowed in a response sent to a request in gRPC. I would recommend creating an issue in the Google Cloud Platform Node.js repository.

In the meantime, you could try setting the maxInProgress property on the subscription to a smaller number, which should force the subscriber to pull smaller batches of messages, which should make the response smaller. Currently, if not specified, the maximum number of messages in a single response defaults to 1000. However, if your messages are large in size, this may not help.

Error in pubsub subscriber: Max message size exceeded

1 Answers