How to make a Subscriber sleep if there are no messages from publisher in Google PUBSUB

Question

I am using Google PubSub Publisher to trigger a notification and send a message to Subscriber when a document got uploaded in Google Cloud Storage Bucket.

I created a notification topic by -> gsutil notification create -t [TOPIC_NAME] -f json -e OBJECT_FINALIZE gs://[BUCKET_NAME]

My Subscriber Function is : **

def callback(message):
    try:
        #storage_client = storage.Client.from_service_account_json('storage_service_key.json')
        print('Received message: {}'.format(message.data.decode("utf-8")))
        data = json.loads(message.data.decode("utf-8"))

        filename = data['name']
        file_name = re.search(r'/(.*)', filename).group(1)
        #filelink = data['selfLink']
        print("Processing the file : {}".format(file_name))
        path = "sample/"+file_name
        download_files(path, file_name)
        rming.image_remover(file_name) ## my custom function
        message.ack()
        os.remove(file_name)


    except Exception as error_message:
         print("Error in callback method: {}".format(error_message))
         pass


flow_control = pubsub_v1.types.FlowControl(max_messages=1)

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(proj_name, sub_name)

streaming_pull_future = subscriber.subscribe(
    subscription_path, callback=callback,flow_control=flow_control
)
print('Listening for messages on: {}'.format(subscription_path))
with subscriber:
    try:
        # When `timeout` is not set, result() will block indefinitely,
        # unless an exception is encountered first.
        #flag=0
        streaming_pull_future.result()
    except Exception as error_message:  # noqa
        print("exception occured while handling subscription: {}").format(error_message)
        pass

** I want to deploy this Subscriber Function in a docker container. Then it will make the Subscriber run 24/7.

Can I put the subscriber to sleep/(not make a pull request) to Publisher when there are no messages available OR My point in other term is can I make Subscriber sleep at idle times when there are no incoming messages.

Any help is appreciated!!!!

GCP Pub/Sub has two modes. Push and pull. With pull, you basically ask GCP if it has messages. If none available, you still return immediately and it is up to you when you pull next. Think of this as polling. The second story is that a subscription can be flagged as push. What this means is that when a new message is available, GCP call YOUR app via a REST call and hand delivers you the message. This may be what you need if your messages are infrequent. — Kolban
In addition to @Kolban comment, don't implement pull (polling). This consumes API calls and you can hit quota limits. — John Hanley
Thanks Kolban and John!!! I am new to GCP and PUB/SUB can you guys give me a working example where a subscription can be flagged as push in python... — Vighnesh.P Vicky
And an additional query will I be able to get the same kind of attributes like "name", "'selfLink" from the message attributes provided by Publisher as used in the code and will I be able to use the subscriber with callback and flow_control? — Vighnesh.P Vicky
Here is a link to docs on push ... cloud.google.com/pubsub/docs/push if you also google search on "gcp pubsub push" you'll get a ton of hits including some of the great videos at youtube.com/user/googlecloudplatform — Kolban

guillaume blaquiere guillaume blaquiere · Accepted Answer · 2020-03-27T10:18:53

In addition, and in summary of the comments, there is many things

You want to put your function into a container. Thus use Cloud Run for your app
Define a push subscription on your topic that call the endpoint of your Cloud Run service
- Be careful, the message format is slightly different between push subscription and direct call to function
If your Cloud Run is deployed in private mode (that I recommend you), I mean you don't set the --allow-unauthenticated param when you deploy your Cloud Run, create a service account with the role run.invoker and set this service account on your Push subscription for allowing PubSub to call your Cloud Run. Follow correctly the documentation, you have to grant the PubSub service agent with the token creator role.

Cloud Run scale to 0 and you may only the processing time, rounded to the upper 100ms. It sleeps and costs nothing when there is no message.

How to make a Subscriber sleep if there are no messages from publisher in Google PUBSUB

1 Answers