Google pubsub into HTTP triggered cloud function?

Question

Is it possible to trigger an HTTP cloud function in response to a pubsub message?

When editing a subscription, google makes it possible to push the message to an HTTPS endpoint, but for abuse reasons one has to be able to prove that you own the domain in order to do this, and of course you can't prove that you own google's own *.cloudfunctions.net domain which is where they get deployed.

The particular topic I'm trying to subscribe to is a public one, projects/pubsub-public-data/topics/taxirides-realtime. The answer might be use a background function rather than HTTP triggered, but that doesn't work for different reasons:

gcloud functions deploy echo --trigger-resource projects/pubsub-public-data/topics/taxirides-realtime --trigger-event google.pubsub.topic.publish ERROR: gcloud crashed (ArgumentTypeError): Invalid value 'projects/pubsub-public-data/topics/taxirides-realtime': Topic must contain only Latin letters (lower- or upper-case), digits and the characters - + . _ ~ %. It must start with a letter and be from 3 to 255 characters long.

This seems to indicate this is only permitted on topics I own, which is a strange limitation.

I'm trying to take data from a public pubsub topic (projects/pubsub-public-data/topics/taxirides-realtime) and put it into a google cloud function. I can't do this with an HTTP triggered cloud function because I don't own the domain, and I can't do it with a background function because it doesn't permit creating a subscription to a topic I don't control. @DougStevenson — FrobberOfBits
I don't think there's such a thing as a "public pubsub topic". You can only get messages from a topic from within the project where the topic has been created. — Doug Stevenson
@DougStevenson there is such a thing as a public pubsub topic -- the topic listed above is one such. stackoverflow.com/questions/38861321/… — FrobberOfBits
Could you edit your question to point to the documentation for public pubsub topics and how they're intended to work? — Doug Stevenson

Nebulastic Nebulastic · Accepted Answer · 2019-06-24T08:30:56

It is possible to publish from a pub/sub topic to a cloud function. I was looking for a way to publish messages from a topic in project A to a function in project B. This was not possible with a regular topic trigger, but it is possible with http-trigger. Overall steps to follow:

Creata a http-triggered function in project B.
Create a topic in project A.
Create a push subscription on that topic in project A.
Domain verification

Push subscription

Here we have to fill in three things: the endpoint, the audience and the service account under which the function runs.

Push Endpoint: https://REGION-PROJECT_ID.cloudfunctions.net/FUNC_NAME/ (slash at end)
Audience: https://REGION-PROJECT_ID.cloudfunctions.net/FUNC_NAME (no slash at end)
Service Account: Choose a service account under which you want to send the actual message. Be sure the service account has the "roles/cloudfunctions.invoker" role on the cloud function that you are sending the messages to. Since november 2019, http-triggered functions are automatically secured because AllUsers is not set by default. Do not set this property unless you want your http function to be public!

Domain verification

Now you probably can't save your subscription because of an error, that is because the endpoint is not validated by Google. Therefore you need to whitelist the function URL at: https://console.cloud.google.com/apis/credentials/domainverification?project=PROJECT_NAME.

Following this step will also bring you to the Google Search Console, where you would also need to verify you own the endpoint. Sadly, at the time of writing this process cannot be automated.

Next we need to add something in the lines of the following (python example) to your cloud function to allow google to verify the function:

if request.method == 'GET':
    return '''
        <html>
            <head>
                <meta name="google-site-verification" content="{token}" />
            </head>
            <body>
            </body>
        </html>
    '''.format(token=config.SITE_VERIFICATION_CODE)

Et voila! This should be working now.

Sources: https://github.com/googleapis/nodejs-pubsub/issues/118#issuecomment-379823198, https://cloud.google.com/functions/docs/calling/http

Google pubsub into HTTP triggered cloud function?

3 Answers