How to read from a PubSub sub-directory within dataflow pipeline

Question

I am sending messages via MQTT to Google IoT core.
My Iot registry is associated to 3 PubSub topics, each topics has several sub-directory.

My messages are fowarded to PubSub and then treated by a dataflow pipeline.
In my pipeline I read from Pub/Sub as follow:

    common = (
        pipeline
        | 'Get common on pub/sub' >> beam.io.ReadFromPubSub(
        topic='projects/project-id/topics/topics-id',
    with_attributes=True)
)

but I'd like to be able to read from my pubsub topic Sub-Directory, is it possible ?
I already tried to add it like this:

topic='projects/project-id/topics/topics-id/sub-directory'

not successful

What a sub directory in PubSub? Where did you see this concept? What is it for you? — guillaume blaquiere
I think Allison's answer covers your use case and provides you with the documentation needed. Just to clarify: seems like you mixed MQTT and Pub/Sub topic. Pub/Sub does not have the concept of subfolders as MQTT does, so you need to match each resource as indicated. Messages are forwarded to the default Cloud Pub/Sub topic instead of the additional Cloud Pub/Sub topic if the subfolder is specified in the MQTT topic doesn't have a matching Pub/Sub topic in the device registry. — Sergi Muñoz

Allison Fisher Allison Fisher · Accepted Answer · 2020-10-29T21:15:22

You could publish to additional subdirectory topics and have separate subscriptions reading from those. From the documentation on publishing telemetry events to additional Cloud Pub/Sub topics:

Devices can publish data to additional Cloud Pub/Sub topics. By default, MQTT messages published to /devices/DEVICE_ID/events are forwarded to the corresponding registry's default telemetry topic. You can specify a subfolder in the MQTT topic to forward data to additional Cloud Pub/Sub topics. The subfolder is the subtopic after /devices/DEVICE_ID/events.

The documentation linked above provides more details about how to set this up, and the cases where messages WILL and WON'T be published to subdirectory topics.

How to read from a PubSub sub-directory within dataflow pipeline

1 Answers