Google Cloud Platform: Pub/Sub to Bigtable

Question

I am building a pipeline on Google Cloud Platform in Python. I have my data in Cloud Pub/Sub. I want to store it into Bigtable by using Dataflow. So far I have some examples of streaming data from Pub/Sub to Bigtable in java.

Can anyone please help me with some resources or links on how to stream my data from Pub/Sub to Bigtable using Dataflow in Python?

Balu Balu · Accepted Answer · 2017-08-23T06:14:56

Sam, I am not sure how we do this in Python. But I have done this in Java. Hope the idea may help you to solve your problem.

steps that you should keep in mind while doing this are

Read from Pub/Sub, Set the streaming to true

PubsubIO.readStrings().fromTopic(PUBSUB_SUBSCRIPTION))

Group the collection with a constant key

PCollection<KV<String, String>> keyedStream = streamData
        .apply(WithKeys.of(new SerializableFunction<String, String>() {
            /**
             * serial version id
             */
            private static final long serialVersionUID = 1L;

            public String apply(String s) {
                return CONSTANT_KEY;
            }
        }));

A transform is created for dividing the elements in the PCollection into windows and trigger the control when those elements are output.

Window.<String>into(new GlobalWindows())
        .triggering(Repeatedly
                .forever(AfterProcessingTime
                        .pastFirstElementInPane()
                        .plusDelayOf(Duration.standardSeconds(30))
                    )).withAllowedLateness(Duration.standardDays(1)).discardingFiredPanes()

Write the PCollection to Bigtable.

Google Cloud Platform: Pub/Sub to Bigtable

1 Answers