Is it possible to stop streaming jobs without lose the current data?

Question

I have a streaming pipeline that reads from a PubSub subscription. The pipeline applies 1 hour's fixed windows and after doing some transforms, such as GroupByKey, writes the data to BigQuery.

My problem is that if I stop the job execution, the data of the current window are not written to BigQuery and these are lost since neither are in the subscription nor written in BigQuery.

Marcos - do you want to reload your pipeline with a new version of your code? Or do you want to drain out the pipeline and then stop it permanently. — Reuven Lax
Reuven - I want to reload my pipeline with a new version of my code without data loss. — bsmarcosj

Reuven Lax Reuven Lax · Accepted Answer · 2015-08-14T05:43:32

Marcos,

The update feature is now live in Maven. It requires that you run a new compatible pipeline with the same value for jobName and the --update flag. Note however that if you change your code too much (this usually involves removing or changing GroupByKey or other combining operations) --update will fail, since the service won't know how to associate the old state with the new pipeline.

Is it possible to stop streaming jobs without lose the current data?

2 Answers