Apache Beam/Google Dataflow - Exporting Data from Google Datastore to File in Cloud Storage

0
votes

I need create a file report for user request. Each user select the filter for file report, and my application should generate a file in cloud storage and send a notification with the file link generated.

This is the application workflow:

  1. the client selects a filter and request a report file
  2. The application get this request and create a record in datastore with data about user selected filter
  3. Stores the Datastore key URL Safe String from the new record in pubsub.
  4. The Dataflow Pipeline read the key stored in PubSub.
  5. Generate file in google cloud storage
  6. Notifies the client with storage file url

It is possible to create a file for each pubsub entrance ?

How I do to create a file with custom name?

It is correct this architecture ?

1
google-app-enginegoogle-cloud-datastoregoogle-cloud-storagegoogle-cloud-dataflowgoogle-cloud-pubsub

1 Answers

0
votes

Your use case sounds as if it would be more applicable to google cloud storage than cloud datastore. Google cloud storage is meant for opaque file-like blobs of data, and provides a method to receive pubsub notifications on file updates https://cloud.google.com/storage/docs/pubsub-notifications.

However, its a bit unclear why you're using the indirection of pubsub and datastore in this case. Could the server handling the client request instead directly make a call to the google cloud storage api?