GCP dataflow SQL export using service account

0
votes

I have a service account access to a BQ view and I want to query it and export the data to GCS or a BQ dataset in another project.

Dataflow (+create job from SQL) looks perfect for the use case but I can't work out how to set it up so the service account is used for the query. It appears Dataflow accepts using a service account but how is this done?

Thanks in advance,

1
google-cloud-platformgoogle-cloud-dataflow
Why do you want to create dataflow sql? A simple query is enough for the 2 use cases. Why this choice? – guillaume blaquiere
what simple query can i do to select * from the view and export into json? feels the only option is write a script to do this? that is why i was looking at dataflow – rix

1 Answers

1
votes

If you want to export data to another BigQuery table, you can simply use an INSERT INTO <table> SELECT * FROM ... (also known as INSERT-SELECT). If you need to export this regularly, you can use scheduled queries

if you want to export to Google Cloud Storage, you have 2 solutions

  • Export a full table, and use BigQuery export API (you can call this API with Cloud Scheduler if you want to regularly export them)
  • Use export query statement.