Moving raw Google Analytics data from BigQuery to Amazon S3

2
votes

We have integrated Client websites Google Analytics Acc with BigQuery. So GA dump raw data every day into Bigtable. Now my goal is to move that data to Amazon S3.

i can move this raw data from bigtables to google cloud and from there to S3 using GCP transfer job. However GCP cost you for storing the data into cloud.

Now, Google also provide client libraries in diff languages which intern call GCP APIs. Now, i can query BigTable using Java client libraries, however im not sure if they charge you for such queries and if there is any quota limit on these client libraries.

So if anyone has worked on GA & bigQuery altogether, please guide me how i can read the data from Bigtable without costing client any penny.

1
google-bigquerybigtable
Can you clarify your question? – Nico Haase
We have integrated Google Analytics Acc with BigQuery. So GA dump raw data every day into Bigtable. The goal is to move that data to Amazon S3. Google provide, cloud platform from where we can move that data to S3 but GCP cost you to store data into google cloud. – Ganesh
Please add all relevant information to the question itself, not to comments. – Nico Haase
+1 That's actually a very different question (how to move data from BigQuery to S3) compared to what you asked. I'd recommend using an export job to create files in GCS and then copy those files to S3 in this case and not read the rows directly. – Tim Swast

1 Answers

0
votes

One possible solution is to have a job on BigQuery side to generate a compressed file daily and store that at GCS, then have an EC2 instance, install gsutil on the EC2 instance, and trigger gsutil to copy the file from GCS daily.

We are moving compressed JSON files this way daily.

Good luck!