BigQuery Data Transfer does not delete sources

1
votes

I am using an on-demand (as a test before automation) Bigquery Data Transfer job which loads data from Storage to a table. All is working fine however I put "Delete source files after transfer | true" and at the end no file is deleted. They are not loaded again but are always here in my Storage folder.

This deletion is vital since the amount of data could become quite big in a short period of time. I could delete them with another program but then the Transfer Service would become less interesting.

The job itself does not throw any error, which means that something is silently failing. Do you know what could possibly cause this ? Or maybe I am missing the meaning of this option ?

Thanks

1
google-bigquery
Have you checked that you are having "storage.objects.delete" permission? - SANN3
maybe you are right but then it would at least throw an error or an exception. I may have another lead : I can create on-demand jobs but I must be admin to create a scheduled transfer job. I will wait for this and then figure out if something is missing :) - Joseph Yourine

1 Answers

1
votes

Make sure you have enough permissions to do Cloud storage transfer, it won't tell explicitly what are the permissions are missing.

Required permissions

  1. BigQuery
    • bigquery.transfers.update
  2. Cloud Storage
    • storage.objects.get
    • storage.objects.list
    • storage.objects.delete

More info refer here