I have created a BigQuery transfer from AWS S3 to Google BigQuery. It fails with the following error,
No new files found matching "gs://bqdts-amazon_s3-prod-eu-w5jetqct8ohvcjih85apf7gvkbibvbkcj9o6l67/test/files"
But, the data is successfully moved from S3 to Google Cloud
Moving data from Amazon S3 to Google Cloud complete: Moved 10 object(s).
I have created a table in the BigQuery dataset also
Kindly help me resolve this issue
I had the same issue before, and only works for me when I added "*" after the Amazon S3 URI.
Example:
Before (showed the same error as yours): s3://mybucket/path/
After (work success): s3://mybucket/path/*
In General issues section on Amazon S3 transfer page, shows these recomnended action:
Confirm that the Amazon S3 URI in the transfer configuration is correct.
If the transfer configuration was meant to load all files with a common prefix, ensure that the Amazon S3 URI ends with a wildcard. For example, to load all files in s3://my-bucket/my-folder/, the Amazon S3 URI in the transfer configuration must be s3://my-bucket/my-folder/*, not just s3://my-bucket/my-folder/.
I found a similar error Amazon S3 transfers General issues:
Files are transferred from Amazon S3 but not loaded into BigQuery. The transfer logs may look look similar to this:
Moving data from Amazon S3 to Google Cloud complete: Moved object(s). No new files found matching .
Confirm that the Amazon S3 URI in the transfer configuration is correct.
If the transfer configuration was meant to load all files with a common prefix, ensure that the Amazon S3 URI ends with a wildcard. For example, to load all files in s3://my-bucket/my-folder/, the Amazon S3 URI in the transfer configuration must be s3://my-bucket/my-folder/*, not just s3://my-bucket/my-folder/.
Should "gs://bqdts-amazon_s3-prod-eu-w5jetqct8ohvcjih85apf7gvkbibvbkcj9o6l67/test/files" be an Amazon S3 URI : s3://
