tbigqueryinput Talend limit error

1
votes

Whenever I am fetching data from Bigquery table with specifying more than 100000 records, its giving me below error

Running Query : SELECT action_payout FROM [FreeBStaging.fb_appreporting] LIMIT 1000000 Exception in component tBigQueryInput_1 com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found { "code" : 404, "errors" : [ { "domain" : "global", "message" : "Not found: Table igneous-study-87009:_165c6d3f4b2c1dbd0d644a1155ebebf7f2dbacfd.anon22fbed90bfc239ef84ee0345744d6cda6052a060", "reason" : "notFound" } ], "message" : "Not found: Table igneous-study-87009:_165c6d3f4b2c1dbd0d644a1155ebebf7f2dbacfd.anon22fbed90bfc239ef84ee0345744d6cda6052a060" }

Even if I don’t specify the limit its giving me error. Its working fine for less than 100000 records.

2
google-bigquerytalend
Please post a job ID, so the BQ team can investigate. – Pentium10
I am using Talend tool. And there is no jobid specified here. – Manoj Bansal
Copy the query and move to BigQuery web ui: bigquery.cloud.google.com/queries – Pentium10
Job id is : igneous-study-87009:job_m_pq4JxKfMPTKcliU9G9s6vhl1Y – Manoj Bansal

2 Answers

1
votes

The query with the job id you provided in the comments fails because its result is too large. BigQuery has AllowLargeResults options, which scales for such queries, but can be slower. I don't know whether Talend supports setting such query option.

0
votes

At the bottom of the Basic Settings tab, you have 3 options: Small, Large, or Auto. I move about 4,000,000 rows daily in a single query with the configuration below (mine is set to Auto... not sure if that is optimal, but I'm afraid to change it since it works)

enter image description here