I am running a python script in Cloud Run on a daily basis with Cloud Scheduler to pull data from BigQuery and upload it to Google Cloud Storage as a CSV file. The Cloud Scheduler setup utilizes an HTTP "Target" with a GET "HTTP method". Also, Cloud Scheduler authenticates the https endpoint using a service account with the "Add OIDC token" option.
When running Cloud Scheduler and Cloud Run with a very small subset of the BigQuery data for a job that takes a few seconds, the "Result" in Cloud Scheduler always shows "Success" and the job completes as intended. However, when running Cloud Scheduler and Cloud Run with the full BigQuery dataset for a job that takes a few minutes, the "Result" in Cloud Scheduler always shows "Failed", even though the CSV file is typically (although not always) uploaded into Google Cloud Storage as intended.
(1) When running Cloud Scheduler and Cloud Run on the full BigQuery dataset, why does the "Result" in Cloud Scheduler always show "Failed", even though the job is typically finishing as intended?
(2) How can I fix Cloud Scheduler and Cloud Run to ensure the job always completes as intended and the "Result" in Cloud Scheduler always shows "Success"?