Is it possible to set the error limit while loading the data into a snowflake table. I am using COPY INTO option. I know there are options like RETURN_FAILED_ONLY and VALIDATION_MODE, but this does not support if the error limit is reached then fail COPY INTO else continue loading it by ignoring the failed records.
0
votes
I don't think we have option for that, as work around execute the copy command at end copy command check the error count if count is acceptable then commit the transaction.
– Sriga
1 Answers
0
votes
I believe what you are looking for is the SKIP_FILE_num or SKIP_FILE_num%. This will skip the file when a certain number of records or certain % of records fail. When a file is skipped it will be listed with a status of FAILED.
https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html
Snowflake does not currently have the equivalent that you are looking for across all files of a load. Depending on how you are scripting/executing your COPY INTO commands, however, you could wrap the COPY INTO command with a transaction, check the output of the COPY INTO, determine whether it is inside or outside of your threshold and then either commit or rollback the transaction. This would accomplish what you are looking for, but takes a bit of custom coding to accomplish.
