When should we use SNOWPIPE?

0
votes

We have some files between 500KB - 20 MB size in Sharepoint portal. We would like to convert those files to CSV and then stage them to Snowflake. There is no real need for real time ingestion. I am thinking of two options. Which option will be better?

  1. Load the file(CSV) into the cloud providers object storage. Create an external stage. Then have a python program scheduled every hour to ingest the data from stage to snowflake table
  2. Use SNOWPIPE

I am more inclined to #1 primarily because I will have a control on the warehouse. Also, it will allow me to bunch up the files and then load to snowflake.

1
snowflake-cloud-data-platform

1 Answers

0
votes

If you don't need to load your source data in real time option 1 makes more sense, but you need to manage and maintain it. Option 2 is set up once and it will load the files automatically, but will be more costly because you don't have control over warehouse usage.

I have a similar situation and using option 1 like load.