I am looking for best way to move 2 TB of data from on-premises to snowflake. Data is in zipped files of size ~150 MB each and similar files will be generated on going basis. As we don't have cloud account (only have snowflake account) so cannot use cloud native storage like S3 or Azure BLOB. Also we want to use public internet to establish connectivity from on-premises network to Snowflake DB on the cloud. (no VPN or direct connect available or 3rd party tool is to be used) How can we best ensure that data while in-transit from on premises to snowflake DB on the cloud is secure. And without using S3 or Azure BLOB storage the data is loaded into snowflake.
0
votes
2 Answers
0
votes
So you do not have external CLoud Storage Accounts to Store these files into ; i could see one option and that is with regard to use SnowSQL to upload files into Snowflake Storage, the internal stage location using PUT command , have look at PUT Command of SQL at following URL https://docs.snowflake.com/en/sql-reference/sql/put.html It can upload the file to Snowflake's internal Stage as well as User & Table internal Stage. There is optional parameter PARALLEL which Specifies the number of threads to use for uploading files, Increasing the number of threads can improve performance when uploading large files. Larger files are automatically split into chunks, staged concurrently, and reassembled in the target stage. A single thread can upload multiple chunks. Uploaded files are automatically encrypted with 128-bit or 256-bit keys. The CLIENT_ENCRYPTION_KEY_SIZE account parameter specifies the size key used to encrypt the files.
Given the 2TBs of File upload , you should experiment with multiple files of small sizes.
0
votes
You can use any of the Snowflake connectors to move data directly from your on-premises servers to Snowflake. https://docs.snowflake.com/en/user-guide/conns-drivers.html
You can also start simply with the command-line interface snowsql using put commands. https://docs.snowflake.com/en/user-guide/snowsql.html
All traffic to/from Snowflake is encrypted in-transit with SSL. https://resources.snowflake.com/snowflake/automatic-encryption-of-data
