Trying to achieve parallelism in Redshift

Question

I am ingesting some data in the Amazon Redshift table on an hourly basis. The data get available in a single .csv file every hour in the S3 bucket.

PLEASE NOTE: My Redshift cluster is having 4 nodes Slices.

The table I created in the Redshift does have the Distribution Key and Sort Key. The file size is not more than 200MB every hour.

Now my question is, what is the best practice to ingest the data into the Redshift table:

1. Run the Copy command like below -

copy my_schema.my_table from 's3://mybucket/table_data.csv' CREDENTIALS 'aws_access_key_id=<MY_ID>;aws_secret_access_key=<SECRET_KEY>' delimiter '|';

2. Or, do I need any other optional parameter in there to use the advantages of Redshift parallelism. I know the default parallelism it does is EVEN.

Also, if we use any other method to ingest the data will that have any impact on the available storage size? If it improves or remains the same. Or what other benefits can we get if we implement other methods.

Seeking for your expert advice on this.

Thanks

Answer 1

You can't get Redshift to do parallel loading of a single file. The reason is that there is no way to reliably split the file in such a way that all nodes would start reading at the start of a line.

If you want to speed up loads you should split your data into as many pieces as you have slices. For more information about how to optimise COPY see https://docs.aws.amazon.com/redshift/latest/dg/t_Loading-data-from-S3.html

That being said, loading 200 MB data once every hour should be very quick even from a single file. I don't think you will have any problems with that, and you shouldn't spend time optimising that unless it really is a problem.