Concurrent Queries, COPY and Connections in AWS Redshift

1
votes

I am trying to understand the difference between concurrent connections and concurrent queries in Redshift. As per documents, We can make 500 concurrent connections to a Redshift cluster but it says maximum 15 queries can be run at the same time in a cluster. Now what is the exact value?

  1. How many queries can be in running state in a cluster at the same time ? If it is 15, does it include RETURNING state queries as well ?

  2. How many concurrent COPY statement can run in a cluster ?

We are evaluating Redshift as our primary reporting data store. If we cannot run a large number of queries simultaneously it may be difficult for us to go with this model.

1
amazon-web-servicesamazon-redshiftamazon-redshift-spectrum
Don't know much about redshift but I'd say that when you connect to a cluster your query will be queued some how, and that's why you can have 500 concurrent connections but only 15 concurrent queries. Check this link - cpr4t3s

1 Answers

2
votes

I think, you have misread somewhere, Max concurrent queries are 50 per WLM. Refer below thread for Amazon support response for more detail.

How many queries can be in running state in a cluster at the same time ? If it is 15, does it include RETURNING state queries as well ?

At a time, Max 50 queries could be running concurrently. Yes it does include INSERT/UPDATE/DELETE etc all.

How many concurrent COPY statement can run in a cluster ?

Ideally, you could Max go up to 50 concurrently, but Copy works bit differently.

Amazon Redshift automatically loads in parallel from multiple data files.

If you use multiple concurrent COPY commands to load one table from multiple files, Amazon Redshift is forced to perform a serialized load, which is much slower and requires a VACUUM at the end if the table has a sort column defined. For more information about using COPY to load data in parallel, see Loading Data from Amazon S3.

Meaning, you could run concurrent Copy commands but make sure one copy command at a time per table.

So practically, it doesn't depend on Nodes on cluster, but Number of tables as well. So if you have only 1 table, you would like to execute 50 insert concurrently, it will result only 1 Copy concurrently.