SQS maxNumberOfMessages

Question

Using a Java client application I am querying an SQS queue for messages. The queue has 12,000 messages as setup for testing. I am using openJDK with aws-java-sdk latest (software.amazon.awssdk 2.10.62) pom.xml is shown further down.

The issue I am seeing is that despite setting the maxNumberOfMessages(10) I only ever get 3. I understand that it is a maximum not a guarantee of the number of messages however there is no waver in the number of messages returned. It is always 3.

AWS Documentation: MaxNumberOfMessages The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned). Valid values: 1 to 10. Default: 1. Type: Integer Required: No

Consuming Messages Using Short Polling

When you consume messages from a queue using short polling, Amazon SQS samples a subset of its servers (based on a weighted random distribution) and returns messages from only those servers. Thus, a particular ReceiveMessage request might not return all of your messages. However, if you have fewer than 1,000 messages in your queue, a subsequent request will return your messages. If you keep consuming from your queues, Amazon SQS samples all of its servers, and you receive all of your messages.

So we have tested two clients in java using both the older aws sdk and the newer one with the same results. Always only 3 messages back.

Interestingly if instead of running the application externally (on my mighty desktop) you run it as an AWS Lambda you get 10 messages. This lambda test was done using JavaScript by a colleague.

So the question remains why do we only ever get 3 messages per request and seemingly within lambda you can get 10.

Given there is a cost per request is the weighted random distribution based on amazon profit =))

SQS Test Method:

public void SQStart()
{
    AwsBasicCredentials awsCreds = AwsBasicCredentials.create("accessKeyID", "secretKeyID");
    AwsCredentialsProvider creds = StaticCredentialsProvider.create(awsCreds);
    SqsClient sqs = SqsClient.builder().credentialsProvider(creds).region(Region.EU_WEST_1).build();
    GetQueueUrlRequest getQueueRequest = GetQueueUrlRequest.builder()
            .queueName(QUEUE_NAME)
            .build();
    String queueUrl = sqs.getQueueUrl(getQueueRequest).queueUrl();

    for (int x =1; x < 100; x++) {
        ReceiveMessageRequest receiveMessageRequest = ReceiveMessageRequest.builder()
                .queueUrl(queueUrl)
                .maxNumberOfMessages(10)
                .build();


        List<Message> messages = sqs.receiveMessage(receiveMessageRequest).messages();
        if (messages.size() > 3 ) {
            System.out.println("YEY More than 3 Messages: "+ messages.size());
        }
    }
}

POM.XML:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>groupId</groupId>
    <artifactId>SQSTest</artifactId>
    <version>1.0-SNAPSHOT</version>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>


    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.10.62</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>

            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>sqs</artifactId>
        </dependency>

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.9</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.4.10</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.2</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.11.720</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.6.1</version>
        </dependency>
    </dependencies>
</project>

Just a thought, but have you checked the configuration of the queue itself on AWS? Maybe it was setup with a maxNumberOfMessages property which might get priority over the one you set in the java client? Since it works in the javascript lambda, this would be weird, but still worth a shot :) — niekname

Jacob G. Jacob G. · Accepted Answer · 2020-02-18T19:44:10

Given there is a cost per request is the weighted random distribution based on amazon profit =))

It's clear that your goal here is to reduce costs, whether that be by sending fewer requests to SQS, or by forcing SQS to deliver the maximum amount of messages available.

As you've stated in your question, SQS has no obligation to deliver the maximum amount of messages available. However, there is something I'd like to inform you about, assuming you're not already aware of it.

Long Polling

The Developer Guide of Amazon's Simple Queue Service states:

The process of consuming messages from a queue depends on whether you use short or long polling. By default, Amazon SQS uses short polling, querying only a subset of its servers (based on a weighted random distribution) to determine whether any messages are available for a response. You can use long polling to reduce your costs while allowing your consumers to receive messages as soon as they arrive in the queue.

The messages that you sent to SQS may have all been stored on separate servers. As the documentation states, only a subset of servers may be queried if your queue is set to use short polling. My guess is that you were unlucky when invoking receiveMessage and only 3 were returned every time.

If we look at the benefits of long polling on the same documentation page, it states:

Long polling offers the following benefits:

Eliminate empty responses by allowing Amazon SQS to wait until a message is available in a queue before sending a response. Unless the connection times out, the response to the ReceiveMessage request contains at least one of the available messages, up to the maximum number of messages specified in the ReceiveMessage action.

Eliminate false empty responses by querying all—rather than a subset of—Amazon SQS servers.

The second bullet is very important here. Even though you're not seeing empty responses, there may exist more messages stored on servers that aren't being queried. If you enable long polling, you hopefully should see an increase in the amount of messages returned, assuming there's more than 3 servers in total.

Therefore, my suggestion is to enable long polling on your queue. To do this, see the Setting Up Long Polling page.

As DevilCode mentioned in his comment below, he was able to resolve his issue by using a FIFO queue instead of a standard queue, and by enabling long polling on it.

SQS maxNumberOfMessages

3 Answers

Long Polling