AWS: empty SQS queue when subscribed to SNS via boto3

Question

I am not receiving any messages in my SQS queue when subscribing to an SNS topic via boto3.

Is this an issue with the code or the API credentials I am using? The IAM policy associated with this account has AWS PowerUser privileges, which should mean it has unrestricted access to manage SNS topics and SQS queues.

When I create the equivalent structure through the AWS console (create topic, create queue, subscribe queue to topic) and send a message using either boto3, the AWS CLI, or the AWS console, the message comes through correctly.

I don't think it is an issue with the code because the SubscriptionArn is being returned correctly?

I have tried this with both the US-EAST-1 and AP-SE-1 regions, same result.

Sample code:

#!/usr/bin/env python3

import boto3
import json

def get_sqs_msgs_from_sns():
    sqs_client = boto3.client('sqs', region_name='us-east-1')
    sqs_obj = boto3.resource('sqs', region_name='us-east-1')
    sns_client = boto3.client('sns', region_name='us-east-1')
    sqs_queue_name = 'queue1'
    topic_name = 'topic1'

    # Create/Get Queue
    sqs_client.create_queue(QueueName=sqs_queue_name)
    sqs_queue = sqs_obj.get_queue_by_name(QueueName=sqs_queue_name)
    queue_url = sqs_client.get_queue_url(QueueName=sqs_queue_name)['QueueUrl']
    sqs_queue_attrs = sqs_client.get_queue_attributes(QueueUrl=queue_url,
                                                    AttributeNames=['All'])['Attributes']
    sqs_queue_arn = sqs_queue_attrs['QueueArn']
    if ':sqs.' in sqs_queue_arn:
        sqs_queue_arn = sqs_queue_arn.replace(':sqs.', ':')

    # Create SNS Topic
    topic_res = sns_client.create_topic(Name=topic_name)
    sns_topic_arn = topic_res['TopicArn']

    # Subscribe SQS queue to SNS
    sns_client.subscribe(
            TopicArn=sns_topic_arn,
            Protocol='sqs',
            Endpoint=sqs_queue_arn
    )

    # Publish SNS Messages
    test_msg = {'default': {"x":"foo","y":"bar"}}
    test_msg_body = json.dumps(test_msg)
    sns_client.publish(
        TopicArn=sns_topic_arn, 
        Message=json.dumps({'default': test_msg_body}),
        MessageStructure='json')

    # Validate Message
    sqs_msgs = sqs_queue.receive_messages(
            AttributeNames=['All'],
            MessageAttributeNames=['All'],
            VisibilityTimeout=15,
            WaitTimeSeconds=20,
            MaxNumberOfMessages=5
    )
    assert len(sqs_msgs) == 1
    assert sqs_msgs[0].body == test_msg_body
    print(sqs_msgs[0].body) # This should output dict with keys Message, Type, Timestamp, etc., but only returns the test_msg

if __name__ == "__main__":
    get_mock_sqs_msgs_from_sns()

I receive this output:

$ python .\sns-test.py
Traceback (most recent call last):
  File ".\sns-test.py", line 55, in <module>
    get_sqs_msgs_from_sns()
  File ".\sns-test.py", line 50, in get_sqs_msgs_from_sns
    assert len(sqs_msgs) == 1
AssertionError

I also tried the equivalent using the AWS CLI - sns create-topic, sqs create-queue, sqs get-queue-attributes to get the queue arn, sns subscribe, sns publish, sqs receive-message; unfortunately, there is no message in the queue. — angled
I am also wondering if this is the guide to the answer: stackoverflow.com/questions/19329057/… . It doesn't seem that elegant. — angled

angled angled · Accepted Answer · 2018-02-12T14:03:28

The URL above for the similar question posed for the C# AWS SDK put me in the correct direction for this: I needed to attach a policy to the SQS queue to allow the SNS topic to write to it.

def allow_sns_to_write_to_sqs(topicarn, queuearn):
    policy_document = """{{
  "Version":"2012-10-17",
  "Statement":[
    {{
      "Sid":"MyPolicy",
      "Effect":"Allow",
      "Principal" : {{"AWS" : "*"}},
      "Action":"SQS:SendMessage",
      "Resource": "{}",
      "Condition":{{
        "ArnEquals":{{
          "aws:SourceArn": "{}"
        }}
      }}
    }}
  ]
}}""".format(queuearn, topicarn)

    return policy_document

and

policy_json = allow_sns_to_write_to_sqs(topic_arn, queue_arn)

response = sqs_client.set_queue_attributes(
    QueueUrl = queue_url,
    Attributes = {
        'Policy' : policy_json
    }
)
print(response)

AWS: empty SQS queue when subscribed to SNS via boto3

1 Answers