aws sns publish does not reach endpoint

Question

I am working on an app that uses SNS and SQS to publish topics between entities with boto3. The setup is pretty vanilla where one process has subscribed to the topic and is waiting for activity. The second process is just publishing a message to the topic and I'd expect to see that published message arrive to the listening job but nothing ever does. The SQS dashboard shows that are no messages pending or in flight. There are full permissions on both the queue and the topic. I am new to all of this so it is probably something simple.

listening job:

# Create topic
sns_game_topic = sns_client.create_topic(Name='My_Topic')

# Create queue
queue = sqs_client.create_queue(QueueName='My_Queue.fifo', Attributes={'FifoQueue':'true', 'ContentBasedDeduplication': 'true'})
sqs_arn = sqs_client.get_queue_attributes(QueueUrl=queue['QueueUrl'], AttributeNames=['QueueArn'])['Attributes']['QueueArn']

sns_arn = sns_topic['TopicArn']

sns_client.subscribe(TopicArn=sns_arn, Protocol='sqs', Endpoint=sqs_arn)

sending job:

queue = self.sqs_client.create_queue(QueueName='Battleship_Registration.fifo', Attributes={'FifoQueue':'true', 'ContentBasedDeduplication': 'true'})

sqs_arn = self.sqs_client.get_queue_attributes(QueueUrl=queue['QueueUrl'], AttributeNames=['QueueArn'])['Attributes']['QueueArn']

# This topic should already exist but creation is idempotent and we need the arn to subscribe to it
topic = self.sns_client.create_topic(Name='My_Topic') 
sns_arn = topic['TopicArn']

# Connect topic to queue
self.sns_client.subscribe(TopicArn=sns_arn, Protocol='sqs', Endpoint=sqs_arn)
pub = self.sns_client.publish(TopicArn=<SNS ARN goes here>, Message='Hello World')

There are no filter policies active as I want all messages to pass through.

I have 2 questions: Why is the simple publish not getting through?

In order to publish or subscribe I need to know arn values for sns and sqs. The best way I know to obtain them is to call create_(topic/queue) even if the entity already exists. The calls are idempotent so it should be safe but is this the best practice to obtain arn values?

I see you are sending message to a new topic and listening to a existing topic. you should sending and listening on the same topic..see here on how you can construct ARN docs.aws.amazon.com/general/latest/gr/… — Sudharsan Sivasankaran
or you can use aws cli to get the arn “aws sns list-topics” — Sudharsan Sivasankaran
My thinking was that create_topic would not actually create a new topic since it is supposed to be idempotent. I figured it would return the arn of the topic that already exists. I will try to construct the arn instead and see if that works. I had a method that listed all known topics which I iterated through and used regex to find the right topic but I thought there must be an easier way to go about it. — Elsporko

Elsporko Elsporko · Accepted Answer · 2018-02-23T23:19:30

Found the problem. SNS does not work with FIFO queues. The act of subscription did not work so there was never a connection between the topic and the queue.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-subscribe-queue-sns-topic.html

Note

Amazon SNS isn't currently compatible with FIFO queues.

aws sns publish does not reach endpoint

1 Answers