How to setup AWS SNS and SQS for high availability?

7
votes

Problem with SNS

I'm currently using the python boto3 library for SNS (e.g. to create topics, subscribe to topics, send SNS to topics). When I use either a Resource or a Client, I have to specify a region (e.g. 'us-west-2', 'us-east-1'). I don't see any built-in options for handling region failover. My question is, how do you setup AWS SNS and SQS for high availability/region failover?

import boto3

client = boto3.client('sns', region_name='us-west-2')
response = client.create_topic(Name='my_new_topic')
client.subscribe(TopicArn='arn:aws:sns:us-west-2:123456789:some-topic', Protocol='HTTP', Endpoint='Some-HTTP-Endpoint')

Thoughts on Solution for SNS

I was thinking of checking the response of the client. For example, the below is the response of a successful client.create_topic().

{'ResponseMetadata': {'HTTPStatusCode': 200, 'RequestId': 'adbb58ef-9047-5d44-834d-04a41599eb2b'}, u'TopicArn': 'arn:aws:sns:us-west-2:123456789:some-topic'}

If the request didn't succeed, I can retry say X times before attempting a new region_name, but this method seems really hacky (since it would always attempt to try the region that's down first and need to keep switching to new clients).

Problem with SQS

If the above were to succeed (where we can handle SNS across regions), now I have multiple SQS Queues that I would need to read from (again with what seems to be a hacky solution of looping through clients with different regions).

1
amazon-web-servicesboto3

1 Answers

7
votes

Amazon SNS and SQS are regional services. There is no built-in multi-region availability method for them. If a region fails and you cannot access your topic or queue, you cannot access those same topics/queues by using another region.

There also isn't any standard way to handle them in a multi-region fashion. It really depends on the workings of your application.

One method:

  • Writers - Write to one region, if that fails, write to another.
  • Readers - Read from multiple queues across multiple regions.

Another method:

  • Writers - Write to multiple topics/queues, ensures the message gets somewhere.
  • Readers - Read from multiple queues, treat all equally, you will need to manage message duplication.