Update cloudwatch alarm threshold based on the stoarge size of RDS

0
votes

We have a setup where RDS is provisioned through terraform, cloudwatch alarm for RDS' free storage space metric is set to <10% total RDS storage through terraform. Total RDS storage is a static value. Whenever there's storage shortage we manually increase the RDS storage through AWS console. Is there way to update the cloudwatch alarm threshold automatically, to set it to 10% of new total storage after increasing RDS storage?

2
amazon-web-servicesamazon-rdsamazon-cloudwatchamazon-sns
I see, RDS config change DB event --(publishes)--> SNS topic --(consumed by)--> lambda function --(updates cloudwatch thresholds)--> Done, being a possible solution. Is there a simple way to achieve this that I've missed? – Sharat Naik
Are you manually increasing storage through the AWS console, or through your terraform code? – Foghorn
Manually increasing storage through AWS console – Sharat Naik
Aurora has this metric: AuroraVolumeBytesLeftTotal & VolumeBytesUsed, and I see this topic: https://stackguides.com/questions/47331463/how-to-know-rds-free-storage If you can query the size, I think you can set alert. – Franxi Hidro

2 Answers

1
votes

Although considered an anti-pattern, there is an alternative. It is considered an anti-pattern because it is generally a bad idea to have two different processes controlling the configuration of the same resource. However, if you insist on controlling the size of the RDS outside the terraform plan, you can automate the cloudwatch metric settings.

The root problem is that the FreeSpaceAvailable RDS metric only understands bytes, not percentages. Terraform does that conversion for you so you don't see it. That is why the alarm is not changing when you change the allocated space in the RDS console. You could develop a simple lambda function that uses one the of the AWS APIs (like boto3 for python) to periodically query the size the RDS database and update the alarm on FreeSpaceAvailable accordingly. It could be even more sophisticated by having SNS notify a SQS queue about changes in RDS configuration which could then trigger the lambda to evaluate the metric, which would make the update very fast and efficient. You would need to apply a IAM policy which allows the lambda to read data on the RDS instance in question and permission to update the metric in question. Alternatively, you could have it be manually run and then run it right after the AWS console update, but in that case you could probably just as easily manually update the metric yourself.

I still wouldn't recommend this be the path due to the control of an already controlled resource by terraform.

0
votes

This should be automatically updated by terraform if you use terraform code to update the allocated space. Cloudwatch itself doesn't understand the relationship between what you are watching - it is just programmed with a threshold value by terraform when you apply the template. Since terraform is doing both actions on your behalf (allocating the RDS instance and setting up cloudwatch metrics) it understands what needs to be set. However, using the AWS console is essentially side-stepping the terraform. While the size increases, terraform is unaware of the change to also change the cloudwatch metric. Using terraform to apply the size update should fix the alarm, depending on how you setup your alarm threshold.