Handle I/O requests in amazon ec2 instances

Question

After having learnt node, javascript and all the rest the hard way, I am finally about to release my first web app. So I subscribed to Amazon Web Services and created a micro instance, planning on the first year free tier to allow me to make the app available to the world.

My concern is more about hidden costs. I know that with the free tier comes 1 million I/O requests per month for the Amazon EC2 EBS.

Thing is, I started testing my app one the ec2 instance to check that everything was running fine; and I am already at more than 100, 000 I/O requests. And I have basically been the only one using it so far (37 hours that the instance runs).

So I am quite afraid of what could happen if my app gets some traffic, and I don't want to end up with a huge unexpected bill at the end of the month.

I find it quite surprising, because I mainly serve static stuff, and my server side code consists in :

• Receving a search request from a client
• 1 http request to a website
• 1 https request to the youtube api
• saving the data to a mongoDB
• Sending the results to the client

Do you have any advice on how to dramatically reduce my IO? I do not use any other Amazon services so far, maybe am I missing something?

Or maybe Amazon free tier in not enough in my case, but then what can it be enough for? I mean, my app is really simple after all.

I'd be really glad for any help you could provide me

Thanks!

Answer 1

You did not mention total number of visits to your app. So I am assuming you have fairly less visits.

What are I/O requests ? A single I/O request is a read/write instruction that reaches the EBS volumes. Beware! Execution of large read/writes is broken into multiple smaller pieces, which is the block size of the volume.

Possible reasons of high I/O:

1. Your app uses lot of RAM. After you hit the limit, the OS starts swapping memory to and fro from swap area in your disk, constantly.
2. This is most likely the problem, the mongoDB search. mongoDB searches can be long complex queries internally. From one of the answers to this question, the person was using mySQL and it caused him 1 billion I/O requests in 24 days. So 1 database search can be many I/O requests.
3. Cache is disabled, or you write/modify lot of files. You mentioned you were testing. Free-teir is just not suitable for developing stuff.

You should read this, in case you want to know what happens after free-tier expires.

Answer 2

I recently ran into a similar situation recording very high I/O request rates for a website with little to no traffic. The culprit seems to be a variation of what @prajwalkman discovered testing Chef deployments on a micro instance.

I am not using Chef, but I have been using boto3, Docker and Git to automatically 'build' test images inside a micro instance. Each time I go through my test script, a new image is built and I was not careful to read the fine print regarding default settings on the VolumeType argument on the boto3 run_instance command. Each test image was being built with the 'standard' volume type which, according to current EBS pricing, bills at the rate of $0.05/million I/Os. Whereas, the 'gp2' general purpose memory has a flat cost of $0.10 per GB per month with no extra charge for I/O.

With a handful of lean docker containers taking up a total of 2GB, on top of the 1.3GB for the amazon-ecs-optimized-ami, my storage is well within the free-tier usage. So, once I fixed the volumetype attribute in the blockdevicemappings settings in my scripts to 'gp2', I no longer had an I/O problem on my server.

Prior to that point, the constant downloading of docker images and git repos produced nearly 10m I/Os in less than a week.

Answer 3

The micro instance and the free tier is meant for testing their offerings, not a free way for you to host your site/web application.

You may have to pay money at the end of the month, but I really doubt if you can get away with paying less by using some other company for hosting. AFAIK AWS really is the rock bottom of the price charts.

As for the IO requests themselves, it's hard to give generic advice. I once was in a situation where my micro instance racked up ridiculous number of IO requests. Turns out testing Chef deployments on EC2 is a bad idea.

Answer 4

I/O Requests have to do with reading and writing blocks to EBS volumes. You can reduce this by using as much in memory caching as possible. Micro instances only have about 613 MB of memory available, so you may not be able to do much here.

Answer 5

Ok, so it seems like I/O requests are related to the EBS volume, and that caching may reduce it.

Something I had not considered though is all the operations I made to get my app running. I updated the linux image, installed node and npm, several modules, mongodb, ....

This is likely to be the main cause of the I/O. The number requests hasn't grown much in the last days, where the server stayed mostly idle.