Hardware for a sharded + redundant MongoDB-environment [closed]

7
votes

I'm trying to plan a new database-environment from scratch and I'm wondering how many servers are needed and how much performance they should provide.

Since I want it to be fast, I'm considering using SSD-memory and loads of RAM. However, flash-memory is expensive and makes up for the biggest part of the cost of a server. Thus, the entire system should be set up for horizontal scaling from the start, so I can add more nodes when I need more storage/performance.

To get started I'm thinking of using 2 shards, each consisting of a master and a replica-slave for redundancy. MongoDB-documentation suggests using 1 master and 2 slaves, but I'm afraid that won't be in the available budget, since each of these servers will be equipped with about 200 GB of RAM and 6x400 GB SSD as Raid 10.

When using shards, it is also suggested using 3 config-servers for failsafe/high-availability. Same as above, I'm thinking 1 master and 1 slave as a start.

  • What sort of hardware would you recommend to put the config-servers on? Should they be somewhat equally performant as the shard-nodes in terms of cpu/memory/harddisk? Or can I put them on virtualization or on cheaper hardware?
  • Does the setup I described even make sense? How about the ratio of RAM vs. harddisk on the shard-nodes? At the moment it would probably be easier and cheaper to just put twice the number of discs into 1 shard (1 master, 1 slave) and skip the sharding until I really need it. However (as mentioned above) - the system should be ready for sharding from the beginning, because storage-needs can change overnight. Or is it possible to set it all up, but have it running on 1 shard only for now?
  • Since I'm only planning to use 2 instead of 3 servers for high-availability/failsafe I probably need arbiters as well. Do they need dedicated hardware as well? Or can I use one arbiter in a virtual-mashine that serves config-servers and shard-nodes? Or is using 3 seperate servers for redundancy an absolute must in your opinion?
2
performancemongodbhardware
SSD-memory? I am guessing you mean some kind of hybrid where swap is used on SSD as well. Don't, your working set should fit into RAM, SSD helps when you need to hit disk. Flash-memory should actually be quite cheap with most providers, SSD should be mega expensive. Why do yyou need such a huge server?? Also all members don't need to be as powerful as the machines which may receive operations. The config servers can sit on bare bones commodity hardware, they are only storing settings for your cluster. Arbiters do not need dedicated hardware - Sammaye
An arbiter is nothing more than an empty mognod pinging a vote, I would question why you need such a huge server otherwise, I mean are you planning on having 200GB working set on this budget?? Also you might find a few smaller servers better - Sammaye
Also config servers are not served by arbiters, they are connected to two completely different things, one being replication and the other being sharding. In fact the more I read you seem to confuse sharding and replication completely - Sammaye

2 Answers

5
votes

Rock on. Sounds like an awesome setup. Given your configuration choices, I could not imagine a budget that would limit your choices too much.

  • You will not need a dedicated physical server for config servers. These run fairly lightly. You will need low latency between your mongos and your config servers. You will always want to make sure the hosts are reliable, and prepared for disaster. Ensure you double check the backup procedures for a sharded environment. The backups require coordination among the moving pieces of a sharded cluster. If possible, run the config servers on virtual servers in the same datacenter.

  • Yes, the hardware you described makes sense if you are going to run multiple shards on a single machine. A single MongoDB on that powerful machine will leave the machine mostly idle. A single mongod process cannot use that much RAM, I/O, or CPU. You will want to "core shard" the host. At MongoHQ, we do this by running each mongod in a container, which we can isolate from the other instances on the same machine. With your specifications, you can run as many as 10 shards on a single host, or more if you want to stretch the hosts.

  • You can start this with a single shard and migrate to a sharded cluster later. This is our recommended approach to sharding: don't shard until you must. By delaying sharding, you increase your flexibility to make changes to your system. When sharding is in place, you have committed to a particular path, with no flexibility (which is okay when you know the future). By delaying sharding, you have no tradeoffs.

  • The arbiters do not need dedicated hardware. You can run those on virtual machines. These do not require the same level of backup requirements, but should have good uptimes.

  • Using 3 servers is not a requirement for solid uptime. However, when one data host goes down for a few hours, you are relegated to a single data host. While relegated to the single data host, the single host will work fine. Because you only have one second of data running, you do run a higher risk of having an outage. That being said, 2 nodes plus an arbiter is okay for most use cases, and will remain up if one of the data nodes fails.

Hope this helps! We run similar configurations at MongoHQ, and are very happy with the level of performance we get from the hosts.

2
votes

I am going to put some thoughts down here.

This answer is pretty much useless without knowing your working set however, maybe it might create some pointers.

What sort of hardware would you recommend to put the config-servers on?

Config servers, despite better being on their own hardware (server, not virtual machine) can run easily on the most base of commodity hardware, you don't need anything fancy for that. All they do is store your sets configuration, and even then they are not used all the time, the mongoss will cache the clusters configuration for intervals.

Or can I put them on virtualization or on cheaper hardware?

I wouldn'ty put them on virtualisation because that normally denotes that they are either physicall on the same server or close. You need to put them on real redundant servers however, yes you can get cheap hardware for them.

Making sure you haver a decent network between your shards and your config servers should be natural knowledge.

Does the setup I described even make sense?

No idea without an understanding of your working set, however, by the sounds of it you think MongoDB MUST ALL fit in memory. This is untrue, only the working set does ( http://docs.mongodb.org/manual/faq/storage/#what-is-the-working-set ) which can be an absolute fraction of your data in a specific time frame (normally 10 minutes) if you play your cards right.

How about the ratio of RAM vs. harddisk on the shard-nodes?

A little much, MongoDB can use that server but I bet it will sit idle most of the time, I bet you haven't really calculated your working set right.

At the moment it would probably be easier and cheaper to just put twice the number of discs into 1 shard (1 master, 1 slave) and skip the sharding until I really need it.

Yes. That is a safe bet if you need those disks. I am going to go as far as to say you should really look into whether you do or not.

the system should be ready for sharding from the beginning

As said in the answer above, you can create a 1 member sharded set and just scale up from there.

Since I'm only planning to use 2 instead of 3 servers for high-availability/failsafe I probably need arbiters as well.

Yes, using 3 servers is normally quite solid for automatic failover, which is the important thing here. If the majority of the servers failover (50% or more) then you will have to manually fix your replica set.

This means on a two server environment per shard you will infact have no automated failover and so MongoDB will infact not be fullfilling one of it core advantages: automatic failover (this is also a core purpose of repilica sets...).

Not only this but your users must wait until you get round to fixing that set, I am sure this, in itself, holds problems.

Do they need dedicated hardware as well?

No, as said in my comment (and in the documentation), arbiters do not need to run on dedicated hardware. However, this being said it can be useful, especially if you wish to create failover redundancy by splitting off the arbiters to another data center.

The arbiter can run on the bare bones of a server so if you decide to split it out don't bother with 200-odd GB of RAM and 6x400GB SSDs. Just get a...well a mobile phone could technically run an arbiter (if MongoDB supported Android and iOS).

Or is using 3 seperate servers for redundancy an absolute must in your opinion?

It is not, but anything less, as said above, will cause problems in the event of a failover.

A good option is to actually put your arbiter onto your application server(s), maybe even having one per application server.