Tuning mercurial server and identifying bottleneck

Question

I have a very large mercurial repo that takes hours for an initial clone. Clone is being done over https, via scmmanager. I would like to try and get this down to minutes, if possible.

My mercurial repo is running on a server with 24 cores, the load is around 2 while doing a clone from my workstation. I'm wondering how I can tune mercurial on the server to use more cores perhaps. iowait is at 0. Network traffic is low on the server, iftop shows 5Mb/s TX, and I have gigabit ethernet. 4 gigs of ram are being used out of 64 gigs total, and 24 gigs are used for disk cache and 1.5 gigs for buffers.

On my workstation, I have tried renicing hg to -5. Load is around 0.5. My workstation has 8 cores. My workstation has gigabit ethernet also, and seeing minimal traffic also, around 2 Mb/s RX. iowait is also zero on my workstation.

OS for mercurial server is CentOS 6. OS for workstation is Debian jessie. Any ideas would be greatly appreciated.

Edit: tried doing a clone over ssh as well as a local non-hardlinked uncompressed clone. Both take a very long time. (Hours) Repo size is 8 gigs. Unsure how to proceed.

Answer 1

The clone itself is very low complexity operation for Mercurial. Suspect scmmanager. Try cloning direct from the filesytem using a ssh:// URL and load should be just about zilch.

Also try disabling compression on the clone with --uncompressed. If it helps hgweb makes that settable on the server side.

Answer 2

What is the total size of your repositories?

The suggestions Ry4an makes, should indeed resolve your issue. If they don't, you could also create a bundle and transfer that for the initial clone. See hg help bundle for details. Basically: this creates a single file containing revisions (in your case, you'll want to use '--all') to make sure it contains the entire repository). You can then use wget or any other command to download the file (which should be much faster than 5 Mbps).