Speeding up RSpec tests in a large Rails application

Question

I have a Rails application with over 2,000 examples in my RSpec tests. Needless to say, it's a large application and there's a lot to be tested. Running these tests at this point is very inefficient and because it takes so long, we're almost at the point of being discouraged from writing them before pushing a new build. I added --profile to my spec.opts to find the longest running examples and there are at least 10 of them that take an average of 10 seconds to run. Is that normal amongst you RSpec experts? Is 10 seconds entirely too long for one example? I realize that with 2,000 examples, it will take a non-trivial amount of time to test everything thoroughly - but at this point 4 hours is a bit ludicrous.

What kind of times are you seeing for your longest running examples? What can I do to troubleshoot my existing specs in order to figure out bottlenecks and help speed things up. Every minute would really help at this point.

Answer 1

10 seconds is a very long time for any single test to run. My gut feeling is that your spec target is running both unit and integration tests at the same time. This is a typical thing that projects fall into and at some stage, you will need to overcome this technical debt if you want to produce more, faster. There are a number of strategies which can help you to do this... and I'll recommend a few that I have used in the past.

1. Separate Unit From Integration Tests

The first thing I would do is to separate unit from integration tests. You can do this either by:

1. Moving them (into separate folders under the spec directory) - and modifying the rake targets
2. Tagging them (rspec allows you to tag your tests)

The philosophy goes, that you want your regular builds to be quick - otherwise people won't be too happy to run them often. So get back to that territory. Get your regular tests to run quick, and use a continuous integration server to run the more complete build.

An integration test is a test that involves external dependencies (e.g. Database, WebService, Queue, and some would argue FileSystem). A unit test just tests the specific item of code that you want checked. It should run fast (9000 in 45 secs is possible), i.e. most of it should run in memory.

2. Convert Integration Tests To Unit Tests

If the bulk of your unit tests is smaller than your integration test suite, you have a problem. What this means is that inconsistencies will begin to appear more easily. So from here, start creating more unit tests to replace integration tests. Things you can do to help in this process are:

1. Use a mocking framework instead of the real resource. Rspec has an inbuilt mocking framework.
2. Run rcov on your unit test suite. Use that to gauge how thorough your unit test suite is.

Once you have a proper unit test(s) to replace an integration test - remove the integration test. Duplicate testing only makes maintenance worse.

3. Don't Use Fixtures

Fixtures are evil. Use a factory instead (machinist or factorybot). These systems can build more adaptable graphs of data, and more importantly, they can build in-memory objects which you can use, rather than load things from an external data source.

4. Add Checks To Stop Unit Tests Becoming Integration Tests

Now that you have faster testing in place, time to put in checks to STOP this from occurring again.

There are libraries which monkey patch active record to throw an error when trying to access the database (UnitRecord).

You could also try pairing and TDD which can help force your team to write faster tests because:

1. Somebody's checking - so nobody gets lazy
2. Proper TDD requires fast feedback. Slow tests just make the whole thing painful.

5. Use Other Libraries To Overcome The Problem

Somebody mentioned spork (speeds up load times for the test suite under rails3), hydra/parallel_tests - to run unit tests in parallel (across multiple cores).

This should probably be used LAST. Your real problem is all the way in step 1, 2, 3. Solve that and you will be in a better position to role out additional infrastructure.

Answer 2

For a great cookbook on improving the performance of your test suite, check out the Grease Your Suite presentation.

He documents a 45x speedup in test suite run time by utilizing techniques such as:

• fast_context
• quickerclip
• hydra
• scary file system tweaks
• tcmalloc
• Ruby Enterprise Edition GC tuning

Answer 3

You can use Spork. It has support for 2.3.x ,

https://github.com/sporkrb/spork

or ./script/spec_server which may work for 2.x

Also you can edit the database configuration ( which essentially speeds up the database queries etc ), which will also increase performance for tests.

Answer 4

10 seconds per example seems like a very long time. I've never seen a spec that took more than one second, and most take far less. Are you testing network connections? Database writes? Filesystem writes?

Use mocks and stubs as much as possible - they are much faster than writing code that hits the database. Unfortunately mocking and stubbing also take more time to write (and are harder to do correctly). You have to balance the time spent writing tests vs. the time spent running tests.

I second Andrew Grimm's comment about looking into a CI system which might allow you to parallelize your test suite. For something that size, it might be the only viable solution.

Answer 5

If you are using ActiveRecord models, you should also consider the cost of BCrypt encryption.

You can read more about it on this blog post: http://blog.syncopelabs.co.uk/2012/12/speed-up-rspec-test.html

Answer 6

faster_require gem might help you. Besides that your only way is to (like you did) profile and optimize, or use spork or something that runs your specs in parallel for you. http://ruby-toolbox.com/categories/distributed_testing.html

Answer 7

Delete the existing test suite. Will be incredibly effective.