autotest and rspec giving different answers

Question

I have an application that I'm testing with rspec2 (2.6.4), capybara and selenium.

When I run any request spec directly with rspec, the test passes.

When I run the request specs with autotest they all fail with a message saying 'Cannot find a User with ID= **.

When autotest reloads the first request test automatically, it passes

Autotest then reloads all tests and when it reaches the request tests they all fail again

Users are being created by factory-girl and logged in using devise as follows:

before(:each) do
  @user = Factory(:user)   
  login_as @user
end 

after(:each){
    logout
  }

and in the spec helper I have

def login(user)
  post login_path, :login => user.login, :password => 'testing'
end

These are the relevant gems i have installed (group :test)

  gem "rspec"
  gem "rspec-rails"
  gem  "autotest-rails"
  gem "selenium-webdriver", ">= 0.2.2"
  gem 'capybara', :git => 'git://github.com/jnicklas/capybara.git'
  gem 'launchy'
  gem 'database_cleaner'

This has been baffling me for a while now - any thoughts anyone? Even any thoughts on how to start looking at the stack trace to see where the two calls are differing?

hmmm - that's a thought - not sure there is anything in the spec helper that would be required for the other tests - I'll stick a debug in and see ... thanks — chrispanda
well, thanks for the suggestion, but sadly that's not it ... but what is becoming clear from the debugging I've been doing is that the problem is not with the logged in user, but with users being created in the tests, so I'm starting to look at transactions and something going on with cleaner ... — chrispanda

chrispanda chrispanda · Accepted Answer · 2011-07-13T08:34:20

Solved, sort of. What is really going on here remains a mystery, but here is an accurate summary of what the problem really is, and how I resolved it. Autotest is not the culprit - the problem is running all the tests in one batch. Running rspec spec/**/* causes the same error. This is a clue that the issue is to do with a failure to clean up the database properly between tests. I am using devise, and so am ultimately relying on warden for rack-based authentication. Following the Warden documentation I put a call to Warden.test_reset! in the after(:suite) block. If I move this to the after(:all) block, the tests pass the same whether run individually, as request tests, or all the tests in one block.

So what did we learn? I think the problem (for me at least) was ultimately caused by a confusing naming of the rspec hooks. I think of a "suite" of rooms in a hotel as being less than "all" the rooms in a hotel. But apparently a "suite" of rspec examples is more than "all" the examples. In fact a suite of rspec examples is just "all the examples under the spec directory". Cleaning up warden after they have all been run has no effect. We need to clean them up after each spec - and to do that need to run test_reset! in the after(:all) block.

Hope this helps someone ...

autotest and rspec giving different answers

1 Answers