Feature specs fail only on CircleCI or Codeship continuous integration services

Question

My very basic feature specs are passing just fine locally but failing on CircleCI and Codeship. The tests that are failing:

require 'spec_helper'

describe 'Authentication' do

  describe "sign in" do
    it "is the landing page for non-authenticated users" do
      user = create(:user)
      visit root_path

      expect( page ).to have_content 'LOG IN' # On sign-in page
      fill_in 'Email', with: user.email
      fill_in "Password", with: user.password
      click_button 'Sign in'

      expect( current_path ).to eq user_path(user)
    end
  end

  describe 'registration' do
    it "allows new users to register" do
      visit root_path

      click_link 'Sign up'
      fill_in 'Email', with: '[email protected]'
      fill_in 'Password', with: 'password'
      fill_in 'Password confirmation', with: 'password'
      fill_in 'First name', with: 'John'
      fill_in 'Last name', with: 'Doe'
      click_button "Sign up"

      expect( current_path ).to include '/users/'
      expect( page ).to have_content "Welcome! You have signed up successfully."
    end
  end
end

The tests both fail on the first lines where they set expectations of the pages (expect( page ).to have_content "LOG IN" and click_link "Sign up", respectively), with errors suggesting the page HTML is completely blank:

expected to find text "LOG IN" in ""

I saved screenshots on CircleCI, and they indeed show a completely blank page.

Here's where it gets interesting. I tried debugging the problem by running/watching the specs on Circle using a VNC. When I a) set driver: :selenium for the tests, b) add a sleep 1 or two to the tests before testing the page expectations, and c) manually run the test after SSHing into their servers with the VNC, I can see the tests run in Selenium (they open a browser in the VNC) and they pass perfectly.

Outside of the VNC, however, the tests fail consistently in both CI servers. With or without tons of sleeps and driver: :selenium. Any ideas what could be causing this discrepancy between the regular CircleCI/Codeship servers and their VCN/my local test environment? I got in touch with the folks at CircleCI, but they're stumped for the moment.

If relevant, I'm running Ruby 2.2.0, Rails 4.2, Capybara 2.4.4, Capybara-Webkit 1.4.1, and Selenium-Webdriver 2.44.0

Some potentially relevant files:

spec_helper.rb

ENV["RAILS_ENV"] = "test"

require File.expand_path("../../config/environment", __FILE__)

require "rspec/rails"
require "shoulda/matchers"
require "webmock/rspec"
require 'vcr'

Dir[Rails.root.join("spec/support/**/*.rb")].each { |file| require file }

module Features
  include Warden::Test::Helpers
  Warden.test_mode!

  def sign_in(user)
    login_as(user, scope: :user)
  end
end

module Controllers
  # Pre-parse controller responses for easy access
  def response_body
    body = JSON.parse(response.body)
    body.is_a?(Hash) ? body.to_sh : body.map(&:to_sh)
  end
end

module Mock
  def disable_webmock(&block)
    WebMock.disable!
    yield
    WebMock.enable!
  end
end

RSpec.configure do |config|
  config.expect_with :rspec do |c|
    c.syntax = :expect
  end

  # Save a screenshot to CircleCI when a feature test fails
  config.after(:each, :type => :feature) do |example|
    if example.exception
      artifact = save_page
      puts "\"#{example.description}\" failed. Page saved to #{artifact}"
    end
  end

  config.include Features, type: :feature
  config.include Controllers, type: :controller
  config.include Mock
  config.include Formulaic::Dsl, type: :feature
  config.infer_spec_type_from_file_location!
  config.infer_base_class_for_anonymous_controllers = false
  config.order = "random"
  config.use_transactional_fixtures = false
end

RSpec::Matchers.define :hash_eq do |expected|
  match do |actual|
    actual.recursive_symbolize_keys == expected.recursive_symbolize_keys
  end
end

VCR.configure do |c|
  c.cassette_library_dir = 'spec/fixtures/vcr_cassettes'
  c.hook_into :webmock
  c.allow_http_connections_when_no_cassette = true
  c.configure_rspec_metadata!
  c.ignore_hosts '127.0.0.1', 'localhost:3000'
end

ActiveRecord::Migration.maintain_test_schema!
Capybara.javascript_driver = :webkit

if ENV['CIRCLE_ARTIFACTS']
  Capybara.save_and_open_page_path = ENV['CIRCLE_ARTIFACTS']
end

WebMock.disable_net_connect!(allow_localhost: true)

database_cleaner.rb

RSpec.configure do |config|
  config.use_transactional_fixtures = false

  config.before(:suite) do
    DatabaseCleaner.strategy = :truncation
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end

end

This might be a CircleCI config problem. Can you share your cicle.yml file as well? — Jan Klimo
Are you using headless or a Saas service? Most of those CI programs are purely headless and can't run something like selenium without some real work. — TIMBERings

Dave Schweisguth Dave Schweisguth · Accepted Answer · 2015-08-08T11:36:42

I can think of a couple of things to try:

If you want to get your specs to run headless, try using poltergeist rather than capybara-webkit. poltergeist has several advantages which I described here: https://stackoverflow.com/a/24108439/634576
Selenium might just be a distraction, but, if you want to get it to work, be sure that your CI environment has the right version of Firefox. Each version of selenium-webdriver seems to need a narrow range of versions of Firefox. On CircleCI, you can configure circle.yml to install a specific version of Firefox. Right now we use selenium-webdriver 2.46.2 and Firefox 39.0.3, installed with the following in circle.yml:
```
dependencies:
  pre:
    - sudo apt-get update; sudo apt-get install firefox=39.0.3+build2-0ubuntu0.12.04.1
```
I don't know about Codeship.

Feature specs fail only on CircleCI or Codeship continuous integration services

2 Answers