Ember upgrade breaks one test, and only in Safari

Question

I'm trying to catch a Heisenbug.

I'm updating our project from Ember CLI 0.2.0 and Ember 1.10.0 to Ember CLI 0.2.3 and Ember 1.11.1. This has been a pretty pain-free process, but I have exactly one test which now fails in Safari (7.1.5) only. It passes in PhantomJS, Chrome and Firefox.

Annoyingly, the test only fails when the test run is initiated by Testem (that is, when a change in the code triggers an auto-update test run). If I initiate tests from inside the Qunit web interface, it passes. Both of these things are true regardless of the test grouping. The feature being tested works just fine when run manually in the browser.

It's an integration test and verifies that when a value is changed in an input, the input updates with the value returned from the server. In the tests, the "server" is a Pretender instance. Here's what the test itself looks like:

test('Editing allocation cell', function() {
  visit('/district/periods');

  fillIn(SELECTORS.definitionRowInput(1,0), '100');
  triggerEvent(SELECTORS.definitionRowInput(1,0), 'focusout');
  // The triggerEvent should be tripping the focusOut event on a particular
  // Ember.Textfield subclass, which subsequently leads to a POST request to
  // the server. On Safari, however, the focusOut event isn't being called here.
  // It is called elsewhere in the app, and it works in production.
  // Things that also don't work: keyEvent(element, 'keypress', 16) (a tab), 
  // sending 'blur', sending 'focus-out'.
  // 'focus-out' also fails in Firefox, 'blur' and tab fail in all 4 envs

  andThen(function() {
    equal($(SELECTORS.definitionRowInput(1,0)).val(), '90', 'The updated input takes the return value from the server (even if it is different from input)');
    equal($(SELECTORS.gradeTotal(2)).text(), '120', 'Grade total updates with the new sum');
  });
});

Note the second andThen() block: by sending focusout to the control, we should be prompting code in the backing component to update the data back to the server. The other browsers do this - I put a console.log() in the Pretender responder to verify it - but Safari doesn't. I'm guessing it's not responding properly to the focusout event.

Considering that this test passes in a branch which only differs by the Ember CLI update... what's likely to have changed to make this break?

ETA: I've individually rolled back all the libraries updated in this update and the test continues to fail. The only change which seems to work is rolling back Ember itself. It breaks in 1.11.0 in the same way as 1.11.1, so the change is between 1.10.0 and 1.11.0. (That only leaves me ~600 commits to sift through...)

ETA2: I've narrowed the scope to between 1.11.0-beta.5 and 1.11.0. I'm trying to use bower link to use local builds of ember but the ember builds have been unreliable so far in running the tests and the relationship of those two tags isn't one which leads to effective bisecting.

I have had the same issues with testem when running on the command line, I suspect it's a race condition with resolving promises. Have you considered using an Ember.run... method around that one call? — jonathanKingston
Around the triggerEvent() call? No, I hadn't thought of that - I'll give it a try. Thanks! — pjmorse
...nope. Still there. Two things seem key to me: (1) There's a pretty clear pair of "it works here" and "it's broken here" tags in Ember itself, and (2) it affects Safari but not other browsers. With a race condition I would expect it to be less reliably reproducible? — pjmorse
Perhaps race condition is the wrong term, it seems however that it is time based issues which was similar to how I solved mine with Ember.run.next, Ember.run.later or Ember.run.once. This was used around the trigger call. I think it could be an issue with the browser not treating the calls as synchronous as they perhaps should be. I have never really had chance to debug it enough to find the root cause of it but it has impacted other applications, safari and also mobile browsers behaved in this manner. — jonathanKingston

user3710044 user3710044 · Accepted Answer · 2015-04-25T09:57:49

I can't help you with actually running your tests, but flattening the history isn't that difficult. There are 49 patches between the tags you mention. A stream of git cherry-pick commands gives the branch I've uploaded at https://github.com/rdebath/test/tree/ember.js

Each one of the commits on that branch (after the v1.11.0-beta.5 tag) comes from a 'nice' route between the tags you mentioned. The commit hashes are all different (obviously) but the final tree hash is the same as v1.11.0 so this should be a good path for git bisect.

The build problems can also be avoided, for example, I would suggest using a bisect to find the patch that causes them and git rebase -i that patch as late as you can. That should put a problem right next to it's fix; but it's probably not a good idea to "squash" those commits as you want to be able to relate everything back to the real tree.

To help choose the list of commits I used the command:

git log --graph --decorate --oneline --date-order --all

With that the "nice" path is reasonably obvious.

Ember upgrade breaks one test, and only in Safari

1 Answers