If you want to know how your app will behave in different browsers, you'll want to run it in those browsers. Different browsers have different DOM implementations, JavaScript versions and features, etc.
For example, if you were to run the following code in Chrome or PhantomJS it would work fine, but in IE8 it would fail:
var arr = [1, 2, 3]
arr.forEach(function(item) {
console.log(item);
});
forEach
isn't available in IE8, but it is in Chrome. A Unit test that ran the code against IE8 would have caught that, a unit test against PhantomJS would not have caught that.
... so if you want to run something "headless" like PhantomJS for your development cycle, that's fine, but be very sure you're testing all of the browsers you care about when you build (Hopefully with CI)