ember-data: Loading hasMany association on demand

Question

(Updated for ember-data API Rev 11...)

TL;DR

What is the right way to use DS.Adapter.findAssociation(...)DS.Adapter.findHasMany(...) to load hasMany associations on demand? Especially, once you load child records, how do you deal with the fact that re-loading the parent record from the server empties the hasMany array? I do not want to (perhaps can't) include the id's of the child records in an array in the parent. Or is there another way to do this that I'm missing?

As a side note, I'm confused about what options should be passed in a hasMany/belongsTo definition for key linkage (and am I supposed to use mappings if I have no sideloaded data or array of ids?), so if you think my problem could lie in my association definition, there's a good chance you're right.

Long version

I am writing my own subclass of DS.RESTAdapter to tie ember-data to an ASP.NET WebAPI backend (using Entity Framework). So far so good, but I'm having a heck of a time getting associations to work right.

Similar to this poster, I noticed that ember-data's front page says used to say that if you have a hasMany association in your model, and you get that property, the store will issue a request for the child records. Quoting from the page:

If you were to request the profile, like this:

author.get('profile');

…the REST adapter would send a request to the URL /profiles?author_id=1.

The implication is this is what happens if you don't sideload and don't include an array of ids. I realize that these docs are somewhat out of date, but I haven't been able to make this happen, either in API version 7 or more recently version 9. However in version 9 I did find the findAssociation method, in version 11 there is the findHasMany method, which I'm guessing is what might have been used to make this happen, and which I'm now trying to use.

Why not include an array of ids or sideload?

Three main reasons I don't want to do this (and possibly can't):

1. It's not obvious how to do either of these things with ASP.NET WebAPI, at least not with the simple decoration-based approach I'm using. And, I really like the simplicity and thinness of the backend right now, with EF and WebAPI it's almost entirely boilerplate for each entity and I'm done! I even get OData filtering support "free".

2. My child records will often be generated via expensive queries (aggregates...metrics rollups, for instance). And there are lots of different classes of child entities for a single parent entity. So even getting the ids for all the child types would be expensive, and generating and sideloading all the child records are out of the question.

3. I have child entities where the primary key is a composite key. I haven't seen an example of this even being supported/possible in ember-data, at least not for dealing with associations (e.g. how would you do an array of ids?). I made a computed property in my client-side model that coerces the composite key into a single string, so I can retrieve a single record from the store using find(...), but again I have no idea how this would even work with an association.

Trying to use findAssociationfindHasMany

I've figured out that in API version 9 (and some earlier versions but not all?) 11, I can perhaps implement the DS.Adapter.findAssociation DS.Adapter.findHasMany method to retrieve the child records of a hasMany association. This mostly works, but requires some gymnastics. Here is my generalized findAssociation findHasMany method:

findHasMany: function (store, record, relationship, ids) {

    var adapter = this;
    var root = this.rootForType(relationship.type);
    var query = relationship.options.query(record);

    var hits = store.findQuery(relationship.type, query);

    hits.on('didLoad', function () {
        // NOTE: This MUST happen in the callback, because findHasMany is in
        // the execution path for record.get(relationship.key)!!! Otherwise causes
        // infinite loop!!!
        var arrMany = record.get(relationship.key);

        if (hits.get('isLoaded')) {
            arrMany.loadingRecordsCount = 1 + hits.get('length') + (typeof arrMany.loadingRecordsCount == "number" ? arrMany.loadingRecordsCount : 0);
            hits.forEach(function (item, index, enumerable) {
                arrMany.addToContent(item);
                arrMany.loadedRecord();
            });
            arrMany.loadedRecord(); // weird, but this and the "1 +" above make sure isLoaded/didLoad fires even if there were zero results.
        }
    });

}

To make this work, my hasMany definitions set a query option value which is a function on the record that returns a hash of parameters for the query string in the request for the children. Since this is for a ASP.NET WebAPI backend, this will probably be an OData filter, e.g.:

App.ParentEntity = DS.Model.extend({
    ...
    children: DS.hasMany('App.ChildEntity', {
        query: function (record) {
            return {
                "$filter": "ChildForeignKey eq '" + record.get('id') + "'"
            };
        }
    })
});

One of the tricks is adding the items to the ManyArray using addToContent(item), so that the parent record doesn't get marked "dirty" as if it has been edited. The other is, when I retrieve the JSON for the parent record initially, I have to manually set a value of true for the association name's key (the JSON coming from the server does not have the key at all). I.e.:

    var a = Ember.isArray(json) ? json : [json];
    for (var i = 0; i < a.length; i++) {
        type.eachAssociation(function (key) {
            var meta = type.metaForProperty(key);
            a[i][key] = true;
        });
    }

This sounds nuts, but this is why: If you look at the implementation of DS.Store.findMany and find where findAssociation findHasMany is called, you'll find:

findMany: function(type, ids, record, relationship){
...
if (!Ember.isArray(ids)) {
  var adapter = this.adapterForType(type);
  if (adapter && adapter.findHasMany) { adapter.findHasMany(this, record, relationship, ids); }
  else { throw fmt("Adapter is either null or does not implement `findMany` method", this); }

  return this.createManyArray(type, Ember.A());
}

And if you look at this line from the ember-data internal function hasAssociation hasRelationship where findMany is called, you'll see what is passed for the 2nd parameter:

relationship = store.findMany(type, ids || [], this, meta);

So, the only way to get findAssociation findHasMany to be called is to have the value in the JSON be something that is "truthy", but is not an Array--I use true. I'm thinking this is either a bug/incomplete, or an indication that I'm on the wrong track--if someone could tell me which that would be great too.

With all that, I can get ember-data to automatically issue a request to the server for the child records, e.g. to http://myserver.net/api/child_entity/$filter=ChildForeignKey eq '42' and it works--the child records get loaded, and they get associated with the parent record (incidentally, the inverse belongsTo relationship gets populated properly too, despite the fact that I'm not explicitly touching it--I have no idea where or how that is happening).

But that breaks down pretty quickly if I don't stop there.

Dealing with reloading the parent record

So say I successfully load up the child records into a parent record, but then navigate to a place where all the parent records are retrieved (to populate a menu). Since the newly loaded parent records have no array of ids and nothing is sideloaded, the parent record is refreshed without any children again! Worse, the ManyArray's isLoaded property remains true! So I can't even observe anything to re-load the children.

So if I simultaneously have a view onscreen displaying the child values, it immediately zaps to having no child record values. Or if I navigate back to one, when App.store.find('App.ParentEntity', 42) is called, the record is loaded from the store without a request to the server, and of course it has no child records.

This is hint #2 that I probably am going about this the wrong way. So...what is the right way to load child records on demand?

Thanks much!

Answer 1

Based on the latest Ember Data (as of Jan 25th 2013)... here's my solution to lazy loading hasMany relationships. I modifed DS.hasMany and added a method to DS.Adapter.

I changed two lines in DS.hasMany:

DS.hasMany = function(type, options) {
  Ember.assert("The type passed to DS.hasMany must be defined", !!type);
  return (function(type, options) {
    options = options || {};
    var meta = { type: type, isRelationship: true, options: options, kind: 'hasMany' };
    return Ember.computed(function(key, value) {
      var data = get(this, 'data').hasMany,
          store = get(this, 'store'),
          ids, relationship;

      if (typeof type === 'string') {
        type = get(this, type, false) || get(Ember.lookup, type);
      }

      meta.key = key;
      ids = data[key];
      relationship = store.findMany(type, ids, this, meta);
      set(relationship, 'owner', this);
      set(relationship, 'name', key);
      return relationship;
    }).property().meta(meta);
  })(type, options);

};

First, I added the key to the meta object...

meta.key = key;

...and second, as previously noted above, I removed the empty array from the findMany call by changing...

relationship = store.findMany(type, ids || [], this, meta);

...to...

relationship = store.findMany(type, ids, this, meta);

...allowing ids to be passed in to findMany as undefined.

Next, I added a didFindHasMany hook to DS.Adapter:

DS.Adapter.reopen({

  /**
   Loads the response to a request for records by findHasMany.

   Your adapter should call this method from its `findHasMany`
   method with the response from the backend.

   @param {DS.Store} store
   @param {subclass of DS.Model} type
   @param {any} payload
   @param {subclass of DS.Model} record the record of which the relationship is a member (parent record)
   @param {String} key the property name of the relationship on the parent record
   */
  didFindHasMany: function(store, type, payload, record, key) {

    var loader = DS.loaderFor(store);

    loader.populateArray = function(references) {
      store.loadHasMany(record, key, references.map(function(reference) { return reference.id; }));
    };

    get(this, 'serializer').extractMany(loader, payload, type);
  }

});

I modeled this after the DS.Adapter's didFindQuery hook using the loadHasMany that I found already implemented on the DS.Store. Then in my custom adapter I implemented a findHasMany method that uses the following code in its success callback:

Ember.run(this, function() {
  adapter.didFindHasMany(store, type, response.data, record, key);
});

I haven't tested this extensively but it seems to be working correctly. Looking through recent modifications made to ember-data's code, it seems to me that they have slowly been moving in a direction in which something similar to this approach will be supported at some point in the future. Or at the very least that's my hope.