Extending the Lucene document with additional fields programatically

Question

I'm trying to include a related document (Sitecore-specifically, the Lucene document is an Item) so that when a Lucene index creates a document for an item of type A, it will also include all properties from another item B.

The end result being that when the user searches for data that is found on item B, the user gets a hit on item A. Essentially, I guess that I'm trying to "extend" a Lucene document programatically.

Here's my code so far. I'm extending the indexer class and overriding a method in which I'm adding the fields from item B to item A (the context document). In my web.config I've added a specific search index (for debugging speed) with my custom database crawler class.

public class DatabaseCrawlerExtension : Sitecore.Search.Crawlers.DatabaseCrawler
{
    protected override void AddAllFields(Lucene.Net.Documents.Document document, Sitecore.Data.Items.Item item, bool versionSpecific)
    {
        base.AddAllFields(document, item, versionSpecific);

        string fieldName;
        if (/* item is of template A */)
        {
            var targetItems = /* get items based on a property */;
            foreach (var additionalIndexItem in targetItems)
            {
                foreach (var fieldKey in additionalIndexItem.Fields
                    .Select(f => f.Key)
                    .Where(fk => !fk.StartsWith("_")))
                {
                    document.Add(base.CreateValueField(fieldKey, additionalIndexItem[fieldKey]));
                }
            }
        }
    }
}

I've debugged this code and can see that it hits the line calling document.Add, with the correct data being added. What I've tried differently is varying between calling the base method first or last, and trying to use the method AddSpecialFields instead of AddAllFields. This has not produced any additional data in the index.

To debug/look at the index, I've been both rebuilding the index (in Sitecore) and looking at the end result, as well as looking directly in the generated index files using a tool called Luke.

Answer 1

This is just a guess, but instead of using base.CreateValueField(), you might try creating a new Lucene.Net.Documents.Field yourself and adding it to the document. If you want to see the value in the index with Luke, be sure to set Store to Field.Store.YES on the Field, otherwise, the value will be indexed, but not stored in the index.

Answer 2

CreateValueField is a convenience method that creates an indexed and unstored field in the document (per sitecore documentation). . I suspect this is the problem you are running into. Luke will not display index-only fields when fetched from the index, as one might expect.

This sounds like it is correct behavior for your use, as described. You want to Search values in Item B, and return the values in Item A, so values in Item B don't need to be returned from a search. Try testing whether a search on a value from Item B is functional, rather than just viewing stored fields. You can also try pressing "Reconstruct & Edit", which may be able to build a document containing unstored field values.

The other possible problem is in the type of data you are attempting to include from Item B. CreateValueField attempts to index a value, such as a number or date. These are handled differently, and I don't really know how it would behave with a text value. If you are trying to include textual values, you should use CreateTextField, instead.

Answer 3

Below is the code of my custom media crawler. It works for me and adds custom values to the lucene index. It doesn't execute base.AddAllFields but this shouldn't be the reason why your code doesn't work.

public class MyMediaCrawler : Sitecore.Search.Crawlers.DatabaseCrawler
{
    protected override void AddAllFields(Document document, Item item, bool versionSpecific)
    {
        MediaItem mediaItem = item;

        document.Add(CreateField(Sitecore.Search.BuiltinFields.Content, item.DisplayName, true, 1f));
        document.Add(CreateField("anc", String.Join(" ", item.Axes.GetAncestors().Select(a => a.ID.ToShortID())), true, 1f));
        document.Add(CreateField("filename", String.IsNullOrEmpty(mediaItem.Title) ? item.DisplayName : mediaItem.Title, false, 1f));
    }
}