Can't nest terms aggregations more than two deep without further aggs being ignored?

Question

I'm querying ElasticSearch using the Nest library for C#, to fetch graph data with multiple pivots. Each pivot is a nested TermsAggregation on a query, and everything works fine with one or two pivots. Once I get to three pivots, though, the SearchRequest object won't generate further aggregations.

The code to build the aggregations looks like this:

TermsAggregation topTermAgg = null;
TermsAggregation currentAgg = null;
foreach (var pivotName in activePivots)
{
    newTermAgg = new TermsAggregation("pivot")
    {
        Field = pivot.ToString().ToLower()
    };

    if (topTermAgg == null)
    {
        topTermAgg = newTermAgg;
    }
    else
    {
        currentAgg.Aggregations = newTermAgg;
    }
    currentAgg = newTermAgg;
}

The SearchRequest itself is pretty straightforward:

var searchRequest = new SearchRequest(Indices.Index("a", "b", "c"))
{
    Size = 0,
    Aggregations = topTermAgg,
    Query = query,
};

Unfortunately, the SearchRequest for 3 or more pivots, when converted to string, looks like this (via nestClient.Serializer.SerializeToString(searchRequest)):

{
  "size": 0,
  "query": {
    "bool": <Fairly complex query, that works fine. It's the aggregation that has the problem.>
  },
  "aggs": {
    "pivot": {
      "terms": {
        "field": "pivot1"
      },
      "aggs": {
        "pivot": {
          "terms": {
            "field": "pivot2"
          }
        }
      }
    }
  }
}

When I inspect the searchRequest object in the debugger, it quite definitely has 3 or more aggregations. What's going on here, and how can I get 3 or more nested terms aggregations to work properly?

I am using Nest version 5.01.

Answer 1

This must be related to the way in which you're building up the nested aggregations. Arbitrarily deep nested aggregations can be built with the client. Here's an example of a three deep nested aggregation

client.Search<Question>(s => s
    .Aggregations(a => a
        .Terms("top", ta => ta
            .Field("top_field")
            .Aggregations(aa => aa
                .Terms("nested_1", nta => nta
                    .Field("nested_field_1")
                    .Aggregations(aaa => aaa
                        .Terms("nested_2", nnta => nnta
                            .Field("nested_field_3")
                        )
                    )
                )
            )
        )
    )
);

which serializes to

{
  "aggs": {
    "top": {
      "terms": {
        "field": "top_field"
      },
      "aggs": {
        "nested_1": {
          "terms": {
            "field": "nested_field_1"
          },
          "aggs": {
            "nested_2": {
              "terms": {
                "field": "nested_field_3"
              }
            }
          }
        }
      }
    }
  }
}

You can also add values to AggregationDictionary directly

var request = new SearchRequest<Question>
{
    Aggregations = new AggregationDictionary
    {
        { "top", new TermsAggregation("top")
            {
                Field = "top_field",
                Aggregations = new AggregationDictionary
                {
                    { "nested_1", new TermsAggregation("nested_1")
                        {
                            Field = "nested_field_1",
                            Aggregations = new AggregationDictionary
                            {
                                { "nested_2", new TermsAggregation("nested_2")
                                    {
                                        Field = "nested_field_2"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
};

client.Search<Question>(request);

is the same as the previous request. You can shorten this even further to

var request = new SearchRequest<Question>
{
    Aggregations = new TermsAggregation("top")
    {
        Field = "top_field",
        Aggregations = new TermsAggregation("nested_1")
        {
            Field = "nested_field_1",
            Aggregations = new TermsAggregation("nested_2")
            {
                Field = "nested_field_2"
            }
        }
    }
};

client.Search<Question>(request);

Answer 2

I got my code working by constructing the aggregation from the bottom-up, rather than from the top-down.

var terminalAggregation = <some aggregation. In my code, there's a lowest aggregation that's different from the rest. For the code I presented, you could just build the lowest pivot.>
TermsAggregation topTermAgg = null;
activePivots.Reverse();
foreach (var pivotName in activePivots)
{
    newTermAgg = new TermsAggregation("pivot")
    {
        Field = pivot.ToString().ToLower(),
        Aggregations = topTermAgg ?? terminalAggregation
    };

    topTermAgg = newTermAgg;
}

This looks like a bug in the Nest library; there are different classes like AggregationBase and BucketAggregationBase and AggregationDictionary that are all assignable to the "Aggregations" property, but it seems like there's some subtle flaw after the second assignment when you do this recursively.

The documentation is also not up-to-date: it claims that you can create an AggregationDictionary yourself, but since AggregationDictionary doesn't have a public Add() method, I really can't. Nor can I use the C#'s {}-after-insantiation syntax to populate its properties – again, because Add() is not public.