Spring data ElastiSearch aggregations with filters

Question

I am trying to perform aggregations on values filtered by some conditions. I am using ElasticSearchTemplate.query() method of spring data too execute query and get the results in result extractor. I am getting the hits correctly (i.e. filters are applied and docs matching those values are only retrieved.). However, aggregations are performed on all the docs. I believe aggregations should be applied to filtered values only. Following is the code I am using:

SearchQuery query = //get the query    
SearchResponse hits = template.query(query, new ResultsExtractor<SearchResponse>() {
                @Override
                public SearchResponse extract(SearchResponse response) {
                    return response;
                }
            });

To debug the problem further, I wrote the code to execute the query rather than using spring data. Following is the code:

SearchRequestBuilder builder = esSetup.client().prepareSearch("document");
            builder.setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), query.getFilter()));

            builder.addFields(query.getFields().toArray(new String[query.getFields().size()]));
            for(AbstractAggregationBuilder aggregation : query.getAggregations()){
                builder.addAggregation(aggregation);
            }
            SearchResponse response = builder.get();

To my surprise, this query executed correctly and filters were applied on aggregates as well. To analyze further, I went through the code of elasticsearchtemplate and found that it uses setPostFilter method to set the filter. I then modified my code to set the filter that way:

SearchRequestBuilder builder = esSetup.client().prepareSearch("document");
//          builder.setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), query.getFilter()));
            builder.setPostFilter(query.getFilter());
            builder.addFields(query.getFields().toArray(new String[query.getFields().size()]));
            for(AbstractAggregationBuilder aggregation : query.getAggregations()){
                builder.addAggregation(aggregation);
            }
            SearchResponse response = builder.get();

When I executed above code, it showed same behavior as spring data! (i.e. filters were applied on query but not aggregates. Is this a bug of spring data es? If not, then, is there any other method which I should be using to retrieve the data the way I want?

Thanks in advance.

Answer 1

This behaviour is by design in Elasticsearch.

In very simple words, input to aggregations AND post filter is the set of documents that match the query section of the request body. Hence aggregations are not applied over the filtered documents.

However, if you do want aggregations to be applied over the filtered documents, "move the filters inside the query section", that is, use filtered query. Now output of the query section will be the filtered set of documents and aggregations will apply on them as expected.

So for your requirements, use filtered query instead of post filter.