45_filtering.asciidoc

Filtering Queries and Aggregations

A natural extension to aggregation scoping are filtering queries. Because the aggregation operates in the context of the query scope, any filter applied to the query will also apply to the aggregation.

Filtering Queries

If we want to find all cars over $10,000 and also calculate the average price for those cars, we can use a constant_score query and its filter clause:

GET /cars/transactions/_search
{
    "size" : 0,
    "query" : {
        "constant_score": {
            "filter": {
                "range": {
                    "price": {
                        "gte": 10000
                    }
                }
            }
        }
    },
    "aggs" : {
        "single_avg_price": {
            "avg" : { "field" : "price" }
        }
    }
}

Fundamentally, using a non-scoring query is no different from using a match query, as we discussed in the previous chapter. The query returns a certain subset of documents, and the aggregation operates on those documents. It just happens to omit scoring and may proactively cache bitsets, etc.

Filter Bucket

But what if you would like to filter just the aggregation results? Imagine we are building the search page for our car dealership. We want to display search results according to what the user searches for. But we also want to enrich the page by including the average price of cars (matching the search) that were sold in the last month.

We can’t use simple scoping here, since there are two different criteria. The search results must match ford, but the aggregation results must match ford AND sold > now - 1M.

To solve this problem, we can use a special bucket called filter. You specify a filter, and when documents match the filter’s criteria, they are added to the bucket.

Here is the resulting query:

GET /cars/transactions/_search
{
   "size" : 0,
   "query":{
      "match": {
         "make": "ford"
      }
   },
   "aggs":{
      "recent_sales": {
         "filter": { (1)
            "range": {
               "sold": {
                  "from": "now-1M"
               }
            }
         },
         "aggs": {
            "average_price":{
               "avg": {
                  "field": "price" (2)
               }
            }
         }
      }
   }
}

1. Using the filter bucket to apply a filter in addition to the query scope.

2. This avg metric will therefore average only docs that are both ford and sold in the last month.

Since the filter bucket operates like any other bucket, you are free to nest other buckets and metrics inside. All nested components will "inherit" the filter. This allows you to filter selective portions of the aggregation as required.

Post Filter

So far, we have a way to filter both the search results and aggregations (a non-scoring filter query), as well as filtering individual portions of the aggregation (filter bucket).

You may be thinking to yourself, "hmm…​is there a way to filter just the search results but not the aggregation?" The answer is to use a post_filter.

This is a top-level search-request element that accepts a filter. The filter is applied after the query has executed (hence the post moniker: it runs post query execution). Because it operates after the query has executed, it does not affect the query scope—​and thus does not affect the aggregations either.

We can use this behavior to apply additional filters to our search criteria that don’t affect things like categorical facets in your UI. Let’s design another search page for our car dealer. This page will allow the user to search for a car and filter by color. Color choices are populated via an aggregation:

GET /cars/transactions/_search
{
    "size" : 0,
    "query": {
        "match": {
            "make": "ford"
        }
    },
    "post_filter": {    (1)
        "term" : {
            "color" : "green"
        }
    },
    "aggs" : {
        "all_colors": {
            "terms" : { "field" : "color" }
        }
    }
}

1. The post_filter element is a top-level element and filters just the search hits.

The query portion is finding all ford cars. We are then building a list of colors with a terms aggregation. Because aggregations operate in the query scope, the list of colors will correspond with the colors that Ford cars are painted.

Finally, the post_filter will filter the search results to show only green ford cars. This happens after the query is executed, so the aggregations are unaffected.

This is often important for coherent UIs. Imagine that a user clicks a category in your UI (for example, green). The expectation is that the search results are filtered, but not the UI options. If you applied a Boolean filter query, the UI would instantly transform to show only green as an option—​not what the user wants!

Warning

Performance consideration Use a post_filter only if you need to differentially filter search results and aggregations. Sometimes people will use post_filter for regular searches. Don’t do this! The nature of the post_filter means it runs after the query, so any performance benefit of filtering (such as caches) is lost completely. The post_filter should be used only in combination with aggregations, and only when you need differential filtering.

Recap

Choosing the appropriate type of filtering—​search hits, aggregations, or both—​often boils down to how you want your user interface to behave. Choose the appropriate filter (or combinations) depending on how you want to display results to your user.

• A non-scoring query inside a filter clause affects both search results and aggregations.

• A filter bucket affects just aggregations.

• A post_filter affects just search results.