40_scope.asciidoc

Scoping Aggregations

With all of the aggregation examples given so far, you may have noticed that we omitted a query from the search request. The entire request was simply an aggregation.

Aggregations can be run at the same time as search requests, but you need to understand a new concept: scope. By default, aggregations operate in the same scope as the query. Put another way, aggregations are calculated on the set of documents that match your query.

Let’s look at one of our first aggregation examples:

GET /cars/transactions/_search
{
    "size" : 0,
    "aggs" : {
        "colors" : {
            "terms" : {
              "field" : "color"
            }
        }
    }
}

You can see that the aggregation is in isolation. In reality, Elasticsearch assumes "no query specified" is equivalent to "query all documents." The preceding query is internally translated as follows:

GET /cars/transactions/_search
{
    "size" : 0,
    "query" : {
        "match_all" : {}
    },
    "aggs" : {
        "colors" : {
            "terms" : {
              "field" : "color"
            }
        }
    }
}

The aggregation always operates in the scope of the query, so an isolated aggregation really operates in the scope of a match_all query—​that is to say, all documents.

Once armed with the knowledge of scoping, we can start to customize aggregations even further. All of our previous examples calculated statistics about all of the data: top-selling cars, average price of all cars, most sales per month, and so forth.

With scope, we can ask questions such as "How many colors are Ford cars available in?" We do this by simply adding a query to the request (in this case a match query):

GET /cars/transactions/_search
{
    "query" : {
        "match" : {
            "make" : "ford"
        }
    },
    "aggs" : {
        "colors" : {
            "terms" : {
              "field" : "color"
            }
        }
    }
}

Since we aren’t specifying "size" : 0, both the search results and the aggregation results are returned:

{
...
   "hits": {
      "total": 2,
      "max_score": 1.6931472,
      "hits": [
         {
            "_source": {
               "price": 25000,
               "color": "blue",
               "make": "ford",
               "sold": "2014-02-12"
            }
         },
         {
            "_source": {
               "price": 30000,
               "color": "green",
               "make": "ford",
               "sold": "2014-05-18"
            }
         }
      ]
   },
   "aggregations": {
      "colors": {
         "buckets": [
            {
               "key": "blue",
               "doc_count": 1
            },
            {
               "key": "green",
               "doc_count": 1
            }
         ]
      }
   }
}

This may seem trivial, but it is the key to advanced and powerful dashboards. You can transform any static dashboard into a real-time data exploration device by adding a search bar. This allows the user to search for terms and see all of the graphs (which are powered by aggregations, and thus scoped to the query) update in real time. Try that with Hadoop!

Global Bucket

You’ll often want your aggregation to be scoped to your query. But sometimes you’ll want to search for a subset of data, but aggregate across all of your data.

For example, say you want to know the average price of Ford cars compared to the average price of all cars. We can use a regular aggregation (scoped to the query) to get the first piece of information. The second piece of information can be obtained by using a global bucket.

The global bucket will contain all of your documents, regardless of the query scope; it bypasses the scope completely. Because it is a bucket, you can nest aggregations inside it as usual:

GET /cars/transactions/_search
{
    "size" : 0,
    "query" : {
        "match" : {
            "make" : "ford"
        }
    },
    "aggs" : {
        "single_avg_price": {
            "avg" : { "field" : "price" } (1)
        },
        "all": {
            "global" : {}, (2)
            "aggs" : {
                "avg_price": {
                    "avg" : { "field" : "price" } (3)
                }

            }
        }
    }
}

1. This aggregation operates in the query scope (for example, all docs matching ford)

2. The global bucket has no parameters.

3. This aggregation operates on the all documents, regardless of the make.

The single_avg_price metric calculation is based on all documents that fall under the query scope—​all ford cars. The avg_price metric is nested under a global bucket, which means it ignores scoping entirely and calculates on all the documents. The average returned for that aggregation represents the average price of all cars.

If you’ve made it this far in the book, you’ll recognize the mantra: use a filter wherever you can. The same applies to aggregations, and in the next chapter we show you how to filter an aggregation instead of just limiting the query scope.