-
Notifications
You must be signed in to change notification settings - Fork 556
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #95 from DirectXMan12/docs/common-issues
[docs] Config Walkthroughs and FAQs
- Loading branch information
Showing
4 changed files
with
567 additions
and
226 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,230 @@ | ||
| Configuration Walkthroughs | ||
| ========================== | ||
|
|
||
| *If you're looking for reference documentation on configuration, please | ||
| read the the [configuration reference](/docs/config.md)* | ||
|
|
||
| Per-pod HTTP Requests | ||
| --------------------- | ||
|
|
||
| ### Background | ||
|
|
||
| *The [full walkthrough](/docs/walkthrough.md) sets up a the background for | ||
| something like this* | ||
|
|
||
| Suppose we have some frontend webserver, and we're trying to write an | ||
| configuration for the Promtheus adapter so that we can autoscale it based | ||
| on the HTTP requests per second that it receives. | ||
|
|
||
| Before starting, we've gone and instrumented our frontend server with | ||
| a metric, `http_requests_total`. It is exposed with a single label, | ||
| `method`, breaking down the requests by HTTP verb. | ||
|
|
||
| We've configured our Prometheus to collect the metric, and our promethues | ||
| adds the `kubernetes_namespace` and `kubernetes_pod_name` labels, | ||
| representing namespace and pod, respectively. | ||
|
|
||
| If we query Prometheus, we see series that look like | ||
|
|
||
| ``` | ||
| http_requests_total{method="GET",kubernetes_namespace="production",kubernetes_pod_name="frontend-server-abcd-0123"} | ||
| ``` | ||
|
|
||
| ### Configuring the adapter | ||
|
|
||
| The adapter considers metrics in the following ways: | ||
|
|
||
| 1. First, It discovers the metrics available (*Discovery*) | ||
|
|
||
| 2. Then, it figures out which Kubernetes resources each metric is | ||
| associated with (*Association*) | ||
|
|
||
| 3. Then, it figures out how it should expose them to the custom metrics | ||
| API (*Naming*) | ||
|
|
||
| 4. Finally, it figures out how it should query Prometheus to get the | ||
| actual numbers (*Querying*) | ||
|
|
||
| We need to inform the adapter how it should perform each of these steps | ||
| for our metric, `http_requests_total`, so we'll need to add a new | ||
| ***rule***. Each rule in the adapter encodes these steps. Let's add a new | ||
| one to our configuration: | ||
|
|
||
| ```yaml | ||
| rules: | ||
| - {} | ||
| ``` | ||
| If we want to find all `http_requests_total` series ourselves in the | ||
| Prometheus dashboard, we'd write | ||
| `http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}` to | ||
| find all find all `http_requests_total` series that were associated with | ||
| a namespace and pod. | ||
|
|
||
| We can add this to our rule in the `seriesQuery` field, to tell the | ||
| adapter how *discover* the right series itself: | ||
|
|
||
| ```yaml | ||
| rules: | ||
| - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}' | ||
| ``` | ||
|
|
||
| Next, we'll need to tell the adapter how to figure out which Kubernetes | ||
| resources are associated with the metric. We've already said that | ||
| `kubernetes_namespace` represents the namespace name, and | ||
| `kubernetes_pod_name` represents the pod name. Since these names don't | ||
| quite follow a consistent pattern, we use the `overrides` section of the | ||
| `resources` field in our rule: | ||
|
|
||
| ```yaml | ||
| rules: | ||
| - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}' | ||
| resources: | ||
| overrides: | ||
| kubernetes_namespace: {resource: "namespace"} | ||
| kubernetes_pod_name: {resource: "pod"} | ||
| ``` | ||
|
|
||
| This says that each label represents its corresponding resource. Since the | ||
| resources are in the "core" kubernetes API, we don't need to specify | ||
| a group. The adapter will automatically take care of pluralization, so we | ||
| can specify either `pod` or `pods`, just the same way as in `kubectl get`. | ||
| The resources can be any resource available in your kubernetes cluster, as | ||
| long as you've got a corresponding label. | ||
|
|
||
| If our labels followed a consistent pattern, like `kubernetes_<resource>`, | ||
| we could specify `resources: {template: "kubernetes_<<.Resource>>"}` | ||
| instead of specifying an override for each resource. If you want to see | ||
| all resources currently available in your cluster, you can use the | ||
| `kubectl api-resources` command (but the list of available resources can | ||
| change as you add or remove CRDs or aggregated API servers). For more | ||
| information on resources, see [Kinds, Resources, and | ||
| Scopes](https://github.com/kubernetes-incubator/custom-metrics-apiserver/blob/master/docs/getting-started.md#kinds-resources-and-scopes) | ||
| in the custom-metrics-apiserver boilerplate guide. | ||
|
|
||
| Now, cumulative metrics (like those that end in `_total`) aren't | ||
| particularly useful for autoscaling, so we want to convert them to rate | ||
| metrics in the API. We'll call the rate version of our metric | ||
| `http_requests_per_second`. We can use the the `name` field to tell the | ||
| adapter about that: | ||
|
|
||
| ```yaml | ||
| rules: | ||
| - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}' | ||
| resources: | ||
| overrides: | ||
| kubernetes_namespace: {resource: "namespace"} | ||
| kubernetes_pod_name: {resource: "pod"} | ||
| name: | ||
| matches: "^(.*)_total" | ||
| as: "${1}_per_second" | ||
| ``` | ||
|
|
||
| Here, we've said that we should take the name matching | ||
| `<something>_total`, and turning it into `<something>_per_second`. | ||
|
|
||
| Finally, we need to tell the adapter how to actually query Prometheus to | ||
| get some numbers. Since we want a rate, we might write: | ||
| `sum(rate(http_requests_total{kubernetes_namespace="production",kubernetes_pod_name=~"frontend-server-abcd-0123|fronted-server-abcd-4567"}) by (kubernetes_pod_name)`, | ||
| which would get us the total requests per second for each pod, summed across verbs. | ||
|
|
||
| We can write something similar in the adapter, using the `metricsQuery` | ||
| field: | ||
|
|
||
| ```yaml | ||
| rules: | ||
| - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}' | ||
| resources: | ||
| overrides: | ||
| kubernetes_namespace: {resource: "namespace"} | ||
| kubernetes_pod_name: {resource: "pod"} | ||
| name: | ||
| matches: "^(.*)_total" | ||
| as: "${1}_per_second" | ||
| metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)' | ||
| ``` | ||
|
|
||
| The adapter will automatically fill in the right series name, label | ||
| matchers, and group-by clause, depending on what we put into the API. | ||
| Since we're only working with a single metric anyway, we could replace | ||
| `<<.Series>>` with `http_requests_total`. | ||
|
|
||
| Now, if we run an instance of the Prometheus adapter with this | ||
| configuration, we should see discovery information at | ||
| `$KUBERNETES/apis/custom.metrics.k8s.io/v1beta1/` of | ||
|
|
||
| ```json | ||
| { | ||
| "kind": "APIResourceList", | ||
| "apiVersion": "v1", | ||
| "groupVersion": "custom.metrics.k8s.io/v1beta1", | ||
| "resources": [ | ||
| { | ||
| "name": "pods/http_requests_total", | ||
| "singularName": "", | ||
| "namespaced": true, | ||
| "kind": "MetricValueList", | ||
| "verbs": ["get"] | ||
| }, | ||
| { | ||
| "name": "namespaces/http_requests_total", | ||
| "singularName": "", | ||
| "namespaced": false, | ||
| "kind": "MetricValueList", | ||
| "verbs": ["get"] | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Notice that we get an entry for both "pods" and "namespaces" -- the | ||
| adapter exposes the metric on each resource that we've associated the | ||
| metric with (and all namespaced resources must be associated with | ||
| a namespace), and will fill in the `<<.GroupBy>>` section with the | ||
| appropriate label depending on which we ask for. | ||
|
|
||
| We can now connect to | ||
| `$KUBERNETES/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_per_second`, | ||
| and we should see | ||
|
|
||
| ```json | ||
| { | ||
| "kind": "MetricValueList", | ||
| "apiVersion": "custom.metrics.k8s.io/v1beta1", | ||
| "metadata": { | ||
| "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_per_second", | ||
| }, | ||
| "items": [ | ||
| { | ||
| "describedObject": { | ||
| "kind": "Pod", | ||
| "name": "frontend-server-abcd-0123", | ||
| "apiVersion": "/__internal", | ||
| }, | ||
| "metricName": "http_requests_per_second", | ||
| "timestamp": "2018-08-07T17:45:22Z", | ||
| "value": "16m" | ||
| }, | ||
| { | ||
| "describedObject": { | ||
| "kind": "Pod", | ||
| "name": "frontend-server-abcd-4567", | ||
| "apiVersion": "/__internal", | ||
| }, | ||
| "metricName": "http_requests_per_second", | ||
| "timestamp": "2018-08-07T17:45:22Z", | ||
| "value": "22m" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| This says that our server pods are receiving 16 and 22 milli-requests per | ||
| second (depending on the pod), which is 0.016 and 0.022 requests per | ||
| second, written out as a decimal. That's about what we'd expect with | ||
| little-to-no traffic except for the Prometheus scrape. | ||
|
|
||
| If we added some traffic to our pods, we might see `1` or `20` instead of | ||
| `16m`, which would be `1` or `20` requests per second. We might also see | ||
| `20500m`, which would mean 20500 milli-requests per second, or 20.5 | ||
| requests per second in decimal form. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.