From 3fd1af000acc598cb27879053e5cfd3574d14c65 Mon Sep 17 00:00:00 2001 From: Andrea Longo Date: Fri, 13 Sep 2024 17:02:29 -0600 Subject: [PATCH 1/5] draft: more metrics v2/v3 rework --- ...collect-minio-metrics-using-prometheus.rst | 365 +++++------------- .../monitoring/metrics-and-alerts.rst | 54 +-- .../monitoring/metrics-v2-deprecated.rst | 305 ++++++++++++++- .../mc-admin-prometheus-generate.rst | 133 ++++++- 4 files changed, 542 insertions(+), 315 deletions(-) diff --git a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst index 12259f65..ce2fbd95 100644 --- a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst +++ b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst @@ -1,25 +1,25 @@ .. _minio-metrics-collect-using-prometheus: ======================================== -Monitoring and Alerting using Prometheus +Monitoring and alerting using Prometheus ======================================== .. default-domain:: minio .. contents:: Table of Contents :local: - :depth: 1 + :depth: 2 .. container:: extlinks-video - `Monitoring with MinIO and Prometheus: Overview `__ - `Monitoring with MinIO and Prometheus: Lab `__ -MinIO publishes cluster, node, bucket, and resource metrics using the :prometheus-docs:`Prometheus Data Model `. +MinIO publishes metrics using the :prometheus-docs:`Prometheus Data Model `. The procedure on this page documents the following: -- Configuring a Prometheus service to scrape and display metrics from a MinIO deployment -- Configuring an Alert Rule on a MinIO Metric to trigger an AlertManager action +- Configure a Prometheus service to scrape and display metrics from a MinIO deployment +- Configure an Alert Rule on a MinIO Metric to trigger an AlertManager action .. admonition:: Prerequisites :class: note @@ -32,315 +32,136 @@ The procedure on this page documents the following: - An :mc:`mc` installation on your local host configured to :ref:`access ` the MinIO deployment -Configure Prometheus to Collect and Alert using MinIO Metrics -------------------------------------------------------------- - -1) Generate the Scrape Configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Use the :mc:`mc admin prometheus generate` command to generate the scrape configuration for use by Prometheus in making scraping requests: - -.. tab-set:: - - .. tab-item:: MinIO Server - - The following command scrapes metrics for the MinIO cluster. - - .. code-block:: shell - :class: copyable - - mc admin prometheus generate ALIAS - - Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the MinIO deployment. - - The command returns output similar to the following: - - .. code-block:: yaml - :class: copyable - - global: - scrape_interval: 60s - - scrape_configs: - - job_name: minio-job - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/cluster - scheme: https - static_configs: - - targets: [minio.example.net] - - .. tab-item:: Nodes - - The following command scrapes metrics for a node on the MinIO Server. - - .. code-block:: shell - :class: copyable - - mc admin prometheus generate ALIAS node - - Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the MinIO deployment. - - .. code-block:: yaml - :class: copyable - - global: - scrape_interval: 60s - - scrape_configs: - - job_name: minio-job-node - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/node - scheme: https - static_configs: - - targets: [minio-1.example.net, minio-2.example.net, minio-N.example.net] - - .. tab-item:: Buckets - - The following command scrapes metrics for buckets on the MinIO Server. - - .. code-block:: shell - :class: copyable - - mc admin prometheus generate ALIAS bucket - - Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the MinIO deployment. - - .. code-block:: yaml - :class: copyable - - global: - scrape_interval: 60s - - scrape_configs: - - job_name: minio-job-bucket - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/bucket - scheme: https - static_configs: - - targets: [minio.example.net] - - .. tab-item:: Resources - - .. versionadded:: RELEASE.2023-10-07T15-07-38Z - - The following command scrapes metrics for resources on the MinIO Server. - - .. code-block:: shell - :class: copyable - - mc admin prometheus generate ALIAS resource - - Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the MinIO deployment. - - .. code-block:: yaml - :class: copyable - - global: - scrape_interval: 60s - - scrape_configs: - - job_name: minio-job-resource - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/resource - scheme: https - static_configs: - - targets: [minio.example.net] - -- Set an appropriate ``scrape_interval`` value to ensure each scraping operation completes before the next one begins. - The recommended value is 60 seconds. +.. admonition:: Metrics Version 2 Deprecated + :class: note - Some deployments require a longer scrape interval due to the number of metrics being scraped. - To reduce the load on your MinIO and Prometheus servers, choose the longest interval that meets your monitoring requirements. + Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 replaces the deprecated :ref:`metrics version 2 `. -- Set the ``job_name`` to a value associated to the MinIO deployment. + MinIO recommends new monitoring configurations use :ref:`version 3 metrics `. - Use a unique value to ensure isolation of the deployment metrics from any others collected by that Prometheus service. +Collect and alert on metrics +---------------------------- -- MinIO deployments started with :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` set to ``"public"`` can omit the ``bearer_token`` field. +Generate the scrape configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Set the ``scheme`` to http for MinIO deployments not using TLS. +Use :mc-cmd:`mc admin prometheus generate --api-version v3 ` to generate a scrape configuration for each :ref:`type of metric ` you want to scrape with Prometheus. -- Set the ``targets`` array with a hostname that resolves to the MinIO deployment. +For example, the following command scrapes all version 3 audit metrics for the MinIO cluster: - This can be any single node, or a load balancer/proxy which handles connections to the MinIO nodes. +.. code-block:: shell + :class: copyable - .. cond:: k8s + mc admin prometheus generate ALIAS audit --api-version v3 - For Prometheus deployments in the same cluster as the MinIO Tenant, you can specify the service DNS name for the ``minio`` service. +Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the MinIO deployment. - For Prometheus deployments external to the cluster, you must specify an ingress or load balancer endpoint configured to route connections to and from the MinIO Tenant. +The command returns output similar to the following: -2) Restart Prometheus with the Updated Configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Append the desired ``scrape_configs`` job generated in the previous step to the configuration file: - -.. tab-set:: - - .. tab-item:: Cluster - - Cluster metrics aggregate node-level metrics and, where appropriate, attach labels to metrics for the originating node. - - .. code-block:: yaml - :class: copyable - - global: - scrape_interval: 60s - - scrape_configs: - - job_name: minio-job - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/cluster - scheme: https - static_configs: - - targets: [minio.example.net] - - - .. tab-item:: Nodes - - Node metrics are specific for node-level monitoring. You need to list all MinIO nodes for this configuration. - - .. code-block:: yaml - :class: copyable - - global: - scrape_interval: 60s - - scrape_configs: - - job_name: minio-job-node - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/node - scheme: https - static_configs: - - targets: [minio-1.example.net, minio-2.example.net, minio-N.example.net] - - - .. tab-item:: Bucket - - .. code-block:: yaml - :class: copyable - - global: - scrape_interval: 60s - - scrape_configs: - - job_name: minio-job-bucket - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/bucket - scheme: https - static_configs: - - targets: [minio.example.net] - - .. tab-item:: Resource - - .. code-block:: yaml - :class: copyable - - global: - scrape_interval: 60s - - scrape_configs: - - job_name: minio-job-resource - bearer_token: TOKEN - metrics_path: /minio/v2/metrics/resource - scheme: https - static_configs: - - targets: [minio.example.net] - -Start the Prometheus cluster using the configuration file: +.. code-block:: yaml + :class: copyable -.. code-block:: shell + scrape_configs: + - job_name: minio-job + bearer_token: TOKEN + metrics_path: /minio/metrics/v3 + scheme: https + static_configs: + - targets: [minio.example.net] + +To scrape multiple types of metrics, run :mc-cmd:`mc admin prometheus generate --api-version v3 ` for each type and add the ``job_name`` section to the ``scrape_configs`` in your Prometheus configuration. + +The following example scrapes audit and system metrics every 60 seconds: + +.. code-block:: yaml :class: copyable - prometheus --config.file=prometheus.yaml + global: + scrape_interval: 60s -3) Analyze Collected Metrics -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + scrape_configs: + - job_name: minio-job-audit + bearer_token: TOKEN + metrics_path: /minio/metrics/v3/audit + scheme: https + static_configs: + - targets: [minio.example.net] -Prometheus includes an :prometheus-docs:`expression browser `. -You can execute queries here to analyze the collected metrics. + - job_name: minio-job-system + bearer_token: TOKEN + metrics_path: /minio/metrics/v3/system + scheme: https + static_configs: + - targets: [minio.example.net] -.. tab-set:: +If needed, edit the generated configuration for your environment. +Common changes include: - .. tab-item:: Examples +- Set an appropriate ``scrape_interval`` value to ensure each scraping operation completes before the next one begins. + The recommended value is 60 seconds. - The following query examples return metrics collected by Prometheus every five minutes for a scrape job named ``minio-job``: + Some deployments require a longer scrape interval due to the number of metrics being scraped. + To reduce the load on your MinIO and Prometheus servers, choose the longest interval that meets your monitoring requirements. - .. code-block:: shell - :class: copyable + You can specify a ``scrape_interval`` for each job in its ``job_name`` section, or all jobs in a separate ``global`` section. - minio_node_drive_free_bytes{job-"minio-job"}[5m] - minio_node_drive_free_inodes{job-"minio-job"}[5m] +- Set the ``job_name`` to a value associated to the MinIO deployment. - minio_node_drive_latency_us{job-"minio-job"}[5m] + Use a unique value for each job to ensure isolation of the deployment metrics from any others collected by that Prometheus service. - minio_node_drive_offline_total{job-"minio-job"}[5m] - minio_node_drive_online_total{job-"minio-job"}[5m] +- MinIO deployments started with :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` set to ``"public"`` can omit the ``bearer_token`` field. - minio_node_drive_total{job-"minio-job"}[5m] +- Set the ``scheme`` to http for MinIO deployments not using TLS. - minio_node_drive_total_bytes{job-"minio-job"}[5m] - minio_node_drive_used_bytes{job-"minio-job"}[5m] +- Set the ``targets`` array with a hostname that resolves to the MinIO deployment. - minio_node_drive_errors_timeout{job-"minio-job"}[5m] - minio_node_drive_errors_availability{job-"minio-job"}[5m] + This can be any single node, or a load balancer/proxy which handles connections to the MinIO nodes. - minio_node_drive_io_waiting{job-"minio-job"}[5m] + .. cond:: k8s - .. tab-item:: Recommended Metrics + For Prometheus deployments in the same cluster as the MinIO Tenant, you can specify the service DNS name for the ``minio`` service. - MinIO recommends the following as a basic set of metrics to monitor. + For Prometheus deployments external to the cluster, you must specify an ingress or load balancer endpoint configured to route connections to and from the MinIO Tenant. - See :ref:`minio-metrics-and-alerts` for information about all available metrics. +Restart Prometheus with the updated configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - .. list-table:: - :header-rows: 1 - :widths: 40 60 - :width: 100% +Add the desired ``scrape_configs`` jobs to your Prometheus configuration file and start the Prometheus cluster: - * - Metric - - Description - - * - ``minio_node_drive_free_bytes`` - - Total storage available on a drive. +.. code-block:: shell + :class: copyable - * - ``minio_node_drive_free_inodes`` - - Total free inodes. + prometheus --config.file=prometheus.yaml - * - ``minio_node_drive_latency_us`` - - Average last minute latency in µs for drive API storage operations. - * - ``minio_node_drive_offline_total`` - - Total drives offline in this node. +Analyze collected metrics +~~~~~~~~~~~~~~~~~~~~~~~~~ - * - ``minio_node_drive_online_total`` - - Total drives online in this node. +Prometheus includes an :prometheus-docs:`expression browser `. +You can execute queries here to analyze the collected metrics. - * - ``minio_node_drive_total`` - - Total drives in this node. +The following query examples return metrics collected by Prometheus every five minutes for a scrape job named ``minio-job``: - * - ``minio_node_drive_total_bytes`` - - Total storage on a drive. +.. code-block:: shell + :class: copyable - * - ``minio_node_drive_used_bytes`` - - Total storage used on a drive. + minio_system_drive_used_bytes{job-"minio-job"}[5m] + minio_system_drive_used_inodes{job-"minio-job"}[5m] - * - ``minio_node_drive_errors_timeout`` - - Total number of drive timeout errors since server start. + minio_cluster_usage_buckets_total_bytes{job-"minio-job"}[5m] + minio_cluster_usage_buckets_objects_count{job-"minio-job"}[5m] - * - ``minio_node_drive_errors_availability`` - - Total number of drive I/O errors, permission denied and timeouts since server start. + minio_api_requests_total{job-"minio-job"}[5m] + minio_api_requests_errors_total{job-"minio-job"}[5m] - * - ``minio_node_drive_io_waiting`` - - Total number of I/O operations waiting on drive. -4) Configure an Alert Rule using MinIO Metrics -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Configure an alert rule +~~~~~~~~~~~~~~~~~~~~~~~ -You must configure :prometheus-docs:`Alert Rules ` on the Prometheus deployment to trigger alerts based on collected MinIO metrics. +To trigger alerts based on metrics, configure :prometheus-docs:`Alert Rules ` on the Prometheus deployment. -The following example alert rule files provide a baseline of alerts for a MinIO deployment. -You can modify or otherwise use these examples as guidance in building your own alerts. +The following example alert provides a baseline of alerts for a MinIO deployment. +You can modify or use these examples as guidance for building your own alerts. .. code-block:: yaml :class: copyable @@ -349,7 +170,7 @@ You can modify or otherwise use these examples as guidance in building your own - name: minio-alerts rules: - alert: NodesOffline - expr: avg_over_time(minio_cluster_nodes_offline_total{job="minio-job"}[5m]) > 0 + expr: avg_over_time(minio_cluster_health_nodes_offline_count{job="minio-job"}[5m]) > 0 for: 10m labels: severity: warn @@ -358,7 +179,7 @@ You can modify or otherwise use these examples as guidance in building your own description: "Node(s) in cluster {{ $labels.instance }} offline for more than 5 minutes" - alert: DisksOffline - expr: avg_over_time(minio_cluster_drive_offline_total{job="minio-job"}[5m]) > 0 + expr: avg_over_time(minio_system_drive_offline_count{job="minio-job"}[5m]) > 0 for: 10m labels: severity: warn @@ -378,5 +199,5 @@ Once triggered, Prometheus sends the alert to the configured AlertManager servic Dashboards ---------- -MinIO provides Grafana Dashboards to display metrics collected by Prometheus. +For v2 metrics, MinIO provides Grafana Dashboards to display metrics collected by Prometheus. For more information, see :ref:`minio-grafana` diff --git a/source/operations/monitoring/metrics-and-alerts.rst b/source/operations/monitoring/metrics-and-alerts.rst index 118180d4..15465632 100644 --- a/source/operations/monitoring/metrics-and-alerts.rst +++ b/source/operations/monitoring/metrics-and-alerts.rst @@ -17,10 +17,10 @@ Metrics and Alerts Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 replaces the deprecated :ref:`metrics version 2 `. -MinIO publishes cluster and node metrics using the :prometheus-docs:`Prometheus Data Model `. +MinIO publishes metrics using the :prometheus-docs:`Prometheus Data Model `. You can use any scraping tool to pull metrics data from MinIO for further analysis and alerting. -For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint by appending an additional path for each category. +For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint, optionally by appending an additional path for each category. For example, the following endpoint returns audit metrics: @@ -36,66 +36,80 @@ By default, MinIO requires authentication to scrape the metrics endpoints. To generate the needed bearer tokens, use :mc:`mc admin prometheus generate`. You can also disable metrics endpoint authentication by setting :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``. -MinIO provides the following scraping endpoints, relative to the base URL: +You can also access metrics using :mc-cmd:`mc admin prometheus metrics` and the metric type for the desired category. +For more information, see the :mc-cmd:`MinIO Admin Client reference `. + +MinIO provides the following types and scraping endpoints, relative to the base URL: .. list-table:: :header-rows: 1 - :widths: 30 70 + :widths: 25 25 50 :width: 100% * - Category + - Metric type - Path * - API + - ``api`` - ``/api/requests`` - + ``/bucket/api`` * - Audit + - ``audit`` - ``/audit`` * - Cluster + - ``cluster`` - ``/cluster/config`` - + ``/cluster/erasure-set`` - + ``/cluster/health`` - + ``/cluster/iam`` - + ``/cluster/usage/buckets`` - + ``/cluster/usage/objects`` * - Debug + - ``debug`` - ``/debug/go`` * - ILM + - ``ilm`` - ``/ilm`` * - Logger webhook + - ``logger`` - ``/logger/webhook`` * - Notification + - ``notification`` - ``/notification`` * - Replication + - ``replication`` - ``/replication`` - + ``/bucket/replication`` * - Scanner + - ``scanner`` - ``/scanner`` * - System + - ``system`` - ``/system/drive`` - + ``/system/memory`` - + ``/system/cpu`` - + ``/system/network/internode`` - + ``/system/process`` For a complete list of metrics for each endpoint, see :ref:`Available Metrics `. @@ -103,23 +117,17 @@ For a complete list of metrics for each endpoint, see :ref:`Available Metrics ` for visualizing collected metrics. -For more complete documentation on configuring a Prometheus-compatible data source for Grafana, see the :prometheus-docs:`Prometheus documentation on Grafana Support `. - .. _minio-metrics-and-alerts-available-metrics: Available Metrics diff --git a/source/operations/monitoring/metrics-v2-deprecated.rst b/source/operations/monitoring/metrics-v2-deprecated.rst index fb907566..058258c1 100644 --- a/source/operations/monitoring/metrics-v2-deprecated.rst +++ b/source/operations/monitoring/metrics-v2-deprecated.rst @@ -9,14 +9,17 @@ Metrics Version 2 .. contents:: Table of Contents :local: - :depth: 3 + :depth: 1 .. admonition:: Metrics Version 2 Deprecated :class: note Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, :ref:`metrics version 3 ` replaces the deprecated metrics version 2. -The following sections describe the deprecated endpoints and metrics. +The following sections describe the deprecated version 2 endpoints and metrics. + +Metrics version 2 endpoints +--------------------------- .. tab-set:: @@ -67,6 +70,304 @@ The following sections describe the deprecated endpoints and metrics. For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. +Configure Prometheus to Collect and Alert using MinIO Metrics +------------------------------------------------------------- + +1) Generate the Scrape Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc:`mc admin prometheus generate` command to generate the scrape configuration for use by Prometheus in making scraping requests: + +.. tab-set:: + + .. tab-item:: MinIO Server + + The following command scrapes metrics for the MinIO cluster. + + .. code-block:: shell + :class: copyable + + mc admin prometheus generate ALIAS + + Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the MinIO deployment. + + The command returns output similar to the following: + + .. code-block:: yaml + :class: copyable + + global: + scrape_interval: 60s + + scrape_configs: + - job_name: minio-job + bearer_token: TOKEN + metrics_path: /minio/v2/metrics/cluster + scheme: https + static_configs: + - targets: [minio.example.net] + + .. tab-item:: Nodes + + The following command scrapes metrics for a node on the MinIO Server. + + .. code-block:: shell + :class: copyable + + mc admin prometheus generate ALIAS node + + Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the MinIO deployment. + + .. code-block:: yaml + :class: copyable + + global: + scrape_interval: 60s + + scrape_configs: + - job_name: minio-job-node + bearer_token: TOKEN + metrics_path: /minio/v2/metrics/node + scheme: https + static_configs: + - targets: [minio-1.example.net, minio-2.example.net, minio-N.example.net] + + .. tab-item:: Buckets + + The following command scrapes metrics for buckets on the MinIO Server. + + .. code-block:: shell + :class: copyable + + mc admin prometheus generate ALIAS bucket + + Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the MinIO deployment. + + .. code-block:: yaml + :class: copyable + + global: + scrape_interval: 60s + + scrape_configs: + - job_name: minio-job-bucket + bearer_token: TOKEN + metrics_path: /minio/v2/metrics/bucket + scheme: https + static_configs: + - targets: [minio.example.net] + + .. tab-item:: Resources + + .. versionadded:: RELEASE.2023-10-07T15-07-38Z + + The following command scrapes metrics for resources on the MinIO Server. + + .. code-block:: shell + :class: copyable + + mc admin prometheus generate ALIAS resource + + Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the MinIO deployment. + + .. code-block:: yaml + :class: copyable + + global: + scrape_interval: 60s + + scrape_configs: + - job_name: minio-job-resource + bearer_token: TOKEN + metrics_path: /minio/v2/metrics/resource + scheme: https + static_configs: + - targets: [minio.example.net] + +- Set an appropriate ``scrape_interval`` value to ensure each scraping operation completes before the next one begins. + The recommended value is 60 seconds. + + Some deployments require a longer scrape interval due to the number of metrics being scraped. + To reduce the load on your MinIO and Prometheus servers, choose the longest interval that meets your monitoring requirements. + +- Set the ``job_name`` to a value associated to the MinIO deployment. + + Use a unique value to ensure isolation of the deployment metrics from any others collected by that Prometheus service. + +- MinIO deployments started with :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` set to ``"public"`` can omit the ``bearer_token`` field. + +- Set the ``scheme`` to http for MinIO deployments not using TLS. + +- Set the ``targets`` array with a hostname that resolves to the MinIO deployment. + + This can be any single node, or a load balancer/proxy which handles connections to the MinIO nodes. + + .. cond:: k8s + + For Prometheus deployments in the same cluster as the MinIO Tenant, you can specify the service DNS name for the ``minio`` service. + + For Prometheus deployments external to the cluster, you must specify an ingress or load balancer endpoint configured to route connections to and from the MinIO Tenant. + +2) Restart Prometheus with the Updated Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Append the desired ``scrape_configs`` job generated in the previous step to the configuration file: + +.. tab-set:: + + .. tab-item:: Cluster + + Cluster metrics aggregate node-level metrics and, where appropriate, attach labels to metrics for the originating node. + + .. code-block:: yaml + :class: copyable + + global: + scrape_interval: 60s + + scrape_configs: + - job_name: minio-job + bearer_token: TOKEN + metrics_path: /minio/v2/metrics/cluster + scheme: https + static_configs: + - targets: [minio.example.net] + + + .. tab-item:: Nodes + + Node metrics are specific for node-level monitoring. You need to list all MinIO nodes for this configuration. + + .. code-block:: yaml + :class: copyable + + global: + scrape_interval: 60s + + scrape_configs: + - job_name: minio-job-node + bearer_token: TOKEN + metrics_path: /minio/v2/metrics/node + scheme: https + static_configs: + - targets: [minio-1.example.net, minio-2.example.net, minio-N.example.net] + + + .. tab-item:: Bucket + + .. code-block:: yaml + :class: copyable + + global: + scrape_interval: 60s + + scrape_configs: + - job_name: minio-job-bucket + bearer_token: TOKEN + metrics_path: /minio/v2/metrics/bucket + scheme: https + static_configs: + - targets: [minio.example.net] + + .. tab-item:: Resource + + .. code-block:: yaml + :class: copyable + + global: + scrape_interval: 60s + + scrape_configs: + - job_name: minio-job-resource + bearer_token: TOKEN + metrics_path: /minio/v2/metrics/resource + scheme: https + static_configs: + - targets: [minio.example.net] + +Start the Prometheus cluster using the configuration file: + +.. code-block:: shell + :class: copyable + + prometheus --config.file=prometheus.yaml + +3) Analyze Collected Metrics +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Prometheus includes an :prometheus-docs:`expression browser `. +You can execute queries here to analyze the collected metrics. + + +The following query examples return metrics collected by Prometheus every five minutes for a scrape job named ``minio-job``: + +.. code-block:: shell + :class: copyable + + minio_node_drive_free_bytes{job-"minio-job"}[5m] + minio_node_drive_free_inodes{job-"minio-job"}[5m] + + minio_node_drive_latency_us{job-"minio-job"}[5m] + + minio_node_drive_offline_total{job-"minio-job"}[5m] + minio_node_drive_online_total{job-"minio-job"}[5m] + + minio_node_drive_total{job-"minio-job"}[5m] + + minio_node_drive_total_bytes{job-"minio-job"}[5m] + minio_node_drive_used_bytes{job-"minio-job"}[5m] + + minio_node_drive_errors_timeout{job-"minio-job"}[5m] + minio_node_drive_errors_availability{job-"minio-job"}[5m] + + minio_node_drive_io_waiting{job-"minio-job"}[5m] + + +4) Configure an Alert Rule using MinIO Metrics +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You must configure :prometheus-docs:`Alert Rules ` on the Prometheus deployment to trigger alerts based on collected MinIO metrics. + +The following example alert rule files provide a baseline of alerts for a MinIO deployment. +You can modify or otherwise use these examples as guidance in building your own alerts. + +.. code-block:: yaml + :class: copyable + + groups: + - name: minio-alerts + rules: + - alert: NodesOffline + expr: avg_over_time(minio_cluster_nodes_offline_total{job="minio-job"}[5m]) > 0 + for: 10m + labels: + severity: warn + annotations: + summary: "Node down in MinIO deployment" + description: "Node(s) in cluster {{ $labels.instance }} offline for more than 5 minutes" + + - alert: DisksOffline + expr: avg_over_time(minio_cluster_drive_offline_total{job="minio-job"}[5m]) > 0 + for: 10m + labels: + severity: warn + annotations: + summary: "Disks down in MinIO deployment" + description: "Disks(s) in cluster {{ $labels.instance }} offline for more than 5 minutes" + +In the Prometheus configuration, specify the path to the alert file in the ``rule_files`` key: + +.. code-block:: yaml + + rule_files: + - minio-alerting.yml + +Once triggered, Prometheus sends the alert to the configured AlertManager service. + + +Available metrics +----------------- + - :ref:`Cluster Metrics ` - :ref:`Bucket Metrics ` - :ref:`Resource Metrics ` diff --git a/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst b/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst index 5d837713..dcfaee03 100644 --- a/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst +++ b/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst @@ -10,6 +10,8 @@ .. mc:: mc admin prometheus generate +Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 replaces the deprecated :ref:`metrics version 2 `. + Description ----------- @@ -19,7 +21,7 @@ The :mc:`mc admin prometheus generate` command generates a metrics scraping conf .. end-mc-admin-prometheus-generate-desc -For more complete documentation on using MinIO with Prometheus, see :ref:`How to monitor MinIO server with Prometheus ` +For more complete documentation on using MinIO with Prometheus, see :ref:`How to monitor MinIO server with Prometheus ` and :ref:`minio-metrics-and-alerts`. .. admonition:: Use ``mc admin`` on MinIO Deployments Only :class: note @@ -32,12 +34,12 @@ For more complete documentation on using MinIO with Prometheus, see :ref:`How to .. tab-item:: EXAMPLE - The following command generates a Prometheus scrape configuration that collects bucket metrics from the deployment at :term:`alias` ``myminio``: + The following command generates a Prometheus scrape configuration that collects audit metrics from the deployment at :term:`alias` ``myminio``: .. code-block:: shell :class: copyable - mc admin prometheus generate myminio bucket + mc admin prometheus generate myminio audit --api_version v3 .. tab-item:: SYNTAX @@ -46,9 +48,11 @@ For more complete documentation on using MinIO with Prometheus, see :ref:`How to .. code-block:: shell :class: copyable - mc [GLOBALFLAGS] admin prometheus generate \ - ALIAS \ - [TYPE] + mc [GLOBALFLAGS] admin prometheus generate \ + ALIAS \ + [TYPE] \ + [--api_version v3] \ + [TYPE --bucket --api_version v3] .. include:: /includes/common-minio-mc.rst :start-after: start-minio-syntax @@ -63,23 +67,58 @@ Parameters The :mc:`alias ` of a configured MinIO deployment for which the command generates a Prometheus-compatible configuration file. +.. mc-cmd:: --api-version v3 + :optional: + + Generate a scrape configuration for metrics version 3. + Omit to generate a metrics version 2 configuration. + +.. mc-cmd:: --bucket + :optional: + + For v3 metric types that return bucket-level metrics, specify a bucket name. + Use with :mc-cmd:`~mc admin prometheus generate --api-version`. + + ``--bucket`` works for the following v3 metric types: + + - ``api`` + - ``replication`` + + The following example generates a configuration for API metrics from the bucket ``mybucket``: + + .. code-block:: shell + :class: copyable + + mc admin prometheus generate play api --bucket mybucket --api-version v3 + .. mc-cmd:: TYPE :optional: The type of metrics to scrape. - .. versionchanged:: RELEASE.2023-10-07T15-07-38Z + Valid values for metrics version 3 are: + + - ``api`` + - ``audit`` + - ``cluster`` + - ``debug`` + - ``ilm`` + - ``logger`` + - ``notification`` + - ``replication`` + - ``scanner`` + - ``system`` - ``resource`` metrics added + If not specified, a ``v3`` command returns all metrics. - Valid values are: + Valid values for metrics version 2 are: - ``bucket`` - ``cluster`` - ``node`` - ``resource`` - If not specified, the command returns cluster metrics. + If not specified, a ``v2`` command returns cluster metrics. Cluster metrics also include node metrics. Global Flags @@ -90,18 +129,18 @@ Global Flags :end-before: end-minio-mc-globals -Example -------- +Examples +-------- -Generate a scrape config for bucket metrics -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Generate a default metrics v3 config +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Use :mc-cmd:`mc admin prometheus generate` to generate a scrape configuration that collects bucket metrics for a MinIO deployment: +Use :mc-cmd:`mc admin prometheus generate --api-version v3` to generate a scrape configuration that collects all v3 metrics for a MinIO deployment: .. code-block:: shell :class: copyable - mc admin prometheus generate ALIAS bucket + mc admin prometheus generate ALIAS --api-version v3 - Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. @@ -110,9 +149,67 @@ The output resembles the following: .. code-block:: shell scrape_configs: - - job_name: minio-job-bucket + - job_name: minio-job bearer_token: [auth token] - metrics_path: /minio/v2/metrics/bucket + metrics_path: /minio/metrics/v3 scheme: http static_configs: - targets: ['localhost:9000'] + + +Generate a v3 bucket replication metrics config +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following example generates a scrape configuration for v3 replication metrics of bucket ``mybucket``: + +.. code-block:: shell + :class: copyable + + mc admin prometheus generate ALIAS replication --bucket mybucket --api-version v3 + +- Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. + +The output resembles the following: + +.. code-block:: shell + + scrape_configs: + - job_name: minio-job-replication + bearer_token: [auth token] + metrics_path: /minio/metrics/v3/bucket/replication/mybucket + scheme: https + static_configs: + - targets: [`localhost:9000`] + + +Generate a default metrics v2 config +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +By default, :mc-cmd:`mc admin prometheus generate` generates a scrape configuration for v2 cluster metrics: + +.. code-block:: shell + :class: copyable + + mc admin prometheus generate ALIAS + +- Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. + +The output resembles the following: + +.. code-block:: shell + + scrape_configs: + - job_name: minio-job + bearer_token: [auth token] + metrics_path: /minio/v2/metrics + scheme: http + static_configs: + - targets: ['localhost:9000'] + +To generate a configuration for another metric type, specify the type. +The following generates a scrape configuration for v2 bucket metrics: + +.. code-block:: shell + :class: copyable + + mc admin prometheus generate ALIAS bucket From 1ff838abed2f0d58c2fd5e162aab921ebe81e552 Mon Sep 17 00:00:00 2001 From: Andrea Longo Date: Fri, 13 Sep 2024 17:12:43 -0600 Subject: [PATCH 2/5] fix random build warning --- source/reference/minio-mc/mc-encrypt-set.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/source/reference/minio-mc/mc-encrypt-set.rst b/source/reference/minio-mc/mc-encrypt-set.rst index 67782e34..67e922be 100644 --- a/source/reference/minio-mc/mc-encrypt-set.rst +++ b/source/reference/minio-mc/mc-encrypt-set.rst @@ -145,7 +145,6 @@ server cannot support may result in undesired behavior. Setting or modifying the default server-side encryption settings does *not* automatically encrypt or decrypt the existing bucket contents. If the bucket contents *must* have consistent encryption, use the -:mc:`mc mv` mc with the :mc-cmd:`~mc mv --encrypt` or -:mc-cmd:`~mc mv --encrypt-key` arguments to manually modify the +:mc:`mc mv` mc with the :mc-cmd:`~mc mv --enc-c` argument to manually modify the encryption settings or encrypted state of the bucket contents *before* changing the bucket default. From 2e384342bf82c3a4813bd0568cce6396cdd35c5a Mon Sep 17 00:00:00 2001 From: Andrea Longo Date: Thu, 19 Sep 2024 15:10:20 -0600 Subject: [PATCH 3/5] rearrange more metrics v2 content --- source/operations/checklists/software.rst | 2 +- source/operations/monitoring.rst | 1 - ...collect-minio-metrics-using-prometheus.rst | 6 --- source/operations/monitoring/grafana.rst | 1 + .../monitoring/metrics-v2-deprecated.rst | 6 +++ .../mc-admin-prometheus-generate.rst | 39 ++++++++++++++++--- 6 files changed, 42 insertions(+), 13 deletions(-) diff --git a/source/operations/checklists/software.rst b/source/operations/checklists/software.rst index 2201329b..9c9703b9 100644 --- a/source/operations/checklists/software.rst +++ b/source/operations/checklists/software.rst @@ -59,7 +59,7 @@ MinIO Pre-requisites - :ref:`Prometheus ` or a Prometheus-compatible setup for monitoring and metrics * - :octicon:`circle` - - :ref:`Grafana configured ` for dashboards + - (optional) `Grafana `__ for custom dashboards * - :octicon:`circle` - (optional) :mc:`mc` installed on the local host system diff --git a/source/operations/monitoring.rst b/source/operations/monitoring.rst index 1575a93c..50f117e5 100644 --- a/source/operations/monitoring.rst +++ b/source/operations/monitoring.rst @@ -68,4 +68,3 @@ See :ref:`minio-healthcheck-api` for more information. /operations/monitoring/metrics-and-alerts /operations/monitoring/minio-logging /operations/monitoring/healthcheck-probe - /operations/monitoring/grafana diff --git a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst index ce2fbd95..086c46fe 100644 --- a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst +++ b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst @@ -195,9 +195,3 @@ In the Prometheus configuration, specify the path to the alert file in the ``rul - minio-alerting.yml Once triggered, Prometheus sends the alert to the configured AlertManager service. - -Dashboards ----------- - -For v2 metrics, MinIO provides Grafana Dashboards to display metrics collected by Prometheus. -For more information, see :ref:`minio-grafana` diff --git a/source/operations/monitoring/grafana.rst b/source/operations/monitoring/grafana.rst index 6e1821b3..39787c0a 100644 --- a/source/operations/monitoring/grafana.rst +++ b/source/operations/monitoring/grafana.rst @@ -1,3 +1,4 @@ +:orphan: .. _minio-grafana: =================================== diff --git a/source/operations/monitoring/metrics-v2-deprecated.rst b/source/operations/monitoring/metrics-v2-deprecated.rst index 058258c1..3f23b655 100644 --- a/source/operations/monitoring/metrics-v2-deprecated.rst +++ b/source/operations/monitoring/metrics-v2-deprecated.rst @@ -364,6 +364,12 @@ In the Prometheus configuration, specify the path to the alert file in the ``rul Once triggered, Prometheus sends the alert to the configured AlertManager service. +Dashboards +---------- + +For v2 metrics, MinIO provides Grafana Dashboards to display the metrics collected by Prometheus. +For more information, see :ref:`minio-grafana` + Available metrics ----------------- diff --git a/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst b/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst index dcfaee03..9c8809ef 100644 --- a/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst +++ b/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst @@ -70,8 +70,8 @@ Parameters .. mc-cmd:: --api-version v3 :optional: - Generate a scrape configuration for metrics version 3. - Omit to generate a metrics version 2 configuration. + Generate a scrape configuration for :ref:`metrics version 3 `. + Omit to generate a :ref:`metrics version 2 ` configuration. .. mc-cmd:: --bucket :optional: @@ -140,7 +140,7 @@ Use :mc-cmd:`mc admin prometheus generate --api-version v3` to generate a scrape .. code-block:: shell :class: copyable - mc admin prometheus generate ALIAS --api-version v3 + mc admin prometheus generate ALIAS --api-version v3 - Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. @@ -157,6 +157,33 @@ The output resembles the following: - targets: ['localhost:9000'] +Generate a v3 cluster metrics config +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use :mc-cmd:`mc admin prometheus generate --api-version v3` to generate a scrape configuration that collects v3 cluster metrics for a MinIO deployment: + +.. code-block:: shell + :class: copyable + + mc admin prometheus generate ALIAS cluster --api-version v3 + +- Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. + +The output resembles the following: + +.. code-block:: shell + + scrape_configs: + - job_name: minio-job-cluster + bearer_token: [auth token] + metrics_path: /minio/metrics/v3/cluster + scheme: http + static_configs: + - targets: ['localhost:9000'] + +To generate a configuration for a :mc-cmd:`different metric type `, replace ``cluster`` with the desired type. + + Generate a v3 bucket replication metrics config ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -181,6 +208,8 @@ The output resembles the following: static_configs: - targets: [`localhost:9000`] +To generate a configuration for API metrics for a bucket, replace ``replication`` with ``api``. + Generate a default metrics v2 config ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -190,7 +219,7 @@ By default, :mc-cmd:`mc admin prometheus generate` generates a scrape configurat .. code-block:: shell :class: copyable - mc admin prometheus generate ALIAS + mc admin prometheus generate ALIAS - Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. @@ -212,4 +241,4 @@ The following generates a scrape configuration for v2 bucket metrics: .. code-block:: shell :class: copyable - mc admin prometheus generate ALIAS bucket + mc admin prometheus generate ALIAS bucket From e6c442e909985f9f14ca35bbcfc5e24117a88d0a Mon Sep 17 00:00:00 2001 From: Andrea Longo Date: Fri, 20 Sep 2024 16:27:48 -0600 Subject: [PATCH 4/5] partially address feedback, plus other edits --- source/operations/checklists/software.rst | 3 - ...collect-minio-metrics-using-prometheus.rst | 1 - .../monitoring/metrics-and-alerts.rst | 10 +- .../monitoring/metrics-v2-deprecated.rst | 2 +- .../mc-admin-prometheus-generate.rst | 57 +++-- .../mc-admin-prometheus-metrics.rst | 207 ++++++++---------- 6 files changed, 144 insertions(+), 136 deletions(-) diff --git a/source/operations/checklists/software.rst b/source/operations/checklists/software.rst index 9c9703b9..d3eee548 100644 --- a/source/operations/checklists/software.rst +++ b/source/operations/checklists/software.rst @@ -58,9 +58,6 @@ MinIO Pre-requisites * - :octicon:`circle` - :ref:`Prometheus ` or a Prometheus-compatible setup for monitoring and metrics - * - :octicon:`circle` - - (optional) `Grafana `__ for custom dashboards - * - :octicon:`circle` - (optional) :mc:`mc` installed on the local host system diff --git a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst index 086c46fe..58011ed2 100644 --- a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst +++ b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst @@ -37,7 +37,6 @@ The procedure on this page documents the following: Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 replaces the deprecated :ref:`metrics version 2 `. - MinIO recommends new monitoring configurations use :ref:`version 3 metrics `. Collect and alert on metrics ---------------------------- diff --git a/source/operations/monitoring/metrics-and-alerts.rst b/source/operations/monitoring/metrics-and-alerts.rst index 15465632..eaae840a 100644 --- a/source/operations/monitoring/metrics-and-alerts.rst +++ b/source/operations/monitoring/metrics-and-alerts.rst @@ -15,12 +15,12 @@ Metrics and Alerts .. admonition:: Metrics Version 2 Deprecated :class: note - Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 replaces the deprecated :ref:`metrics version 2 `. + Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 (v3) replaces the deprecated :ref:`metrics version 2 (v2) `. MinIO publishes metrics using the :prometheus-docs:`Prometheus Data Model `. You can use any scraping tool to pull metrics data from MinIO for further analysis and alerting. -For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint, optionally by appending an additional path for each category. +For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint, optionally appending an additional path for each category. For example, the following endpoint returns audit metrics: @@ -36,9 +36,6 @@ By default, MinIO requires authentication to scrape the metrics endpoints. To generate the needed bearer tokens, use :mc:`mc admin prometheus generate`. You can also disable metrics endpoint authentication by setting :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``. -You can also access metrics using :mc-cmd:`mc admin prometheus metrics` and the metric type for the desired category. -For more information, see the :mc-cmd:`MinIO Admin Client reference `. - MinIO provides the following types and scraping endpoints, relative to the base URL: .. list-table:: @@ -114,6 +111,9 @@ MinIO provides the following types and scraping endpoints, relative to the base For a complete list of metrics for each endpoint, see :ref:`Available Metrics `. +You can also access metrics using :mc-cmd:`mc admin prometheus metrics` and the metric type for the desired category. +For more information, see the :mc-cmd:`MinIO Admin Client reference `. + .. cond:: k8s The MinIO Operator supports deploying a per-tenant Prometheus instance configured to support metrics and visualization. diff --git a/source/operations/monitoring/metrics-v2-deprecated.rst b/source/operations/monitoring/metrics-v2-deprecated.rst index 3f23b655..3d02b941 100644 --- a/source/operations/monitoring/metrics-v2-deprecated.rst +++ b/source/operations/monitoring/metrics-v2-deprecated.rst @@ -14,7 +14,7 @@ Metrics Version 2 .. admonition:: Metrics Version 2 Deprecated :class: note - Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, :ref:`metrics version 3 ` replaces the deprecated metrics version 2. + Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics :ref:`version 3 (v3) ` replaces the deprecated metrics version 2 (v2). The following sections describe the deprecated version 2 endpoints and metrics. diff --git a/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst b/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst index 9c8809ef..a1ce4dad 100644 --- a/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst +++ b/source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst @@ -10,7 +10,7 @@ .. mc:: mc admin prometheus generate -Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 replaces the deprecated :ref:`metrics version 2 `. +Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics :ref:`version 3 (v3) ` replaces the deprecated :ref:`version 2 (v2) `. Description ----------- @@ -51,8 +51,8 @@ For more complete documentation on using MinIO with Prometheus, see :ref:`How to mc [GLOBALFLAGS] admin prometheus generate \ ALIAS \ [TYPE] \ - [--api_version v3] \ - [TYPE --bucket --api_version v3] + [--api_version v3] \ + [TYPE --bucket --api_version v3] .. include:: /includes/common-minio-mc.rst :start-after: start-minio-syntax @@ -67,30 +67,32 @@ Parameters The :mc:`alias ` of a configured MinIO deployment for which the command generates a Prometheus-compatible configuration file. -.. mc-cmd:: --api-version v3 +.. mc-cmd:: --api-version :optional: - Generate a scrape configuration for :ref:`metrics version 3 `. - Omit to generate a :ref:`metrics version 2 ` configuration. + To generate a scrape configuration for :ref:`v3 metrics `, include an ``--api-version v3`` parameter. + ``v3`` is the only accepted value. + + Omit ``--api-version`` to generate a :ref:`v2 metrics ` configuration. .. mc-cmd:: --bucket :optional: For v3 metric types that return bucket-level metrics, specify a bucket name. Use with :mc-cmd:`~mc admin prometheus generate --api-version`. - + ``--bucket`` works for the following v3 metric types: - ``api`` - ``replication`` The following example generates a configuration for API metrics from the bucket ``mybucket``: - + .. code-block:: shell :class: copyable - mc admin prometheus generate play api --bucket mybucket --api-version v3 - + mc admin prometheus generate ALIAS api --bucket mybucket --api-version v3 + .. mc-cmd:: TYPE :optional: @@ -119,7 +121,7 @@ Parameters - ``resource`` If not specified, a ``v2`` command returns cluster metrics. - Cluster metrics also include node metrics. + Cluster metrics also include rollups of certain node metrics. Global Flags ~~~~~~~~~~~~ @@ -135,7 +137,7 @@ Examples Generate a default metrics v3 config ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Use :mc-cmd:`mc admin prometheus generate --api-version v3` to generate a scrape configuration that collects all v3 metrics for a MinIO deployment: +Use :mc-cmd:`mc admin prometheus generate --api-version v3 ` to generate a scrape configuration that collects all v3 metrics for a MinIO deployment: .. code-block:: shell :class: copyable @@ -160,7 +162,7 @@ The output resembles the following: Generate a v3 cluster metrics config ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Use :mc-cmd:`mc admin prometheus generate --api-version v3` to generate a scrape configuration that collects v3 cluster metrics for a MinIO deployment: +Use :mc-cmd:`mc admin prometheus generate --api-version v3 ` to generate a scrape configuration that collects v3 cluster metrics for a MinIO deployment: .. code-block:: shell :class: copyable @@ -181,7 +183,7 @@ The output resembles the following: static_configs: - targets: ['localhost:9000'] -To generate a configuration for a :mc-cmd:`different metric type `, replace ``cluster`` with the desired type. +To generate a configuration for a :mc-cmd:`different metric type `, replace ``cluster`` with the desired type. Generate a v3 bucket replication metrics config @@ -208,9 +210,32 @@ The output resembles the following: static_configs: - targets: [`localhost:9000`] -To generate a configuration for API metrics for a bucket, replace ``replication`` with ``api``. - +Generate a v3 config for bucket API metrics +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following example generates a scrape configuration for v3 API metrics for bucket ``mybucket``: + +.. code-block:: shell + :class: copyable + + mc admin prometheus generate ALIAS api --bucket mybucket --api-version v3 + +- Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. + +The output resembles the following: + +.. code-block:: shell + + scrape_configs: + - job_name: minio-job-api + bearer_token: [auth token] + metrics_path: /minio/metrics/v3/bucket/api/mybucket + scheme: https + static_configs: + - targets: [`localhost:9000`] + + Generate a default metrics v2 config ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/reference/minio-mc-admin/mc-admin-prometheus-metrics.rst b/source/reference/minio-mc-admin/mc-admin-prometheus-metrics.rst index eca5c2ee..db957ee7 100644 --- a/source/reference/minio-mc-admin/mc-admin-prometheus-metrics.rst +++ b/source/reference/minio-mc-admin/mc-admin-prometheus-metrics.rst @@ -10,6 +10,8 @@ .. mc:: mc admin prometheus metrics +Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics :ref:`version 3 (v3) ` replaces the deprecated :ref:`version 2 (v2) `. + Description ----------- @@ -19,6 +21,8 @@ The :mc:`mc admin prometheus metrics` command prints Prometheus metrics for a cl .. end-mc-admin-prometheus-metrics-desc +The output includes additional information about each metric, such as if its value is a ``counter`` or ``gauge``. + For more complete documentation on using MinIO with Prometheus, see :ref:`How to monitor MinIO server with Prometheus ` .. admonition:: Use ``mc admin`` on MinIO Deployments Only @@ -37,7 +41,7 @@ For more complete documentation on using MinIO with Prometheus, see :ref:`How to .. code-block:: shell :class: copyable - mc admin prometheus metrics myminio cluster + mc admin prometheus metrics play cluster --api-version v3 .. tab-item:: SYNTAX @@ -47,8 +51,10 @@ For more complete documentation on using MinIO with Prometheus, see :ref:`How to :class: copyable mc [GLOBALFLAGS] admin prometheus metrics \ - ALIAS \ - [TYPE] + ALIAS \ + [TYPE] \ + [--api_version v3] \ + [TYPE --bucket --api_version v3] .. include:: /includes/common-minio-mc.rst :start-after: start-minio-syntax @@ -63,25 +69,62 @@ Parameters The :mc:`alias ` of a configured MinIO deployment for which the command prints metrics. +.. mc-cmd:: --api-version + :optional: + + To print :ref:`version 3 (v3) ` metrics, include an ``--api-version v3`` parameter. + ``v3`` is the only accepted value. + + Omit ``--api-version`` to print :ref:`version 2 (v2) ` metrics. + +.. mc-cmd:: --bucket + :optional: + + For v3 metric types that return bucket-level metrics, specify a bucket name. + Use with :mc-cmd:`~mc admin prometheus metrics --api-version`. + + ``--bucket`` works for the following v3 metric types: + + - ``api`` + - ``replication`` + + The following example prints API metrics for the bucket ``mybucket``: + + .. code-block:: shell + :class: copyable + + mc admin prometheus metrics ALIAS api --bucket mybucket --api-version v3 + .. mc-cmd:: TYPE :optional: The type of metrics to print. - .. versionchanged:: RELEASE.2023-10-07T15-07-38Z + Valid values for metrics version 3 are: + + - ``api`` + - ``audit`` + - ``cluster`` + - ``debug`` + - ``ilm`` + - ``logger`` + - ``notification`` + - ``replication`` + - ``scanner`` + - ``system`` - ``resource`` metrics added + If not specified, a ``v3`` command returns all metrics. - Valid values are: + Valid values for metrics version 2 are: - ``bucket`` - ``cluster`` - ``node`` - ``resource`` - If not specified, the command returns cluster metrics. - Cluster metrics include rollups of certain node metrics. - The output includes additional information about each metric, such as if its value is a ``counter`` or ``gauge``. + If not specified, a ``v2`` command returns cluster metrics. + Cluster metrics also include rollups of certain node metrics. + Global Flags ~~~~~~~~~~~~ @@ -94,115 +137,59 @@ Global Flags Example ------- -Print bucket metrics -~~~~~~~~~~~~~~~~~~~~ +Print v3 metrics +~~~~~~~~~~~~~~~~ + +Use :mc-cmd:`mc admin prometheus metrics --api-version v3 ` to print all available v3 metrics and their current values for a MinIO deployment: + +.. code-block:: shell + :class: copyable + + mc admin prometheus metrics ALIAS --api-version v3 + +- Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. -Use :mc-cmd:`mc admin prometheus metrics` to print bucket metrics for a MinIO deployment: +To print a specific type of metrics, include the :mc-cmd:`~mc admin prometheus metrics TYPE`. +The following prints all scanner metrics for a deployment: .. code-block:: shell :class: copyable - mc admin prometheus metrics ALIAS bucket + mc admin prometheus metrics ALIAS scanner --api-version v3 + + +Print v3 bucket replication metrics +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Certain metric types accept a :mc-cmd:`~mc admin prometheus metrics --bucket` parameter to specify the bucket for which to print metrics. +The following example prints v3 replication metrics for bucket ``mybucket``: + +.. code-block:: shell + :class: copyable + + mc admin prometheus metrics ALIAS replication --bucket mybucket --api-version v3 - Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. -The output resembles the following: +To print API metrics for a bucket, replace ``replication`` with ``api``. + + +Print v2 metrics +~~~~~~~~~~~~~~~~ + +By default, :mc-cmd:`mc admin prometheus metrics` prints v2 cluster metrics: + +.. code-block:: shell + :class: copyable + + mc admin prometheus metrics ALIAS + +- Replace ``ALIAS`` with the :mc-cmd:`alias ` of the MinIO deployment. + +To print another type of metrics, specify the desired :mc-cmd:`~mc admin prometheus metrics TYPE`. +The following example prints v2 bucket metrics: .. code-block:: shell + :class: copyable - # HELP minio_bucket_objects_size_distribution Distribution of object sizes in the bucket, includes label for the bucket name - # TYPE minio_bucket_objects_size_distribution gauge - minio_bucket_objects_size_distribution{bucket="mybucket",range="BETWEEN_1024B_AND_1_MB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="BETWEEN_1024_B_AND_64_KB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="BETWEEN_10_MB_AND_64_MB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="BETWEEN_128_MB_AND_512_MB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="BETWEEN_1_MB_AND_10_MB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="BETWEEN_256_KB_AND_512_KB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="BETWEEN_512_KB_AND_1_MB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="BETWEEN_64_KB_AND_256_KB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="BETWEEN_64_MB_AND_128_MB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="GREATER_THAN_512_MB",server="127.0.0.1:9000"} 0 - minio_bucket_objects_size_distribution{bucket="mybucket",range="LESS_THAN_1024_B",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_objects_version_distribution Distribution of object sizes in the bucket, includes label for the bucket name - # TYPE minio_bucket_objects_version_distribution gauge - minio_bucket_objects_version_distribution{bucket="mybucket",range="BETWEEN_1000_AND_10000",server="127.0.0.1:9000"} 0 - minio_bucket_objects_version_distribution{bucket="mybucket",range="BETWEEN_100_AND_1000",server="127.0.0.1:9000"} 0 - minio_bucket_objects_version_distribution{bucket="mybucket",range="BETWEEN_10_AND_100",server="127.0.0.1:9000"} 0 - minio_bucket_objects_version_distribution{bucket="mybucket",range="BETWEEN_2_AND_10",server="127.0.0.1:9000"} 0 - minio_bucket_objects_version_distribution{bucket="mybucket",range="GREATER_THAN_10000",server="127.0.0.1:9000"} 0 - minio_bucket_objects_version_distribution{bucket="mybucket",range="SINGLE_VERSION",server="127.0.0.1:9000"} 0 - minio_bucket_objects_version_distribution{bucket="mybucket",range="UNVERSIONED",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_delete_tagging_requests_failures Number of failures in DELETE tagging proxy requests to replication target - # TYPE minio_bucket_replication_proxied_delete_tagging_requests_failures counter - minio_bucket_replication_proxied_delete_tagging_requests_failures{server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_delete_tagging_requests_total Number of DELETE tagging requests proxied to replication target - # TYPE minio_bucket_replication_proxied_delete_tagging_requests_total counter - minio_bucket_replication_proxied_delete_tagging_requests_total{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_get_requests_failures Number of failures in GET requests proxied to replication target - # TYPE minio_bucket_replication_proxied_get_requests_failures counter - minio_bucket_replication_proxied_get_requests_failures{server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_get_requests_total Number of GET requests proxied to replication target - # TYPE minio_bucket_replication_proxied_get_requests_total counter - minio_bucket_replication_proxied_get_requests_total{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_get_tagging_requests_failures Number of failures in GET tagging proxy requests to replication target - # TYPE minio_bucket_replication_proxied_get_tagging_requests_failures counter - minio_bucket_replication_proxied_get_tagging_requests_failures{server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_get_tagging_requests_total Number of GET tagging requests proxied to replication target - # TYPE minio_bucket_replication_proxied_get_tagging_requests_total counter - minio_bucket_replication_proxied_get_tagging_requests_total{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_head_requests_failures Number of failures in HEAD requests proxied to replication target - # TYPE minio_bucket_replication_proxied_head_requests_failures counter - minio_bucket_replication_proxied_head_requests_failures{server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_head_requests_total Number of HEAD requests proxied to replication target - # TYPE minio_bucket_replication_proxied_head_requests_total counter - minio_bucket_replication_proxied_head_requests_total{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_put_tagging_requests_failures Number of failures in PUT tagging proxy requests to replication target - # TYPE minio_bucket_replication_proxied_put_tagging_requests_failures counter - minio_bucket_replication_proxied_put_tagging_requests_failures{server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_proxied_put_tagging_requests_total Number of PUT tagging requests proxied to replication target - # TYPE minio_bucket_replication_proxied_put_tagging_requests_total counter - minio_bucket_replication_proxied_put_tagging_requests_total{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_received_bytes Total number of bytes replicated to this bucket from another source bucket - # TYPE minio_bucket_replication_received_bytes counter - minio_bucket_replication_received_bytes{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_replication_received_count Total number of objects received by this bucket from another source bucket - # TYPE minio_bucket_replication_received_count gauge - minio_bucket_replication_received_count{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_requests_4xx_errors_total Total number of S3 requests with (4xx) errors on a bucket - # TYPE minio_bucket_requests_4xx_errors_total counter - minio_bucket_requests_4xx_errors_total{api="getbucketobjectlockconfig",bucket="mybucket",server="127.0.0.1:9000"} 1 - # HELP minio_bucket_requests_inflight_total Total number of S3 requests currently in flight on a bucket - # TYPE minio_bucket_requests_inflight_total gauge - minio_bucket_requests_inflight_total{api="getbucketlocation",bucket="mybucket",server="127.0.0.1:9000"} 0 - minio_bucket_requests_inflight_total{api="getbucketobjectlockconfig",bucket="mybucket",server="127.0.0.1:9000"} 0 - minio_bucket_requests_inflight_total{api="headbucket",bucket="mybucket",server="127.0.0.1:9000"} 0 - minio_bucket_requests_inflight_total{api="listobjectsv2",bucket="mybucket",server="127.0.0.1:9000"} 0 - minio_bucket_requests_inflight_total{api="putobject",bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_requests_total Total number of S3 requests on a bucket - # TYPE minio_bucket_requests_total counter - minio_bucket_requests_total{api="getbucketlocation",bucket="mybucket",server="127.0.0.1:9000"} 2 - minio_bucket_requests_total{api="getbucketobjectlockconfig",bucket="mybucket",server="127.0.0.1:9000"} 1 - minio_bucket_requests_total{api="headbucket",bucket="mybucket",server="127.0.0.1:9000"} 2 - minio_bucket_requests_total{api="listobjectsv2",bucket="mybucket",server="127.0.0.1:9000"} 1 - minio_bucket_requests_total{api="putobject",bucket="mybucket",server="127.0.0.1:9000"} 1 - # HELP minio_bucket_traffic_received_bytes Total number of S3 bytes received for this bucket - # TYPE minio_bucket_traffic_received_bytes gauge - minio_bucket_traffic_received_bytes{bucket="mybucket",server="127.0.0.1:9000"} 178 - # HELP minio_bucket_traffic_sent_bytes Total number of S3 bytes sent for this bucket - # TYPE minio_bucket_traffic_sent_bytes gauge - minio_bucket_traffic_sent_bytes{bucket="mybucket",server="127.0.0.1:9000"} 1232 - # HELP minio_bucket_usage_deletemarker_total Total number of delete markers - # TYPE minio_bucket_usage_deletemarker_total gauge - minio_bucket_usage_deletemarker_total{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_usage_object_total Total number of objects - # TYPE minio_bucket_usage_object_total gauge - minio_bucket_usage_object_total{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_usage_total_bytes Total bucket size in bytes - # TYPE minio_bucket_usage_total_bytes gauge - minio_bucket_usage_total_bytes{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_bucket_usage_version_total Total number of versions (includes delete marker) - # TYPE minio_bucket_usage_version_total gauge - minio_bucket_usage_version_total{bucket="mybucket",server="127.0.0.1:9000"} 0 - # HELP minio_usage_last_activity_nano_seconds Time elapsed (in nano seconds) since last scan activity - # TYPE minio_usage_last_activity_nano_seconds gauge - minio_usage_last_activity_nano_seconds{server="127.0.0.1:9000"} 5.6046668864e+10 + mc admin prometheus metrics ALIAS bucket From a5bfd27b805bbe0305c1d350f099143dba5f81c3 Mon Sep 17 00:00:00 2001 From: Andrea Longo Date: Fri, 27 Sep 2024 14:50:06 -0600 Subject: [PATCH 5/5] partial edits --- ...collect-minio-metrics-using-prometheus.rst | 2 +- .../monitoring/metrics-and-alerts.rst | 170 +-------------- ...trics-v2-deprecated.rst => metrics-v2.rst} | 37 ++-- source/operations/monitoring/metrics-v3.rst | 197 ++++++++++++++++++ 4 files changed, 228 insertions(+), 178 deletions(-) rename source/operations/monitoring/{metrics-v2-deprecated.rst => metrics-v2.rst} (92%) create mode 100644 source/operations/monitoring/metrics-v3.rst diff --git a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst index 58011ed2..14941855 100644 --- a/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst +++ b/source/operations/monitoring/collect-minio-metrics-using-prometheus.rst @@ -102,7 +102,7 @@ Common changes include: Some deployments require a longer scrape interval due to the number of metrics being scraped. To reduce the load on your MinIO and Prometheus servers, choose the longest interval that meets your monitoring requirements. - You can specify a ``scrape_interval`` for each job in its ``job_name`` section, or all jobs in a separate ``global`` section. + You can specify a global ``scrape_interval`` for all jobs or override a global value by setting a ``scrape_interval`` in an individual ``job_name`` section. - Set the ``job_name`` to a value associated to the MinIO deployment. diff --git a/source/operations/monitoring/metrics-and-alerts.rst b/source/operations/monitoring/metrics-and-alerts.rst index eaae840a..8bddf6d6 100644 --- a/source/operations/monitoring/metrics-and-alerts.rst +++ b/source/operations/monitoring/metrics-and-alerts.rst @@ -12,105 +12,13 @@ Metrics and Alerts :local: :depth: 2 -.. admonition:: Metrics Version 2 Deprecated - :class: note - - Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 (v3) replaces the deprecated :ref:`metrics version 2 (v2) `. +.. versionadded:: RELEASE.2024-07-15T19-02-30Z + Metrics version 3 (v3) provides additional metrics and metric types for more targeted scraping. + MinIO publishes metrics using the :prometheus-docs:`Prometheus Data Model `. You can use any scraping tool to pull metrics data from MinIO for further analysis and alerting. -For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint, optionally appending an additional path for each category. - -For example, the following endpoint returns audit metrics: - -.. code-block:: shell - :class: copyable - - http://HOSTNAME:PORT/minio/metrics/v3/audit - -Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment. -For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. - -By default, MinIO requires authentication to scrape the metrics endpoints. -To generate the needed bearer tokens, use :mc:`mc admin prometheus generate`. -You can also disable metrics endpoint authentication by setting :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``. - -MinIO provides the following types and scraping endpoints, relative to the base URL: - -.. list-table:: - :header-rows: 1 - :widths: 25 25 50 - :width: 100% - - * - Category - - Metric type - - Path - - * - API - - ``api`` - - ``/api/requests`` - - ``/bucket/api`` - - * - Audit - - ``audit`` - - ``/audit`` - - * - Cluster - - ``cluster`` - - ``/cluster/config`` - - ``/cluster/erasure-set`` - - ``/cluster/health`` - - ``/cluster/iam`` - - ``/cluster/usage/buckets`` - - ``/cluster/usage/objects`` - - * - Debug - - ``debug`` - - ``/debug/go`` - - * - ILM - - ``ilm`` - - ``/ilm`` - - * - Logger webhook - - ``logger`` - - ``/logger/webhook`` - - * - Notification - - ``notification`` - - ``/notification`` - - * - Replication - - ``replication`` - - ``/replication`` - - ``/bucket/replication`` - - * - Scanner - - ``scanner`` - - ``/scanner`` - - * - System - - ``system`` - - ``/system/drive`` - - ``/system/memory`` - - ``/system/cpu`` - - ``/system/network/internode`` - - ``/system/process`` - -For a complete list of metrics for each endpoint, see :ref:`Available Metrics `. - You can also access metrics using :mc-cmd:`mc admin prometheus metrics` and the metric type for the desired category. For more information, see the :mc-cmd:`MinIO Admin Client reference `. @@ -128,80 +36,18 @@ For more information, see the :mc-cmd:`MinIO Admin Client reference ` -- :ref:`Audit Metrics ` -- :ref:`Cluster Metrics ` -- :ref:`Debug Metrics ` -- :ref:`ILM Metrics ` -- :ref:`Logger webhook Metrics ` -- :ref:`Notification Metrics ` -- :ref:`Replication Metrics ` -- :ref:`Scanner Metrics ` -- :ref:`System Metrics ` - - -.. _minio-available-v3-api-metrics: - -.. include:: /includes/common-metrics-v3-api.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-audit-metrics: - -.. include:: /includes/common-metrics-v3-audit.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-cluster-metrics: -.. include:: /includes/common-metrics-v3-cluster.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-debug-metrics: - -.. include:: /includes/common-metrics-v3-debug.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-ilm-metrics: - -.. include:: /includes/common-metrics-v3-ilm.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-logger-webhook-metrics: - -.. include:: /includes/common-metrics-v3-logger-webhook.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-notification-metrics: - -.. include:: /includes/common-metrics-v3-notification.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-replication-metrics: - -.. include:: /includes/common-metrics-v3-replication.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-scanner-metrics: - -.. include:: /includes/common-metrics-v3-scanner.md - :parser: myst_parser.sphinx_ - -.. _minio-available-v3-system-metrics: +.. _minio-metrics-and-alerts-available-metrics: -.. include:: /includes/common-metrics-v3-system.md - :parser: myst_parser.sphinx_ +v3 metrics +v2 metrics .. toctree:: :titlesonly: :hidden: + /operations/monitoring/metrics-v3 + /operations/monitoring/metrics-v2 /operations/monitoring/collect-minio-metrics-using-prometheus /operations/monitoring/monitor-and-alert-using-influxdb diff --git a/source/operations/monitoring/metrics-v2-deprecated.rst b/source/operations/monitoring/metrics-v2.rst similarity index 92% rename from source/operations/monitoring/metrics-v2-deprecated.rst rename to source/operations/monitoring/metrics-v2.rst index 3d02b941..6501d25c 100644 --- a/source/operations/monitoring/metrics-v2-deprecated.rst +++ b/source/operations/monitoring/metrics-v2.rst @@ -1,4 +1,3 @@ -:orphan: .. _minio-metrics-v2: ================= @@ -11,15 +10,22 @@ Metrics Version 2 :local: :depth: 1 -.. admonition:: Metrics Version 2 Deprecated - :class: note - Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics :ref:`version 3 (v3) ` replaces the deprecated metrics version 2 (v2). +Metrics version 2 +----------------- + +For metrics version 2, all metrics are available under the base ``/minio/v2/metrics`` endpoint, optionally appending an additional path for each category. + +For example, the following endpoint returns bucket metrics: + +.. code-block:: shell + :class: copyable -The following sections describe the deprecated version 2 endpoints and metrics. + http://HOSTNAME:PORT/minio/v2/metrics/bucket + +Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment. +For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. -Metrics version 2 endpoints ---------------------------- .. tab-set:: @@ -70,11 +76,11 @@ Metrics version 2 endpoints For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. -Configure Prometheus to Collect and Alert using MinIO Metrics +Configure Prometheus to collect and alert using MinIO Metrics ------------------------------------------------------------- -1) Generate the Scrape Configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +1) Generate a v2 scrape configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use the :mc:`mc admin prometheus generate` command to generate the scrape configuration for use by Prometheus in making scraping requests: @@ -208,7 +214,7 @@ Use the :mc:`mc admin prometheus generate` command to generate the scrape config For Prometheus deployments external to the cluster, you must specify an ingress or load balancer endpoint configured to route connections to and from the MinIO Tenant. -2) Restart Prometheus with the Updated Configuration +2) Restart Prometheus with the updated configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Append the desired ``scrape_configs`` job generated in the previous step to the configuration file: @@ -292,7 +298,7 @@ Start the Prometheus cluster using the configuration file: prometheus --config.file=prometheus.yaml -3) Analyze Collected Metrics +3) Analyze collected metrics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Prometheus includes an :prometheus-docs:`expression browser `. @@ -323,7 +329,7 @@ The following query examples return metrics collected by Prometheus every five m minio_node_drive_io_waiting{job-"minio-job"}[5m] -4) Configure an Alert Rule using MinIO Metrics +4) Configure an alert rule using MinIO Metrics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You must configure :prometheus-docs:`Alert Rules ` on the Prometheus deployment to trigger alerts based on collected MinIO metrics. @@ -370,9 +376,10 @@ Dashboards For v2 metrics, MinIO provides Grafana Dashboards to display the metrics collected by Prometheus. For more information, see :ref:`minio-grafana` +.. _minio-metrics-and-alerts-available-v2-metrics: -Available metrics ------------------ +Available v2 metrics +-------------------- - :ref:`Cluster Metrics ` - :ref:`Bucket Metrics ` diff --git a/source/operations/monitoring/metrics-v3.rst b/source/operations/monitoring/metrics-v3.rst new file mode 100644 index 00000000..2b006af0 --- /dev/null +++ b/source/operations/monitoring/metrics-v3.rst @@ -0,0 +1,197 @@ +.. _minio-metrics-v3: + +================= +Metrics version 3 +================= + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint, optionally appending an additional path for each category. + +For example, the following endpoint returns audit metrics: + +.. code-block:: shell + :class: copyable + + http://HOSTNAME:PORT/minio/metrics/v3/audit + +Replace ``HOSTNAME:PORT`` with the :abbr:`FQDN (Fully Qualified Domain Name)` and port of the MinIO deployment. +For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer. + +By default, MinIO requires authentication to scrape the metrics endpoints. +To generate the needed bearer tokens, use :mc:`mc admin prometheus generate`. +You can also disable metrics endpoint authentication by setting :envvar:`MINIO_PROMETHEUS_AUTH_TYPE` to ``public``. + +MinIO provides the following v3 types and scraping endpoints, relative to the base URL: + +.. list-table:: + :header-rows: 1 + :widths: 25 25 50 + :width: 100% + + * - Category + - Metric type + - Path + + * - API + - ``api`` + - ``/api/requests`` + + ``/bucket/api`` + + * - Audit + - ``audit`` + - ``/audit`` + + * - Cluster + - ``cluster`` + - ``/cluster/config`` + + ``/cluster/erasure-set`` + + ``/cluster/health`` + + ``/cluster/iam`` + + ``/cluster/usage/buckets`` + + ``/cluster/usage/objects`` + + * - Debug + - ``debug`` + - ``/debug/go`` + + * - ILM + - ``ilm`` + - ``/ilm`` + + * - Logger webhook + - ``logger`` + - ``/logger/webhook`` + + * - Notification + - ``notification`` + - ``/notification`` + + * - Replication + - ``replication`` + - ``/replication`` + + ``/bucket/replication`` + + * - Scanner + - ``scanner`` + - ``/scanner`` + + * - System + - ``system`` + - ``/system/drive`` + + ``/system/memory`` + + ``/system/cpu`` + + ``/system/network/internode`` + + ``/system/process`` + +For a complete list of metrics for each endpoint, see :ref:`Available Metrics `. + +You can also access metrics using :mc-cmd:`mc admin prometheus metrics` and the metric type for the desired category. +For more information, see the :mc-cmd:`MinIO Admin Client reference `. + +.. cond:: k8s + + The MinIO Operator supports deploying a per-tenant Prometheus instance configured to support metrics and visualization. + + If you deploy the Tenant with this feature disabled *but* still want the historical metric views, you can instead configure an external Prometheus service to scrape the Tenant metrics. + Once configured, you can update the Tenant to query that Prometheus service to retrieve metric data: + +.. cond:: linux or container or macos or windows + + To enable historical data visualization in MinIO Console, set the following environment variables on each node in the MinIO deployment: + +- Set :envvar:`MINIO_PROMETHEUS_URL` to the URL of the Prometheus service +- Set :envvar:`MINIO_PROMETHEUS_JOB_ID` to the unique job ID assigned to the collected metrics + +.. _minio-metrics-and-alerts-available-v3-metrics: + +Available v3 metrics +-------------------- + +MinIO publishes a number of metrics at the cluster, node, or bucket levels. +Each metric includes a label for the MinIO server which generated that metric. + +- :ref:`API Metrics ` +- :ref:`Audit Metrics ` +- :ref:`Cluster Metrics ` +- :ref:`Debug Metrics ` +- :ref:`ILM Metrics ` +- :ref:`Logger webhook Metrics ` +- :ref:`Notification Metrics ` +- :ref:`Replication Metrics ` +- :ref:`Scanner Metrics ` +- :ref:`System Metrics ` + + +.. _minio-available-v3-api-metrics: + +.. include:: /includes/common-metrics-v3-api.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-audit-metrics: + +.. include:: /includes/common-metrics-v3-audit.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-cluster-metrics: + +.. include:: /includes/common-metrics-v3-cluster.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-debug-metrics: + +.. include:: /includes/common-metrics-v3-debug.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-ilm-metrics: + +.. include:: /includes/common-metrics-v3-ilm.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-logger-webhook-metrics: + +.. include:: /includes/common-metrics-v3-logger-webhook.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-notification-metrics: + +.. include:: /includes/common-metrics-v3-notification.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-replication-metrics: + +.. include:: /includes/common-metrics-v3-replication.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-scanner-metrics: + +.. include:: /includes/common-metrics-v3-scanner.md + :parser: myst_parser.sphinx_ + +.. _minio-available-v3-system-metrics: + +.. include:: /includes/common-metrics-v3-system.md + :parser: myst_parser.sphinx_ + + +.. toctree:: + :titlesonly: + :hidden: + + /operations/monitoring/collect-minio-metrics-using-prometheus + /operations/monitoring/monitor-and-alert-using-influxdb