V2 sys metrics (#565)

* Add assertion for sys metrics

* Lint

* Move system metrics to its own oteltest

* Lint

* Revert changes to ott spec file

* Add oteltests of system metrics instrumentor.
Remove support for OTEL_METRICS_ENABLED 1.x env var.

* Lint

RELEASE_NOTES.md

@@ -19,20 +19,20 @@ Summary of changes between Splunk OTel Python major versions 1 and 2.
 
 ## Environment Variables
 
-| Variable                              | v2 default | v1 -> v2 changes                                               | Description                                                                                                        |
-|---------------------------------------|------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
-| SPLUNK_ACCESS_TOKEN                   |            | None                                                           | Adds token to requests to enable direct ingest (for skipping the collector)                                        |
-| OTEL_METRICS_ENABLED                  | true       | None                                                           | Causes metrics to be configured (with an otlp-grpc metric exporter and a SystemMetricInstrumentor)                 |
-| OTEL_METRICS_EXPORTER                 | otlp       | Hard coded in v1, configurable in v2                           | Indicates the metrics exporter                                                                                     |
-| OTEL_TRACE_ENABLED                    | true       | None                                                           | Causes tracing to be configured and instrumentors loaded                                                           |
-| OTEL_TRACES_EXPORTER                  | otlp       | None                                                           | Indicates the traces exporter                                                                                      |
-| OTEL_PYTHON_DISABLED_INSTRUMENTATIONS |            | None                                                           | Disables instrumentations by entrypoint name                                                                       |
-| SPLUNK_PROFILER_ENABLED               | false      | None                                                           | Causes the Splunk profiler to start polling at startup                                                             |
-| OTEL_SPAN_LINK_COUNT_LIMIT            | 1000       | None                                                           | Sets the maximum allowed span link count                                                                           |
-| OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT     | 12000      | None                                                           | Sets the maximum allowed attribute value size                                                                      |
-| SPLUNK_TRACE_RESPONSE_HEADER_ENABLED  | true       | None                                                           | Causes a ServerTimingReponsePropagator to be configured if true (injects tracecontext headers into HTTP responses) |
-| OTEL_EXPERIMENTAL_RESOURCE_DETECTORS  | host       | Not set in v1                                                  | Causes a host resource detector to be configured to set telemetry attributes                                       |
-| OTEL_TRACES_SAMPLER                   | always_on  | Not set in v1 (took upstream default of parentbased_always_on) |                                                                                                                    |
+| Variable                              | v2 default | v1 -> v2 changes                                                                | Description                                                                                                        |
+|---------------------------------------|------------|---------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
+| SPLUNK_ACCESS_TOKEN                   |            | None                                                                            | Adds token to requests to enable direct ingest (for skipping the collector)                                        |
+| OTEL_METRICS_ENABLED                  | [n/a]      | Removed. Set `OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=system_metrics` to disable. | Caused metrics to be configured (with an otlp-grpc metric exporter and a SystemMetricInstrumentor)                 |
+| OTEL_METRICS_EXPORTER                 | otlp       | Hard coded in v1, configurable in v2                                            | Indicates the metrics exporter                                                                                     |
+| OTEL_TRACE_ENABLED                    | true       | None                                                                            | Causes tracing to be configured and instrumentors loaded                                                           |
+| OTEL_TRACES_EXPORTER                  | otlp       | None                                                                            | Indicates the traces exporter                                                                                      |
+| OTEL_PYTHON_DISABLED_INSTRUMENTATIONS |            | None                                                                            | Disables instrumentations by entrypoint name                                                                       |
+| SPLUNK_PROFILER_ENABLED               | false      | None                                                                            | Causes the Splunk profiler to start polling at startup                                                             |
+| OTEL_SPAN_LINK_COUNT_LIMIT            | 1000       | None                                                                            | Sets the maximum allowed span link count                                                                           |
+| OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT     | 12000      | None                                                                            | Sets the maximum allowed attribute value size                                                                      |
+| SPLUNK_TRACE_RESPONSE_HEADER_ENABLED  | true       | None                                                                            | Causes a ServerTimingReponsePropagator to be configured if true (injects tracecontext headers into HTTP responses) |
+| OTEL_EXPERIMENTAL_RESOURCE_DETECTORS  | host       | Not set in v1                                                                   | Causes a host resource detector to be configured to set telemetry attributes                                       |
+| OTEL_TRACES_SAMPLER                   | always_on  | Not set in v1 (took upstream default of parentbased_always_on)                  |                                                                                                                    |
 
 ## Auto-instrumentation
 

src/splunk_otel/distro.py

@@ -16,14 +16,15 @@
 
 from opentelemetry.instrumentation.distro import BaseDistro
 from opentelemetry.instrumentation.propagators import set_global_response_propagator
-from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor
-from opentelemetry.sdk.environment_variables import OTEL_EXPORTER_OTLP_HEADERS, OTEL_RESOURCE_ATTRIBUTES
+from opentelemetry.sdk.environment_variables import (
+    OTEL_EXPORTER_OTLP_HEADERS,
+    OTEL_RESOURCE_ATTRIBUTES,
+)
 
 from splunk_otel.__about__ import __version__ as version
 from splunk_otel.env import (
     DEFAULTS,
     OTEL_LOGS_ENABLED,
-    OTEL_METRICS_ENABLED,
     OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED,
     SPLUNK_ACCESS_TOKEN,
     SPLUNK_PROFILER_ENABLED,
@@ -74,18 +75,3 @@ def configure_headers(self):
     def set_server_timing_propagator(self):
         if self.env.is_true(SPLUNK_TRACE_RESPONSE_HEADER_ENABLED, "true"):
             set_global_response_propagator(ServerTimingResponsePropagator())
-
-    def load_instrumentor(self, entry_point, **kwargs):
-        #  This method is called in a loop by opentelemetry-instrumentation
-        if is_system_metrics_instrumentor(entry_point) and not self.env.is_true(OTEL_METRICS_ENABLED):
-            self.logger.info("%s not set -- skipping SystemMetricsInstrumentor", OTEL_METRICS_ENABLED)
-        else:
-            super().load_instrumentor(entry_point, **kwargs)
-
-
-def is_system_metrics_instrumentor(entry_point):
-    if entry_point.name == "system_metrics":
-        instrumentor_class = entry_point.load()
-        if instrumentor_class == SystemMetricsInstrumentor:
-            return True
-    return False

tests/ott_sysmetrics_disabled.py

@@ -0,0 +1,29 @@
+from ott_lib import project_path, trace_loop
+
+if __name__ == "__main__":
+    trace_loop(1)
+
+
+class SysMetricsOtelTest:
+    def requirements(self):
+        return (project_path(),)
+
+    def environment_variables(self):
+        return {
+            "OTEL_SERVICE_NAME": "my-svc",
+            "OTEL_PYTHON_DISABLED_INSTRUMENTATIONS": "system_metrics",
+        }
+
+    def wrapper_command(self):
+        return "opentelemetry-instrument"
+
+    def on_start(self):
+        return None
+
+    def on_stop(self, telemetry, stdout: str, stderr: str, returncode: int) -> None:
+        from oteltest.telemetry import get_metric_names
+
+        assert "system.cpu.time" not in get_metric_names(telemetry)
+
+    def is_http(self):
+        return False

tests/ott_sysmetrics_enabled.py

@@ -0,0 +1,28 @@
+from ott_lib import project_path, trace_loop
+
+if __name__ == "__main__":
+    trace_loop(1)
+
+
+class SysMetricsOtelTest:
+    def requirements(self):
+        return (project_path(),)
+
+    def environment_variables(self):
+        return {
+            "OTEL_SERVICE_NAME": "my-svc",
+        }
+
+    def wrapper_command(self):
+        return "opentelemetry-instrument"
+
+    def on_start(self):
+        return None
+
+    def on_stop(self, telemetry, stdout: str, stderr: str, returncode: int) -> None:
+        from oteltest.telemetry import get_metric_names
+
+        assert "system.cpu.time" in get_metric_names(telemetry)
+
+    def is_http(self):
+        return False