Gathered hostmetrics process shown in console but not as metric in prometheus

[securom1987]

Component(s)

receiver/hostmetrics

What happened?

Description

As mentioned in description i am using otel collector v0.114 and hostmetrics receiver with processscraper in ubuntu linux.
I want to scrape process information. These are shown in debug output / console for example:

Console output

Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> process.owner: Str(root)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/processscraper 0.114.0
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> Name: process.cpu.time
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> Name: process.memory.usage
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> Name: process.memory.virtual
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> process.pid: Int(616072)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> process.parent_pid: Int(1)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> process.executable.name: Str(loki)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> process.executable.path: Str()
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> process.command: Str(/usr/bin/loki)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> process.command_line: Str(/usr/bin/loki -config.file /etc/loki/config.yml)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: -> process.owner: Str(loki)

based on this otel-collector config:

extensions:
health_check:
endpoint: 0.0.0.0:1133

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
hostmetrics:
collection_interval: 10s
scrapers:
# CPU utilization metrics
#cpu:
# Disk I/O metrics
# disk:
# File System utilization metrics
#filesystem:
# CPU load metrics
#load:
# Memory utilization metrics
#memory:
# Network interface I/O metrics & TCP connection metrics
#network:
# Paging/Swap space utilization and I/O metrics
#paging:
# Process count metrics
process:
# Per process CPU, Memory, and Disk I/O metrics
processes:

processors:
batch:
resource:
attributes:
- action: insert
key: service.name ## setzt im Grafana in der Metrik die job=HOST1
value: NUC-CLOUD

exporters:
debug:
verbosity: detailed
prometheus:
endpoint: 0.0.0.0:8889

service:
extensions: [health_check]
pipelines:
metrics:
receivers: [otlp, hostmetrics]
processors: [resource, batch]
exporters: [debug, prometheus]

The problem:

The metrics which are written to console are not shown in prometheus.

Collector version

v0.114.0

Environment information

Environment

OS: (e.g., "Ubuntu 24.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

extensions:
  health_check:
    endpoint: 0.0.0.0:1133

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  hostmetrics:
    collection_interval: 10s
    scrapers:
      # CPU utilization metrics
      #cpu:
      # Disk I/O metrics
      # disk:
      # File System utilization metrics
      #filesystem:
      # CPU load metrics
      #load:
      # Memory utilization metrics
      #memory:
      # Network interface I/O metrics & TCP connection metrics
      #network:
      # Paging/Swap space utilization and I/O metrics
      #paging:
      # Process count metrics
      process:
      # Per process CPU, Memory, and Disk I/O metrics
      processes:

processors:
  batch:
  resource:
    attributes:
      - action: insert
        key: service.name           ## setzt im Grafana in der Metrik die job=HOST1
        value: NUC-CLOUD

exporters:
  debug:
    verbosity: detailed
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [resource, batch]
      exporters: [debug, prometheus]

Log output

Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> process.owner: Str(root)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/processscraper 0.114.0
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> Name: process.cpu.time
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> Name: process.memory.usage
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> Name: process.memory.virtual
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> process.pid: Int(616072)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> process.parent_pid: Int(1)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> process.executable.name: Str(loki)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> process.executable.path: Str()
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> process.command: Str(/usr/bin/loki)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> process.command_line: Str(/usr/bin/loki -config.file /etc/loki/config.yml)
Nov 22 09:37:35 nuc-cloud otelcol-contrib[1156080]:      -> process.owner: Str(loki)

Additional context

Metrics which are shown in console/debug log are not shown in prometheus.
For example process "loki" in log output

[github-actions]

Pinging code owners:

• receiver/hostmetrics: @dmitryax @braydonk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[securom1987]

/help-wanted receiver/hostmetrics

[VihasMakwana]

@securom1987 do you see any errors logged from the prometheus exporter?

Can you:

• Disable debug processor (as you've confirmed that metrics get logged)
• Use service::telemetry::logs::level: info and see if you get any hints?

[tdg5]

@securom1987, the format of metric names in OTLP disagrees with the format of metric names in prometheus.

I have no experience with the prometheus exporter, but the prometheusremotewrite exporter has a config that handles translating the OTLP metric names to prometheus friendly names, so you might try the prometheusremotewrite exporter instead. Alternatively, you could look for a similar configuration on the prometheus exporter.

It's clunkier, but this workaround would probably also work if you don't mind having to explicitly map each metric tag.