Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weekly Report: 2024-06-12 - 2024-06-19 #10

Closed
LucaLanziani opened this issue Jun 19, 2024 · 0 comments
Closed

Weekly Report: 2024-06-12 - 2024-06-19 #10

LucaLanziani opened this issue Jun 19, 2024 · 0 comments
Labels

Comments

@LucaLanziani
Copy link
Owner

Format

  • {CATEGORY}: {COUNT} ({CHANGE_FROM_PREVIOUS_WEEK})

Issues Report

  • New issues: 34
    Issues
    • ClickHouse exporter produces duplicates and poor compression without sorting attributes (#33634)
    • Span Metrics connector support for OTEP 235 probability sampling (#33632)
    • [kafkareceiver] add support for otlp_json which accepts json formated for Otel Collector kafka receiver (#33627)
    • New component: cfgardenobserver (#33618)
    • [receiver/vcenter] Additional metrics for vCenter receiver (#33607)
    • go generate fails with mdatagen error (#33603)
    • make update-otel shouldn't update the toolchain (#33600)
    • Migrate from otelcoltest.LoadConfigAndValidate to otelcoltest.LoadConfigAndValidateWithSettings (#33594)
    • openid authorization support (otel extension) for opamp supervisor (#33590)
    • [testbed/mockdatasenders/mockdatadogagentexporter] Broken load test (#33577)
    • [cmd/opampsupervisor] Forward Custom Messages to/from agent (#33575)
    • Migrate otel metrics to use mdatagen: deltatocumulative processor (#33573)
    • [extension/observer] Expose host and port separately in endpoint env (#33571)
    • Tail sampling processor: add a way to sample all spans that have a span link to a sampled span. (#33568)
    • Create internal/arrow package for common code used in otelarrowexporter, otelarrowreceiver (#33567)
    • [exporter/datadog] collector panics on invalid trace & span ids in logs (#33566)
    • need k8s.pod.memory.working_set_memory_limit_utilization metric (#33562)
    • [receiver/sqlserver] Allow server config option to include port (#33560)
    • [receiver/dockerstatsreceiver] Does not support setting TLS settings (#33557)
    • [receiver/vcenter] Could use VSAN metrics for vCenter (#33556)
    • [receiver/gitprovider] checks CI/CD action failing due to token limit (#33551)
    • Dynamic topic for googlecloudpubsub exporter (#33547)
    • Support Storage classes in AWSS3 Exporter (#33544)
    • Cardinality or memory limit for prometheus exporters (#33540)
    • ParseJSON doesn't allow leading square bracket (#33535)
    • ottlfunc regex doesn't support double quotes (#33534)
    • New component: Add systemd receiver (#33532)
    • Add OTel-Arrow exporter/receiver to the testbed (#33531)
    • Migrate otel metrics to use mdatagen: routing processor (#33526)
    • Migrate otel metrics to use mdatagen: statsd receiver (#33524)
    • Migrate otel metrics to use mdatagen (#33523)
    • [k8s] The k8s integration tests are failing (#33520)
    • Fetch a GeoIP Database for MaxMind GeoIP Processor (#33510)
    • Metrics for StatefulSet deployments cannot easily be aggregated together (#33507)
  • Issues needing triage: 166
    Issues
    • Prometheus remote write exporter implement v2 (#33661)
    • Load Balancing by Attribute (#33660)
    • windows paths not supported by filelog receiver (include and exclude parameters) (#33657)
    • [processor/tailsampling] invert_match not given precedence when inside and policy (#33656)
    • [receiver/hostmetrics] replace unmaintained library github.com/leoluk/perflib_exporter (#33652)
    • Add a on-demand type of receivers to enable scrape data as needed (#33651)
    • [exporter/otlphttp] Getting errors "VALUE_INVALID", "DIMENSION_KEY_OR_VALUE_EMPTY", "METRIC_UNIT_INVALID_CHARACTERS", "OTLP_MESSAGE_MAX_SIZE_EXCEEDED" (#33650)
    • Require Grafana Dashboard for Azure Monitor (#33649)
    • [exporter/prometheus] Support exporting min/max for histogram metrics (#33645)
    • ClickHouse exporter produces duplicates and poor compression without sorting attributes (#33634)
    • Span Metrics connector support for OTEP 235 probability sampling (#33632)
    • [kafkareceiver] add support for otlp_json which accepts json formated for Otel Collector kafka receiver (#33627)
    • openid authorization support (otel extension) for opamp supervisor (#33590)
    • [testbed/mockdatasenders/mockdatadogagentexporter] Broken load test (#33577)
    • [extension/observer] Expose host and port separately in endpoint env (#33571)
    • Tail sampling processor: add a way to sample all spans that have a span link to a sampled span. (#33568)
    • need k8s.pod.memory.working_set_memory_limit_utilization metric (#33562)
    • [receiver/dockerstatsreceiver] Does not support setting TLS settings (#33557)
    • Dynamic topic for googlecloudpubsub exporter (#33547)
    • Support Storage classes in AWSS3 Exporter (#33544)
    • Cardinality or memory limit for prometheus exporters (#33540)
    • ottlfunc regex doesn't support double quotes (#33534)
    • New component: Add systemd receiver (#33532)
    • Metrics for StatefulSet deployments cannot easily be aggregated together (#33507)
    • Clarify statsd receiver use case description (#33502)
    • S3 Exporter - Sumologic IC marshaler panics when enabled with any other exporter (#33501)
    • Unnecessary call to config.Validate during component's initialization (#33498)
    • Release v0.102.1 to address security risks (#33496)
    • extension/sumologic: lint failure with latest linter (#33494)
    • Validate failed to catch invalid value in resource processor (#33486)
    • [s3exporter] 405 returned when using the S3 exporter cross AWS accounts (#33457)
    • Distributing Load behind ALB/NLBs (#33453)
    • Probabilistic sampling based on a specific attribute (#33449)
    • Line Protocol generated by influxDB exporter seems to be incorrect (#33444)
    • exporter/kafkaexporter: How to create custom trace, metric and log Marshalers (#33439)
    • [exporter/opensearchexporter] Support OLTP Mapping (#33436)
    • [Exporter/LoadBalncer] Increased Memory Utilization after bumping from 0.94.0 to 0.99.0 (#33435)
    • Ingestion of metrics or logs very slow (and high CPU usage on collector) (#33427)
    • New component: Concurrent Batch Processor (#33422)
    • [connector/spanmetricsconnector] Generated counter drops then disappears (#33421)
    • awskinesisexporter: Add raw encode option (#33414)
    • no timestamp value (#33412)
    • supervisor does no retry to connect to opamp server forever (#33408)
    • [receiver/hostmetricsreceiver] Gopsutil error on windows with multiple processor groups. (#33340)
    • [prometheusremotewriteexporter] memory leak when downstream prometheus endpoint is slow/non-responsive leads to GC and "out of order" errors (#33324)
    • attributes processor bot worked (#33343)
    • OpenTelemetry Collector Prometheus exporter fails with "was collected before with the same name and label values" (#33310)
    • [receiver/kafkametrics] Kafka consumer offset gives a random value instead of an actual offset (#33309)
    • [receiver/dockerstats] not generating per container metrics (#33303)
    • syslog exporter does not format structured data with multiple fields properly (#33300)
    • Endpoint from file (#33287)
    • Unable to export traces to elastic search engine backend (#33294)
    • Specify a role_arn for awscloudwatchlogs exporter (#33278)
    • kafkaexporter - support injecting headers (#33260)
    • Enabling WAL is not exporting metrics to Mimir backend using Prometheus remote write exporter (#33238)
    • New component: eG Innovations Telemetry Exporter (#33219)
    • New component: X.509 Certificate Monitoring (#33215)
    • The Source property seems to be the wrong one for mapping client IP to (#33210)
    • [AWS components] aws-sdk-go v1 usage should be upgraded to v2 (#33208)
    • [receiver/kafka]: support receiving from multiple topics (#33204)
    • testbed not working with transform processor in config (#33193)
    • experimental_metricsgeneration divide calculation is not correct (#33179)
    • [exporter/elasticsearch] Bulk indexer error: an id must be provided if version type or value are set (#33139)
    • Replace the RemoteWriteQueue and WAL with the exporterhelper queue (sending_queue) in Prometheusremotewriteexporter (#33137)
    • azureblobreceiver not reading the logs & traces (#33132)
    • Getting ERROR Could not get bootstrap info from the Collector: collector's OpAMP client never connected to the Supervisor (#33129)
    • [exporter/prometheus] Allow setting custom "job" and "instance" attributes (#33118)
    • Access to journal files running in container on k8s (#33104)
    • [exporter/splunkhec] Integration test failing: HTTP response to HTTPS client (#33097)
    • Keep aws cloudwatch metadata (#33080)
    • OpenTelemetry Contrib using Mongodbatlasreceiver prompts "server busy" (#33024)
    • [receiver/filestats] Size of folder should reflect actual used bytes of the folder containing files similar to du (#33016)
    • Add support for Windows Authentication for direct connection to SQL Server instance (#32986)
    • Otel Processor metric filter is not working as expected (#32982)
    • [extension/k8sobserver] ingress ressources (#32971)
    • jmx receiver autodiscover targets in kubernetes? (#32965)
    • Windows event_data format is difficult to consume (#32952)
    • [pkg/ottl] Split ConvertCase function to explicit functions for each case (#32942)
    • I would like to ask if there are any official plans to support the RocketMQ receiver? (#32938)
    • [exporter/prometheusremotewriteexporter] Allow to set batch_send_deadline (#32891)
    • Why trace clickhouse exporter always shows "Exporting Failed: The column Timestamp is not present" (#32886)
    • Easy scaling when using non push based receivers (#32869)
    • system.cpu.time and system.cpu.utilization metrics seem incorrect when running collector on a Windows operating system (#32867)
    • Add support for DigitalOcean droplets to resourcedetetorprocessor (#32858)
    • [CI/CD] Cache Go step failing on windows (#32844)
    • Proposal: Adaptive Filter Processor (#32841)
    • [prometheusreceiver] metric datapoints attributed to two different ServiceMonitor jobs (#32828)
    • spanmetrics: Add a default namespace (#32818)
    • Allow setting of storage policy for clickhouse exporter (#32816)
    • New component: Trace Reshape Processor (#32796)
    • Flaky test: sumologicextension/extension.go:810 (#32785)
    • Enable exporters as Azure Log Analytics Workspace or Azure Application Insight. (#32765)
    • exporter/clickhouse cannot operate without create database, create table permissions (#32738)
    • Exporter/clickhouse support for distributed table (#32736)
    • [receiver/kafka]: Replace "topic" setting by "traces_topic", "logs_topic" and "metrics_topic" (#32735)
    • How should we handle imports of time/tzdata by dependencies? (#32688)
    • [pkg/translator/prometheusremotewrite] Introduce API based around hash based metrics identifiers (#32666)
    • New component: processor for external/remote processing (#32664)
    • [receiver/mongodb] Failing integration test due to timeout (#32658)
    • [receiver/elasticsearch] TestIntegration test times out intermittently (#32656)
    • [receiver/kafka] Ability to provide custom encoders (#32633)
    • [extension/oauth2clientauth] Enable dynamically reading ClientID and ClientSecret from command (#32602)
    • [k8sattributesprocessor] The sources.from types add enum metric_attribute (#32596)
    • otlpjsonfilereceiver: support compressed files from fileexporter (#32565)
    • [exporter/prometheus] does not show metrics from otlp receiver (#32552)
    • sqlqueryreceiver - create one metric per row returned (#32546)
    • [receiver/hostmetrics] Process scrape integration test failing (#32536)
    • [exporter/clickhouse] Integration test hits a panic (#32530)
    • [exporter/prometheus] Support Prometheus Created Timestamp feature (#32521)
    • prometheus exporter precision error with histogram bucket (#32514)
    • [exporter/prometheusremotewrite] Permanent error: Permanent error: context deadline exceeded (#32511)
    • [receiver/chronyreceiver] Receiver is not scraping dial unixgram /var/run/chrony/chronyd.sock (#32487)
    • Resource attribute "service.instance.id" is converted to label "instance", conflicting with auto-generated prometheus label (#32484)
    • loadbalancingexporter makes the collector accept data to produce a reject otelcol_receiver_refused_spans (#32482)
    • [azuremonitorexporter] Duplicate logs on Kubernetes (#32480)
    • Collector fails to restart with persistent queue and health check enabled (#32456)
    • [CI] Unit tests are failing due to timeout for setup-go (#32445)
    • New component: DNS Cache Extension for OpenTelemetry (#32410)
    • [receiver/googlecloudspanner] Test TestItemCardinalityFilter_Filter fails intermittently on Windows runs (#32397)
    • [connector/spanmetrics] Test TestConnectorConsumeTracesExpiredMetrics fails intermittently on actuated ARM runners (#32395)
    • error encoding and sending metric family: write tcp 172.31.204.123:8889->172.31.42.221:60282: write: broken pipe (#32371)
    • Support SNMP Traps in snmp receiver (#32358)
    • Custom Sampler (#32353)
    • Unable to get instance details through mongodb receiver (#32350)
    • [statsdreceiver] fail to parse payloads with empty tag data (#32337)
    • [receiver/windowsperfcounters] When collecting instances with multiple matches, data is lost (#32319)
    • [exporter/clickhouse] Integration test failing due to time out (#32275)
    • Why Does The Kafka Exporter's Raw Marshaler Marshal Everything Except Raw Bytes To JSON? (#32237)
    • the metric of target_info has too mach labels that i not need (#32235)
    • [receiver/awscloudwatch] Missing log stream events (#32231)
    • [Hostmetrics Receiver] Add read and write character values to the process scraper (#32218)
    • Azure monitor exporter authentication (#32163)
    • Failed to connect to opensearch in TLS mode (#32139)
    • Update module github.com/kineticadb/kinetica-api-go to v0.0.4 breaks tests (#32115)
    • Optimize OTEL agent memory usage (#32035)
    • awskinesisexporter: Add support for partitioning records by traceId (#32027)
    • [receiver/datadog] Grafana Cloud Operations table is not detailed by endpoint (#31938)
    • [receiver/httpcheck] Support log pipeline for httpcheck events (#31933)
    • [Makefile.Common] Files under submodule will cause the result of all-pkgs to be empty. (#31928)
    • Jaeger UI SPM spanmetrics not working in 0.95.0 (#31922)
    • prometheusremotewrite context deadline exceeded (#31910)
    • [connector/servicegraph] New labels for service disambiguation and identification (#31889)
    • [exporter/prometheus] Wait for final scrape during collector shutdown (#31887)
    • [connector/spanmetrics] Add maximum span duration metric (#31885)
    • Azure Monitor Exporter Role Name differences (#31884)
    • splunkhecexporter field extraction truncates at 1000 characters (#31817)
    • Support cross account log collection through IAM roles (#31810)
    • Add support for TLS in memcachedreceiver (#31729)
    • Generate gauge metrics from traces (#31696)
    • prometheus receiver: support collectd's binary network protocol (#31546)
    • Load client certificate from hardware security device with Pkcs11 protocol (#31536)
    • [receiver/awscontainerinsight] Gather instance metadata parameters from Kubernetes API when EC2 instance metadata is not accessible (#31511)
    • [exporter/azuremonitor] Forward net.* attributes to Application Insights (#31438)
    • Dynamic selection of log_group_name and log_stream_name in aws cloudwatch logs exporter (#31382)
    • Add Windows Service status metrics (#31377)
    • [processor/resourcedetection] AWS Lambda faas.instance and aws.log.* attributes not set (#31359)
    • Http semantic convention breaking changes in 1.23 (#30935)
    • Add support for Docker container health checks to the collector image (#30798)
    • prometheusremotewrite exporter with histogram is causing metrics export failure due to high memory (90%) (#30675)
    • Generate logs from trace pipeline (#30459)
    • [receiver/redisreceiver] Flaky cluster integration test (#30411)
    • Otel-collector-contrib with prometheus exporter missing exemplars (TraceId and SpanId) (#30197)
    • get a full list of all attributes per resource with full qualified attribute name e.g. from metadata., auth. (#30180)
    • [exporter/clickhouse] exporter fails with IO timeout error under load (#30175)
    • Add 'memory request' feature (#29347)
    • Rogue Parent ID generate in Azure Container App (#28870)
  • Issues ready to merge: 3
    Issues
    • Promote rabbitmqexporter to alpha (#33331)
    • Add support for logs in cloudfoundryreceiver (#33044)
    • [chore]: Document logs_index of OpensearchExporter (#32851)
  • Issues needing sponsorship: 20
    Issues
    • New component: Concurrent Batch Processor (#33422)
    • New component: eG Innovations Telemetry Exporter (#33219)
    • New component: X.509 Certificate Monitoring (#33215)
    • I would like to ask if there are any official plans to support the RocketMQ receiver? (#32938)
    • Proposal: Adaptive Filter Processor (#32841)
    • New component: AWS ApplicationSignals Processor (#32808)
    • New component: Trace Reshape Processor (#32796)
    • New component: processor for external/remote processing (#32664)
    • New component: DNS Cache Extension for OpenTelemetry (#32410)
    • slurm processor (#32312)
    • Data Quality Connector (#31909)
    • New component: DaprExporter and DaprReceiver (#31634)
    • New component: migratecheckpoint (#30656)
    • New component: Fluent Forward Exporter (#29413)
    • New component: IPFIX Lookup (#28692)
    • New component: AWS Lambda Telemetry API Receiver (#26254)
    • New component: Vault Config Source (#24173)
    • New component: Log-based metrics processor (#18269)
    • New component: crash report extension (#16598)
    • New component: AWS CloudWatch metrics receiver (#15667)
  • New issues needing sponsorship: 0
  • Issues and PRs that need code owner review: 14
    Issues
    • Update All github.com/aws packages (#33346)
    • Update module github.com/vmihailenco/msgpack/v4 to v5 (#32641)
    • Update module github.com/vmware/govmomi to v0.37.0 causes build failures (#32453)
    • Update module github.com/opensearch-project/opensearch-go/v2 to v4 (#32444)
    • [chore][receiver/awscontainerinsight] Enable goleak checks in internal packages (#32406)
    • [receiver/awscontainerinsight] Fix memory leak on shutdown in k8s API server (#32405)
    • [chore][receiver/awscontainerinsights] Enable goleak checks (#32404)
    • Failed to connect to opensearch in TLS mode (#32139)
    • Update module github.com/hashicorp/golang-lru to v2 (#30890)
    • Update module github.com/mitchellh/hashstructure to v2 (#30370)
    • Otel-collector-contrib with prometheus exporter missing exemplars (TraceId and SpanId) (#30197)
    • [exporter/clickhouse] exporter fails with IO timeout error under load (#30175)
    • Update module github.com/ReneKroon/ttlcache/v2 to v3 (#29830)
    • prometheusremotewriteexporter logs spurious errors with WAL configured and no metrics to be sent (#24399)

Components Report

LucaLanziani pushed a commit that referenced this issue Nov 9, 2024
… Histo --> Histogram (open-telemetry#33824)

## Description

This PR adds a custom metric function to the transformprocessor to
convert exponential histograms to explicit histograms.

Link to tracking issue: Resolves open-telemetry#33827

**Function Name**
```
convert_exponential_histogram_to_explicit_histogram
```

**Arguments:**

- `distribution` (_upper, midpoint, uniform, random_)
- `ExplicitBoundaries: []float64`

**Usage example:**

```yaml
processors:
  transform:
    error_mode: propagate
    metric_statements:
    - context: metric
      statements:
        - convert_exponential_histogram_to_explicit_histogram("random", [10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0]) 
```

**Converts:**

```
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: response_time
     -> Description: 
     -> Unit: 
     -> DataType: ExponentialHistogram
     -> AggregationTemporality: Delta
ExponentialHistogramDataPoints #0
Data point attributes:
     -> metric_type: Str(timing)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-07-31 09:35:25.212037 +0000 UTC
Count: 44
Sum: 999.000000
Min: 40.000000
Max: 245.000000
Bucket (32.000000, 64.000000], Count: 10
Bucket (64.000000, 128.000000], Count: 22
Bucket (128.000000, 256.000000], Count: 12
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
```

**To:**

```
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: response_time
     -> Description: 
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: Delta
HistogramDataPoints #0
Data point attributes:
     -> metric_type: Str(timing)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-07-30 21:37:07.830902 +0000 UTC
Count: 44
Sum: 999.000000
Min: 40.000000
Max: 245.000000
ExplicitBounds #0: 10.000000
ExplicitBounds #1: 20.000000
ExplicitBounds #2: 30.000000
ExplicitBounds #3: 40.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 60.000000
ExplicitBounds #6: 70.000000
ExplicitBounds #7: 80.000000
ExplicitBounds #8: 90.000000
ExplicitBounds #9: 100.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 2
Buckets #4, Count: 5
Buckets #5, Count: 0
Buckets #6, Count: 3
Buckets #7, Count: 7
Buckets #8, Count: 2
Buckets #9, Count: 4
Buckets #10, Count: 21
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
```

### Testing

- Several unit tests have been created. We have also tested by ingesting
and converting exponential histograms from the `statsdreceiver` as well
as directly via the `otlpreceiver` over grpc over several hours with a
large amount of data.

- We have clients that have been running this solution in production for
a number of weeks.

### Readme description:

### convert_exponential_hist_to_explicit_hist

`convert_exponential_hist_to_explicit_hist([ExplicitBounds])`

the `convert_exponential_hist_to_explicit_hist` function converts an
ExponentialHistogram to an Explicit (_normal_) Histogram.

`ExplicitBounds` is represents the list of bucket boundaries for the new
histogram. This argument is __required__ and __cannot be empty__.

__WARNING:__

The process of converting an ExponentialHistogram to an Explicit
Histogram is not perfect and may result in a loss of precision. It is
important to define an appropriate set of bucket boundaries to minimize
this loss. For example, selecting Boundaries that are too high or too
low may result histogram buckets that are too wide or too narrow,
respectively.

---------

Co-authored-by: Kent Quirk <[email protected]>
Co-authored-by: Tyler Helmuth <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant