fix some metrics which are not actually histograms #117
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mitali-salvi
previously approved these changes
Oct 17, 2023
sky333999
pushed a commit
that referenced
this pull request
Oct 28, 2024
**Description:**
Implements OTel (OpenTelemetry-native) mode serialization for
elasticsearch exporter.
This is an initial cut in order to get the discussion going.
This is approach was tested as internal POC.
It leverages Elasticsearch ```"passthrough"``` fields mapping initially
introduced in Elasticsearch 8.13 allowing users to query the
document/scope/resources attributes as top level fields, making the ECS
queries compatible with OTel sematic convention schema. Another benefit
is the simplicity of conversion of stored document from Elasticsearch
back to Otel data model format.
The document/scope/resources attributes are dynamically mapped and
stored as flattened keys.
Here is an example of index template mappings with ```"passthrough"```
fields:
```
PUT _index_template/logs_otel
{
"priority": 250,
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "logs"
},
"codec": "best_compression",
"mapping": {
"ignore_malformed": "true"
}
}
},
"mappings": {
"_source": {
"enabled": true
},
"date_detection": false,
"dynamic": "strict",
"dynamic_templates": [
{
"all_strings_to_keywords": {
"mapping": {
"ignore_above": 1024,
"type": "keyword"
},
"match_mapping_type": "string"
}
},
{
"complex_attributes": {
"path_match": [
"resource.attributes.*",
"scope.attributes.*",
"attributes.*"
],
"match_mapping_type": "object",
"mapping": {
"type": "flattened"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date_nanos",
"ignore_malformed": false
},
"data_stream": {
"type": "object",
"properties": {
"type": {
"type": "constant_keyword"
},
"dataset": {
"type": "constant_keyword"
},
"namespace": {
"type": "constant_keyword"
}
}
},
"observed_timestamp": {
"type": "date_nanos",
"ignore_malformed": true
},
"severity_number": {
"type": "long"
},
"severity_text": {
"type": "keyword"
},
"body_text": {
"type": "match_only_text"
},
"body_structured": {
"type": "flattened"
},
"attributes": {
"type": "passthrough",
"dynamic": true,
"priority": 2
},
"dropped_attributes_count": {
"type": "long"
},
"trace_flags": {
"type": "byte"
},
"trace_id": {
"type": "keyword"
},
"span_id": {
"type": "keyword"
},
"scope": {
"properties": {
"name": {
"type": "keyword"
},
"version": {
"type": "keyword"
},
"attributes": {
"type": "passthrough",
"dynamic": true,
"priority": 1
},
"dropped_attributes_count": {
"type": "long"
},
"schema_url": {
"type": "keyword"
}
}
},
"resource": {
"properties": {
"dropped_attributes_count": {
"type": "long"
},
"schema_url": {
"type": "keyword"
},
"attributes": {
"type": "passthrough",
"dynamic": true,
"priority": 0
}
}
}
}
}
},
"index_patterns": [
"logs-*.otel-*"
],
"data_stream": {}
}
```
Here is an example of the auditd document in Elasticsearch abbreviated:
```
{
"@timestamp": "2024-05-29T13:30:25.085926000Z",
"attributes": {
"foo": "bar",
"some.bool": true
},
"body_structured": {
"MESSAGE": "AVC apparmor=\"STATUS\" operation=\"profile_replace\" info=\"same as current profile, skipping\" profile=\"unconfined\" name=\"/usr/bin/evince-previewer\" pid=2702 comm=\"apparmor_parser\"",
"SYSLOG_FACILITY": "4",
"SYSLOG_IDENTIFIER": "audit",
"_SOURCE_REALTIME_TIMESTAMP": "1716989425080000",
"_TRANSPORT": "audit",
},
"dropped_attributes_count": 0,
"observed_timestamp": "2024-05-29T14:49:26.534908898Z",
"resource": {
"attributes": {
"data_stream.dataset": "auditd.otel",
"data_stream.namespace": "default",
"data_stream.type": "logs",
"host.arch": "arm64",
"host.cpu.cache.l2.size": 0,
"host.cpu.family": "",
"host.cpu.model.id": "0x000",
"host.cpu.model.name": "",
"host.cpu.stepping": "0",
"host.cpu.vendor.id": "Apple",
"host.id": "cae0e0147d454a80971b0b747c8b62b9",
"host.ip": [
"172.16.3.131",
"fe80::20c:29ff:fe66:3012",
"host.name": "lebuntu",
"host.os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic #117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
"host.os.type": "linux",
"os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic #117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
"os.type": "linux"
},
"dropped_attributes_count": 0,
"schema_url": "https://opentelemetry.io/schemas/1.6.1"
},
"severity_number": 0,
"trace_flags": 0
}
```
Here is an example of ECS compatible query that works on this Otel
native schema:
```
GET logs-auditd.otel-default/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"host.name": "lebuntu"
}
}
]
}
}
}
```
**Link to tracking Issue:**
No tracking issue yet.
**Testing:**
Added unit test for OTel transformation.
Tested with journald OTel receiver.
**Documentation:**
No documentation is added yet.
---------
Co-authored-by: Felix Barnsteiner <[email protected]>
Co-authored-by: Carson Ip <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
These metrics are not histograms, they were not being emitted because the regex waswrong
I don't have
etcd_db_total_size_in_bytesin my cluster because it has been replaced withapiserver_storage_db_total_size_in_bytesLink to tracking Issue:
Testing: end to end
I am open to ideas on how we ensure there are no regressions on these
Documentation: