[Orchestrator] add new check to collect ecs tasks #22060
Conversation
Go Package Import DifferencesBaseline: 46c7bd1
|
kangyili
force-pushed
the
kangyi/ecs-new-core-check
branch
from
0a9bc12 to
19014bd
Compare
Bloop Bleep... Dogbot HereRegression Detector ResultsRun ID: d9233538-ba17-4993-bdc8-1153e942e6ee Performance changes are noted in the perf column of each table:
No significant changes in experiment optimization goalsConfidence level: 90.00% There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
|
| perf | experiment | goal | Δ mean % | Δ mean % CI |
|---|---|---|---|---|
| ➖ | file_to_blackhole | % cpu utilization | +0.02 | [-6.28, +6.32] |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI |
|---|---|---|---|---|
| ➖ | otel_to_otel_logs | ingress throughput | +1.30 | [+0.64, +1.95] |
| ➖ | process_agent_standard_check | memory utilization | +0.63 | [+0.59, +0.67] |
| ➖ | idle | memory utilization | +0.38 | [+0.34, +0.43] |
| ➖ | file_to_blackhole | % cpu utilization | +0.02 | [-6.28, +6.32] |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.02 | [-0.04, +0.07] |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.00 | [-0.06, +0.06] |
| ➖ | trace_agent_json | ingress throughput | +0.00 | [-0.01, +0.01] |
| ➖ | trace_agent_msgpack | ingress throughput | -0.02 | [-0.03, -0.01] |
| ➖ | process_agent_standard_check_with_stats | memory utilization | -0.04 | [-0.09, -0.00] |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.18 | [-2.98, +2.62] |
| ➖ | file_tree | memory utilization | -0.68 | [-0.78, -0.58] |
| ➖ | process_agent_real_time_mode | memory utilization | -0.83 | [-0.87, -0.78] |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.95 | [-1.03, -0.87] |
| ➖ | basic_py_check | % cpu utilization | -2.04 | [-4.28, +0.21] |
| ➖ | pycheck_1000_100byte_tags | % cpu utilization | -2.63 | [-7.52, +2.25] |
Explanation
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
pkg/collector/corechecks/cluster/orchestrator/processors/common/base.go
Outdated
Show resolved
Hide resolved
pkg/collector/corechecks/cluster/orchestrator/collector_bundle.go
Outdated
Show resolved
Hide resolved
pkg/collector/corechecks/cluster/orchestrator/collectors/ecs/task.go
Outdated
Show resolved
Hide resolved
kangyili
force-pushed
the
kangyi/ecs
branch
3 times, most recently
from
8d02b6a to
09f57df
Compare
kangyili
force-pushed
the
kangyi/ecs-new-core-check
branch
from
19014bd to
7bac94c
Compare
kangyili
force-pushed
the
kangyi/ecs-new-core-check
branch
3 times, most recently
from
b6cfa1d to
66bbe5d
Compare
|
I created a new PR only updates code structure #23823 |
kangyili
force-pushed
the
kangyi/ecs-new-core-check
branch
from
58f15d5 to
076db95
Compare
kangyili
force-pushed
the
kangyi/update-code-structure
branch
from
e9f189e to
52fb6a1
Compare
kangyili
force-pushed
the
kangyi/ecs-new-core-check
branch
from
9fd8b7a to
d8faa74
Compare
Test changes on VMUse this command from test-infra-definitions to manually test this PR changes on a VM: inv create-vm --pipeline-id=30572198 --os-family=ubuntu |
Regression DetectorRegression Detector ResultsRun ID: cf9295b8-f32c-4900-9c2c-31aeda2746e4 Performance changes are noted in the perf column of each table:
No significant changes in experiment optimization goalsConfidence level: 90.00% There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
|
| perf | experiment | goal | Δ mean % | Δ mean % CI |
|---|---|---|---|---|
| ➖ | file_to_blackhole | % cpu utilization | +1.32 | [-5.10, +7.75] |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI |
|---|---|---|---|---|
| ➖ | file_to_blackhole | % cpu utilization | +1.32 | [-5.10, +7.75] |
| ➖ | idle | memory utilization | +0.29 | [+0.26, +0.32] |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | +0.15 | [+0.07, +0.23] |
| ➖ | otel_to_otel_logs | ingress throughput | +0.11 | [-0.31, +0.52] |
| ➖ | process_agent_standard_check_with_stats | memory utilization | +0.06 | [+0.04, +0.09] |
| ➖ | trace_agent_msgpack | ingress throughput | +0.03 | [+0.01, +0.04] |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.02 | [-0.02, +0.05] |
| ➖ | trace_agent_json | ingress throughput | +0.00 | [-0.03, +0.04] |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.00 | [-0.20, +0.20] |
| ➖ | pycheck_1000_100byte_tags | % cpu utilization | -0.07 | [-4.97, +4.83] |
| ➖ | file_tree | memory utilization | -0.24 | [-0.33, -0.16] |
| ➖ | process_agent_real_time_mode | memory utilization | -0.59 | [-0.62, -0.56] |
| ➖ | process_agent_standard_check | memory utilization | -0.63 | [-0.67, -0.59] |
| ➖ | basic_py_check | % cpu utilization | -1.14 | [-3.62, +1.34] |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -2.94 | [-5.60, -0.28] |
Explanation
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
pkg/collector/corechecks/cluster/orchestrator/collectors/ecs/task.go
Outdated
Show resolved
Hide resolved
pkg/collector/corechecks/cluster/orchestrator/collectors/ecs/task.go
Outdated
Show resolved
Hide resolved
pkg/collector/corechecks/cluster/orchestrator/collectors/ecs/task.go
Outdated
Show resolved
Hide resolved
pkg/collector/corechecks/cluster/orchestrator/collectors/ecs/task.go
Outdated
Show resolved
Hide resolved
pkg/collector/corechecks/cluster/orchestrator/collectors/ecs/task.go
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| taskModel.Tags = tags | ||
| taskModel.EcsTags = toTags(task.Task.Tags) |
There was a problem hiding this comment.
Is there any normalization needed for those tags?
There was a problem hiding this comment.
I'm not sure if we should do the normalisation here as the tag standard of ecs and datadog are different, changing tags can confuse user. We can probably leave it as it is for now
kangyili
force-pushed
the
kangyi/ecs-new-core-check
branch
from
7225ef1 to
2cc7e05
Compare
|
/merge |
|
🚂 MergeQueue Pull request added to the queue. There are 3 builds ahead! (estimated merge in less than 29m) Use |
|
/merge |
|
🚂 MergeQueue Pull request added to the queue. There are 3 builds ahead! (estimated merge in less than 29m) Use |
…cs-new-core-check
|
/merge |
|
🚂 MergeQueue This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals. Use |
|
🚂 MergeQueue Added to the queue. This build is next! (estimated merge in less than 28m) Use |
* add new check to collect ecs tasks * address feedback * bump agent payload version
What does this PR do?
This PR requires
New check
orchestrator_ecsorchestrator_ecsin core checks to collect running tasks from ecs-ec2 and ecs-fargate.DD_ORCHESTRATOR_EXPLORER_ENABLEDandDD_ORCHESTRATOR_EXPLORER_ECS_COLLECTION_ENABLEDWorkloadmetafor pulling tasks in every rundatadog-agent/pkg/collector/corechecks/cluster/orchestrator/collectors/ecs/task.go
Line 57 in f67c3d6
Motivation
Additional Notes
Possible Drawbacks / Trade-offs
Describe how to test/QA your changes
Reviewer's Checklist
Triagemilestone is set.major_changelabel if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.changelog/no-changeloglabel has been applied.qa/skip-qalabel, with required eitherqa/doneorqa/no-code-changelabels, are applied.team/..label has been applied, indicating the team(s) that should QA this change.need-change/operatorandneed-change/helmlabels have been applied.k8s/<min-version>label, indicating the lowest Kubernetes version compatible with this feature.