Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race observed in TestMultiDimensionalQueueAlgorithmSlowConsumerEffects test. #9253

Open
pstibrany opened this issue Sep 10, 2024 · 2 comments · Fixed by #9360
Open

Race observed in TestMultiDimensionalQueueAlgorithmSlowConsumerEffects test. #9253

pstibrany opened this issue Sep 10, 2024 · 2 comments · Fixed by #9360
Labels
type/tests

Comments

@pstibrany
Copy link
Member

Race observed in TestMultiDimensionalQueueAlgorithmSlowConsumerEffects test.

==================
WARNING: DATA RACE
Write at 0x00c0003ba0b8 by goroutine 2879:
  github.com/grafana/mimir/pkg/scheduler/queue.(*tenantQuerierAssignments).setup()
      /__w/mimir/mimir/pkg/scheduler/queue/tenant_querier_assignment.go:423 +0xc4
  github.com/grafana/mimir/pkg/scheduler/queue.(*MultiQueuingAlgorithmTreeQueue).Dequeue()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue.go:85 +0x115
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:485 +0xc9b
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Previous read at 0x00c0003ba0b8 by goroutine 2880:
  github.com/grafana/mimir/pkg/scheduler/queue.(*queueBroker).dequeueRequestForQuerier()
      /__w/mimir/mimir/pkg/scheduler/queue/tenant_queues.go:152 +0x146
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).trySendNextRequestForQuerier()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:422 +0x76
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).dispatcherLoop()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:340 +0x364
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting.gowrap1()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:289 +0x33

Goroutine 2879 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:422 +0x1e04
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 2880 (running) created at:
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:289 +0x8d
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:442 +0x486
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44
==================
@dimitarvdimitrov
Copy link
Contributor

another one

Details 

==================
WARNING: DATA RACE
Write at 0x00c000539ad8 by goroutine 4103:
  github.com/grafana/mimir/pkg/scheduler/queue.(*tenantQuerierAssignments).setup()
      /__w/mimir/mimir/pkg/scheduler/queue/tenant_querier_assignment.go:237 +0xc4
  github.com/grafana/mimir/pkg/scheduler/queue.(*MultiQueuingAlgorithmTreeQueue).Dequeue()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue.go:85 +0x115
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:485 +0xc9b
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Previous read at 0x00c000539ad8 by goroutine 4104:
  github.com/grafana/mimir/pkg/scheduler/queue.(*queueBroker).dequeueRequestForQuerier()
      /__w/mimir/mimir/pkg/scheduler/queue/tenant_queues.go:156 +0x153
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).trySendNextRequestForQuerier()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:392 +0x[76](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:77)
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).dispatcherLoop()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:310 +0x364
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting.gowrap1()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:259 +0x33

Goroutine 4103 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:422 +0x1d84
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 4104 (running) created at:
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:259 +0x8d
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:442 +0x486
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44
==================
--- FAIL: TestMultiDimensionalQueueAlgorithmSlowConsumerEffects (15.47s)
    --- FAIL: TestMultiDimensionalQueueAlgorithmSlowConsumerEffects/tree:_worker-queue_prioritization_->_tenant-querier_tree,_2_tenants,_first_with_25pct_slow_queries,_second_with_75pct_slow_queries (0.67s)
        multi_queuing_algorithm_tree_queue_benchmark_test.go:4[78](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:79): tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.0290 stddev: 0.01 store-gateway: mean: 0.2595 stddev: 0.17]
        multi_queuing_algorithm_tree_queue_benchmark_test.go:4[79](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:80): tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0329 stddev: 0.01 tenant-1: mean: 0.0169 stddev: 0.00]
        testing.go:1399: race detected during execution of test
    multi_queuing_algorithm_tree_queue_benchmark_test.go:492: Results by query component:
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.1128 stddev: 0.02 store-gateway: mean: 0.0282 stddev: 0.04]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.1191 stddev: 0.02 store-gateway: mean: 0.0270 stddev: 0.04]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.0283 stddev: 0.01 store-gateway: mean: 0.0323 stddev: 0.03]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.2079 stddev: 0.05 store-gateway: mean: 0.0963 stddev: 0.08]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.2359 stddev: 0.07 store-gateway: mean: 0.1067 stddev: 0.08]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.0247 stddev: 0.01 store-gateway: mean: 0.1124 stddev: 0.08]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.3968 stddev: 0.17 store-gateway: mean: 0.2492 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.3963 stddev: 0.18 store-gateway: mean: 0.2500 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.0200 stddev: 0.00 store-gateway: mean: 0.2642 stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.2599 stddev: 0.16 store-gateway: mean: 0.4161 stddev: 0.25]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.2574 stddev: 0.17 store-gateway: mean: 0.4107 stddev: 0.26]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.0139 stddev: 0.00 store-gateway: mean: 0.4102 stddev: 0.25]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.1076 stddev: 0.08 store-gateway: mean: 0.4847 stddev: 0.30]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0822 stddev: 0.07 store-gateway: mean: 0.5012 stddev: 0.31]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0103 stddev: 0.00 store-gateway: mean: 0.4971 stddev: 0.31]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.1291 stddev: 0.04 store-gateway: mean: 0.2398 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.1437 stddev: 0.04 store-gateway: mean: 0.2519 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.0200 stddev: 0.00 store-gateway: mean: 0.2717 stddev: 0.18]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.2099 stddev: 0.08 store-gateway: mean: 0.2400 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.2298 stddev: 0.09 store-gateway: mean: 0.2440 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.0290 stddev: 0.01 store-gateway: mean: 0.2595 stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3658 stddev: 0.17 store-gateway: mean: 0.2202 stddev: 0.14]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3548 stddev: 0.15 store-gateway: mean: 0.2205 stddev: 0.14]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.0257 stddev: 0.00 store-gateway: mean: 0.2188 stddev: 0.14]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:497: Results for ingester-only queries by tenant ID:
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.1128 stddev: 0.02]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.1191 stddev: 0.02]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.0283 stddev: 0.01]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.2079 stddev: 0.05]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.2359 stddev: 0.07]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.0247 stddev: 0.01]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3968 stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3963 stddev: 0.18]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0200 stddev: 0.00]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2599 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2574 stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0139 stddev: 0.00]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.1076 stddev: 0.08]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0822 stddev: 0.07]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0103 stddev: 0.00]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.13[80](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:81) stddev: 0.02 tenant-1: mean: 0.0652 stddev: 0.05]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.1456 stddev: 0.04 tenant-1: mean: 0.1286 stddev: 0.08]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0209 stddev: 0.00 tenant-1: mean: 0.0119 stddev: 0.00]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2139 stddev: 0.07 tenant-1: mean: 0.1968 stddev: 0.10]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2294 stddev: 0.07 tenant-1: mean: 0.2309 stddev: 0.13]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0329 stddev: 0.01 tenant-1: mean: 0.0169 stddev: 0.00]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3527 stddev: 0.17 tenant-1: mean: 0.3802 stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.32[88](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:89) stddev: 0.16 tenant-1: mean: 0.3794 stddev: 0.14]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0261 stddev: 0.00 tenant-1: mean: 0.0252 stddev: 0.00]
FAIL

@chencs @francoposa does anything come to mind?

@chencs chencs mentioned this issue Sep 20, 2024
Merged
4 tasks
@dimitarvdimitrov
Copy link
Contributor

dimitarvdimitrov commented Sep 30, 2024 •
edited
Loading

this failed again, I think the failure is the same - GH action run

Details 

==================
WARNING: DATA RACE
Read at 0x00c0001bc758 by goroutine 2880:
  github.com/grafana/mimir/pkg/scheduler/queue.(*queueBroker).dequeueRequestForQuerier()
      /__w/mimir/mimir/pkg/scheduler/queue/queue_broker.go:154 +0x153
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).trySendNextRequestForQuerier()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:392 +0x[76](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:77)
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).dispatcherLoop()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:310 +0x364
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting.gowrap1()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:259 +0x33

Previous write at 0x00c0001bc758 by goroutine 2879:
  github.com/grafana/mimir/pkg/scheduler/queue.(*tenantQuerierAssignments).setup()
      /__w/mimir/mimir/pkg/scheduler/queue/tenant_querier_assignment.go:239 +0xc4
  github.com/grafana/mimir/pkg/scheduler/queue.(*MultiQueuingAlgorithmTreeQueue).Dequeue()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue.go:90 +0x134
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:491 +0xd06
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 2880 (running) created at:
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:259 +0x8d
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:448 +0x4d6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 2879 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:428 +0x21[77](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:78)
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44
==================
--- FAIL: TestMultiDimensionalQueueAlgorithmSlowConsumerEffects (16.65s)
    --- FAIL: TestMultiDimensionalQueueAlgorithmSlowConsumerEffects/tree:_worker-queue_prioritization_->_tenant-querier_tree,_1_tenant,_90pct_slow_queries (1.17s)
        multi_queuing_algorithm_tree_queue_benchmark_test.go:484: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0182 stddev: 0.01 store-gateway: mean: 0.5084 stddev: 0.30]
        multi_queuing_algorithm_tree_queue_benchmark_test.go:485: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0182 stddev: 0.01]
        testing.go:1399: race detected during execution of test
    multi_queuing_algorithm_tree_queue_benchmark_test.go:498: Results by query component:
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.1237 stddev: 0.02 store-gateway: mean: 0.0285 stddev: 0.03]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.1186 stddev: 0.02 store-gateway: mean: 0.0154 stddev: 0.01]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.0471 stddev: 0.01 store-gateway: mean: 0.0506 stddev: 0.03]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.2933 stddev: 0.08 store-gateway: mean: 0.1186 stddev: 0.08]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.2339 stddev: 0.06 store-gateway: mean: 0.1142 stddev: 0.07]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.0398 stddev: 0.01 store-gateway: mean: 0.1497 stddev: 0.09]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.4435 stddev: 0.21 store-gateway: mean: 0.2[80](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:81)6 stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.4431 stddev: 0.21 store-gateway: mean: 0.2725 stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.0288 stddev: 0.00 store-gateway: mean: 0.2867 stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.2571 stddev: 0.16 store-gateway: mean: 0.4390 stddev: 0.26]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.2957 stddev: 0.19 store-gateway: mean: 0.4087 stddev: 0.25]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.0205 stddev: 0.00 store-gateway: mean: 0.4313 stddev: 0.26]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0902 stddev: 0.07 store-gateway: mean: 0.5085 stddev: 0.31]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0844 stddev: 0.06 store-gateway: mean: 0.5138 stddev: 0.31]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0182 stddev: 0.01 store-gateway: mean: 0.5084 stddev: 0.30]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.1370 stddev: 0.04 store-gateway: mean: 0.24[81](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:82) stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.1542 stddev: 0.04 store-gateway: mean: 0.2623 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.0388 stddev: 0.01 store-gateway: mean: 0.2857 stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.2835 stddev: 0.11 store-gateway: mean: 0.2540 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.2316 stddev: 0.09 store-gateway: mean: 0.2447 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.0264 stddev: 0.01 store-gateway: mean: 0.2565 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3802 stddev: 0.18 store-gateway: mean: 0.2350 stddev: 0.15]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3842 stddev: 0.18 store-gateway: mean: 0.2300 stddev: 0.14]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.0322 stddev: 0.00 store-gateway: mean: 0.2353 stddev: 0.14]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:503: Results for ingester-only queries by tenant ID:
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.1237 stddev: 0.02]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.1186 stddev: 0.02]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.0471 stddev: 0.01]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.2933 stddev: 0.08]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.2339 stddev: 0.06]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.0398 stddev: 0.01]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.4435 stddev: 0.21]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.4431 stddev: 0.21]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0288 stddev: 0.00]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2571 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2957 stddev: 0.19]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0205 stddev: 0.00]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0902 stddev: 0.07]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0844 stddev: 0.06]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.01[82](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:83) stddev: 0.01]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.1466 stddev: 0.03 tenant-1: mean: 0.0440 stddev: 0.04]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.1568 stddev: 0.04 tenant-1: mean: 0.1338 stddev: 0.07]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0404 stddev: 0.00 tenant-1: mean: 0.0239 stddev: 0.01]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.3130 stddev: 0.08 tenant-1: mean: 0.1952 stddev: 0.12]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2260 stddev: 0.07 tenant-1: mean: 0.2480 stddev: 0.13]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0280 stddev: 0.00 tenant-1: mean: 0.0219 stddev: 0.00]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3606 stddev: 0.18 tenant-1: mean: 0.39[85](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:86) stddev: 0.17]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3597 stddev: 0.19 tenant-1: mean: 0.4072 stddev: 0.16]
    multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0315 stddev: 0.00 tenant-1: mean: 0.0331 stddev: 0.00]
FAIL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix race in TestMultiDimensionalQueueAlgorithmSlowConsumerEffects grafana/mimir
3 participants
@pstibrany @56quarters @dimitarvdimitrov