Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky querier unit test #6377

Merged

Conversation

SungJin1212
Copy link
Contributor

@SungJin1212 SungJin1212 commented Nov 27, 2024

This PR fixes flaky querier_test.go due to race condition from https://github.com/cortexproject/cortex/blob/master/pkg/querier/querier.go#L211.

Failed logs:

WARNING: DATA RACE
Write at 0x00000396adb7 by goroutine 2291:
  github.com/cortexproject/cortex/pkg/querier.New()
      /__w/cortex/cortex/pkg/querier/querier.go:211 +0x766
  github.com/cortexproject/cortex/pkg/querier.TestQuerier_ValidateQueryTimeRange_MaxQueryLength_Series()
      /__w/cortex/cortex/pkg/querier/querier_test.go:913 +0x60d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Previous write at 0x00000396adb7 by goroutine 2[65](https://github.com/cortexproject/cortex/actions/runs/12042319929/job/33575766176?pr=6330#step:6:66)0:
  github.com/cortexproject/cortex/pkg/querier.New()
      /__w/cortex/cortex/pkg/querier/querier.go:211 +0x7[66](https://github.com/cortexproject/cortex/actions/runs/12042319929/job/33575766176?pr=6330#step:6:67)
  github.com/cortexproject/cortex/pkg/querier.TestQuerier_ValidateQueryTimeRange_MaxQueryLength.func1()
      /__w/cortex/cortex/pkg/querier/querier_test.go:873 +0x66d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 2291 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:21[68](https://github.com/cortexproject/cortex/actions/runs/12042319929/job/33575766176?pr=6330#step:6:69) +0x85
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:2034 +0xf17
  main.main()
      _testmain.go:153 +0x164

Goroutine 2650 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  github.com/cortexproject/cortex/pkg/querier.TestQuerier_ValidateQueryTimeRange_MaxQueryLength()
      /__w/cortex/cortex/pkg/querier/querier_test.go:856 +0xa0c
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1[69](https://github.com/cortexproject/cortex/actions/runs/12042319929/job/33575766176?pr=6330#step:6:70)0 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44
==================
--- FAIL: TestQuerier_ValidateQueryTimeRange_MaxQueryLength_Series (0.02s)
    testing.go:1399: race detected during execution of test
--- FAIL: TestQuerier_ValidateQueryTimeRange_MaxQueryLength (0.04s)
    --- FAIL: TestQuerier_ValidateQueryTimeRange_MaxQueryLength/should_allow_query_on_short_time_range_and_rate_time_window_close_to_the_limit (0.01s)
        testing.go:1399: race detected during execution of test
--- FAIL: TestConfig_Validate (0.00s)
    testing.go:1399: race detected during execution of test
level=debug org_id=7 component=block.BaseFetcher msg="fetching meta data" concurrency=1
level=info org_id=7 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.001395[76](https://github.com/cortexproject/cortex/actions/runs/12042319929/job/33575766176?pr=6330#step:6:77)s duration_ms=1001 cached=0 returned=0 partial=0
level=error msg="unable to run the initial blocks scan" err="context canceled"
--- FAIL: TestBlocksStoreReplicationSet_GetClientsFor_ShouldSupportRandomLoadBalancingStrategy (2.00s)
    testing.go:1399: race detected during execution of test
--- FAIL: TestBucketIndexBlocksFinder_GetBlocks (0.01s)
    testing.go:1399: race detected during execution of test
level=error msg="unable to run the initial blocks scan" err="context canceled"
FAIL
FAIL	github.com/cortexproject/cortex/pkg/querier	29.[81](https://github.com/cortexproject/cortex/actions/runs/12042319929/job/33575766176?pr=6330#step:6:82)5s

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: SungJin1212 <[email protected]>
Copy link
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:( I am ok with this though if it removes the problem.

Thanks!

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Nov 27, 2024
Copy link
Contributor

@danielblando danielblando left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we already have other tests without the parallel for the same "New" function

@danielblando danielblando merged commit a78470b into cortexproject:master Nov 27, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/querier lgtm This PR has been approved by a maintainer size/XS type/flaky-test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants