-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[close #772] Fix flaky integration tests #770
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Codecov ReportAll modified lines are covered by tests ✅ see 9 files with indirect coverage changes 📢 Thoughts on this report? Let us know!. |
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
pingyu
changed the title
[close #xxx] Fix flaky integration tests
[close #772] Fix flaky integration tests
Oct 3, 2023
Signed-off-by: Ping Yu <[email protected]>
iosmanthus
approved these changes
Oct 7, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #772
Problem Description: Integration tests in CI are flaky
What is changed and how does it work?
The reason of failure of integration tests in CI seems to be TiKV OOM.
We can see PD response with errors that "cluster is not bootstrapped" (see "Run Integration Test" in https://github.com/tikv/client-java/actions/runs/6390085322/job/17342615588?pr=770)
After print TiKV logs after failure, there is no more logs after TiKV print the "using config" (see "Print TiKV logs" in https://github.com/tikv/client-java/actions/runs/6390085322/job/17342615588?pr=770). It was very likely to be OOM killed.
Containers in Github action runner has only
7GB
memory (see https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners). I think it's not enough for 2 PD and 2 TiKV, especially for newer versions.To address the issue, this PR setup only one cluster in CI, and enable APIv2 to accept both RawKV & TxnKV test cases.
At the same time, keep 2 clusters for older versions, and cover the scenarios of
APIv1
&APIv1 TTL
.Changes:
v6.5.3
,v7.1.1
&nightly
, which setup only one cluster and enableAPIv2
.raftstore.capacity
as there is only14GB
disk in Github action runner.storage.block-cache.capacity
to make less chances of OOM killed.--kv.port xxx
argument to TiUP. Otherwise TiKV of txnkv will complain about the port is occupied ([2023/10/03 09:02:28.144 +00:00] [FATAL] [server.rs:1165] ["127.0.0.1_20160 already in use, maybe another instance is binding with this address."]
, see https://github.com/tikv/client-java/actions/runs/6390868184/job/17345000675?pr=770). It may be an issue of TiUP as it should get a free port, but just work around this issue in this PR).Code changes
Check List for Tests
This PR has been tested by at least one of the following methods:
Side effects
Related changes