Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OOMKilled during backfill test with no tombstones #15047

Closed
kwannoel opened this issue Feb 7, 2024 · 5 comments
Closed

OOMKilled during backfill test with no tombstones #15047

kwannoel opened this issue Feb 7, 2024 · 5 comments

Comments

@kwannoel
Copy link
Contributor

kwannoel commented Feb 7, 2024

buildkite: https://buildkite.com/risingwave-test/backfill/builds/319#018d7fcc-2073-40c6-b9f6-69022747fed4
Sql for backfill tests: https://github.com/risingwavelabs/kube-bench/blob/main/manifests/backfill/backfill-sql.template.yaml

K8s Describe Pods Before Delete Namespace:

Screenshot 2024-02-07 at 5 38 22 PM

@github-actions github-actions bot added this to the release-1.7 milestone Feb 7, 2024
@kwannoel
Copy link
Contributor Author

kwannoel commented Feb 7, 2024

All these batch selects return too many results perhaps?

https://github.com/risingwavelabs/kube-bench/blob/main/manifests/backfill/backfill-sql.template.yaml

stat.sql: |
    SELECT * FROM backfill_stat;
  batch-query.sql: |
    SELECT * FROM backfill;
  mv-query.sql: |
    SELECT * FROM backfill_mv;

@cyliu0
Copy link
Collaborator

cyliu0 commented Feb 23, 2024

So is this expected? @kwannoel

@kwannoel
Copy link
Contributor Author

kwannoel commented Feb 23, 2024

So is this expected? @kwannoel

Sorry, didn't have time to look at it. But I think it is OOM. Could you change the select * from backfill and backfill_mv into select count(*) instead?

Think it is expected.

@kwannoel
Copy link
Contributor Author

https://buildkite.com/risingwave-test/backfill/builds/346#018dd231-e9dc-4192-bd06-6cd663d61585

Can see from the most recent logs:

Screenshot 2024-02-23 at 12 04 31 PM

It fails when we run the batch queries.

@kwannoel
Copy link
Contributor Author

I don't think we need to select * from backfill_stat as well. That's to view tombstone metrics. We can avoid running that command as well.

@cyliu0 cyliu0 closed this as completed Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants