Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Declarative jobsets plugin is always enabled, causing buildcompleted events to always get fetched from the DB #1337

Open
delroth opened this issue Jan 12, 2024 · 2 comments
Labels

Comments

@delroth
Copy link
Contributor

delroth commented Jan 12, 2024

Describe the bug
The plugin system is optimized so that it doesn't load a full event from the DB if no plugin is interested in it. Because the declarativejobset plugin says it's interested in everything, all the buildcompleted events are always loaded from the DB by hydra-notify, which is particularly annoying because each of those events are quite heavy to load (many DB queries for each event).

On hydra.nixos.org, this is causing hydra-notify to constantly stream 750Mbps of (seemingly) eval error text from PostgreSQL just to drop them on the floor and do nothing. Not very efficient.

To Reproduce
n/a

Expected behavior
The declarative jobset system isn't a drain on Hydra's resources when not in use.

Screenshots
n/a

Hydra Server:

hydra.nixos.org.

@delroth delroth added the bug label Jan 12, 2024
delroth added a commit to NixOS/infra that referenced this issue Jan 12, 2024
@delroth
Copy link
Contributor Author

delroth commented Jan 13, 2024

Note that working around this issue surfaced a second problem: if no plugin accepts a buildcompleted event, it's never marked as handled in the database because updating notificationpendingsince is dependent on execute being called at least once (which induces ->load being called at least once, which is not wanted).

@delroth delroth moved this to Worked Around in Infra operational issues Jan 14, 2024
@mweinelt
Copy link
Member

Lately the hydra-notify service has been sitting on rhea with ~45 GB memory usage, which is a problem since it impacts hydras ability to successfully carry out evaluations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Worked Around
Development

No branches or pull requests

2 participants