You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
The plugin system is optimized so that it doesn't load a full event from the DB if no plugin is interested in it. Because the declarativejobset plugin says it's interested in everything, all the buildcompleted events are always loaded from the DB by hydra-notify, which is particularly annoying because each of those events are quite heavy to load (many DB queries for each event).
On hydra.nixos.org, this is causing hydra-notify to constantly stream 750Mbps of (seemingly) eval error text from PostgreSQL just to drop them on the floor and do nothing. Not very efficient.
To Reproduce
n/a
Expected behavior
The declarative jobset system isn't a drain on Hydra's resources when not in use.
Screenshots
n/a
Hydra Server:
hydra.nixos.org.
The text was updated successfully, but these errors were encountered:
Note that working around this issue surfaced a second problem: if no plugin accepts a buildcompleted event, it's never marked as handled in the database because updating notificationpendingsince is dependent on execute being called at least once (which induces ->load being called at least once, which is not wanted).
Lately the hydra-notify service has been sitting on rhea with ~45 GB memory usage, which is a problem since it impacts hydras ability to successfully carry out evaluations.
Describe the bug
The plugin system is optimized so that it doesn't load a full event from the DB if no plugin is interested in it. Because the declarativejobset plugin says it's interested in everything, all the buildcompleted events are always loaded from the DB by hydra-notify, which is particularly annoying because each of those events are quite heavy to load (many DB queries for each event).
On hydra.nixos.org, this is causing hydra-notify to constantly stream 750Mbps of (seemingly) eval error text from PostgreSQL just to drop them on the floor and do nothing. Not very efficient.
To Reproduce
n/a
Expected behavior
The declarative jobset system isn't a drain on Hydra's resources when not in use.
Screenshots
n/a
Hydra Server:
hydra.nixos.org.
The text was updated successfully, but these errors were encountered: