watch_job_rules: true is causing huge memory consumption

[bgruening]

Describe the bug

If we run with watch_job_rules: true we see a memory usage of up to 30GB per handler.

More information including a py-syp trace is here: https://gist.github.com/bgruening/7be19830e254b4bca99a3ae2e3844041

Disabling the watchdog, as done here, usegalaxy-eu/infrastructure-playbook#937 is solving the problem for us.

Feel free to close this issue, if this seems to be a EU-specific thing. We are also using TPV, so ping also @nuwang and @cat-bro.

[cat-bro]

we downgraded watchdog from 3.0.0 to 2.2.1 in response to handlers consuming lots of memory

[bgruening]

I can confirm:

(venv) galaxy@sn06:~$ pip list | grep watchdo
watchdog 3.0.0

Should we pin watchdog then, until someone can find the problem in 3.0.0?

[nsoranzo]

Yes, that's what I suggested in https://matrix.to/#/!rfLDbcWEWZapZrujix:gitter.im/$avR6Fjh5QQEs7zx7wNlAcjdecHsbH7E9Sm2xJSadED8?via=gitter.im&via=matrix.org

[mvdbeek]

We already ran into this during the admin training in Ghent. I thought the fix at the time was to move the job rules to a place where they aren't monitored by other watchers as well, so apologies that this was lost.

[bgruening]

No need to apologize, really. I'm not even sure why it has just hit us now. So glad I created an issue and you got a fix so fast. Awesome!

[mvdbeek]

I am fairly certain e9bc593 fixed this, I'd appreciate a little test of course. And thanks @cat-bro for figuring out the version at which this still worked, that made the fix easy.