ftw.monitor - health monitoring and warmup

[jone]

Harvest: OneGov GEVER - Softwarepflege (Refactoring/Bug)

Some of our deployments are currently running in a single-threaded setup with a separate monitoring server. The setup seems to stand the test. Therefore we would like to standardize the setup for GEVER and also provide the tools for other Plone deployments in the company.

Thoughts:

• In a single-threaded setup it is not wise to use a HTTPok monitor running on the regular HTTP port: it would restart the instance when it is currently working on a long-running request since there is no second port to test. HTTPok should be disabled.
• We still want to have a health monitoring. Therefore we need a separate health monitoring server.
• Since we want the health monitoring to hold back until the warmup of the instance is finished, warmup and health monitoring is coupled and needs to go to the same product.

Idea:

• Build a new product ftw.monitor (open for better names).

Monitoring Server:

• Provide a health monitoring server. The essence should be extracted from Monitor server opengever.maintenance#152.
• Avoid building on top of collective.monitor in order to avoid additional ZCML. It should be possible to use ftw.monitor without any additional ZCML code.
• Let port be configurable through environment variable.
• Use the default port of instance_port + 80 (e.g. 10102 --> 10182). This is 4teamwork opinionated and that is ok, since it is configurable through environment variable. At 4teamwork we want to go with no configuration at all.

Warmup:

• Extract the catalog warmup functionality from opengever.maintenance.
• Make the list of indexes configurable, e.g. through an adapter. The default list should be minimal. It shouldn't load all indexes as they might be to big to fit into the RAM.
• While warming up, the monitoring server should respond negatively.

Buildout:

• Provide a default buildout configuration in ftw-buildouts. It's probably better to make that an extension to the production.cfg so that older installations do not change.

@lukasgraf I'd like to know your opinion here. Feel free to enhance and update the concept. We'd like to do that in the 2019.1 if possible.

[lukasgraf]

Related: 4teamwork/ftw-buildouts#138

[lukasgraf]

@jone sounds good to me! (Planned for FedEx Day 2019)

[deiferni]

Lock gemäss 4teamwork/ftw.monitor#2 aufgehoben.

[lukasgraf]

Remaining Tasks to switch SaaS deployments:

• Implement auto-warmup in ftw.monitor: Implement auto-warmup ftw.monitor#1 (or point collective.warmup to the warmup view of ftw.monitor)
• Release ftw.monitor == 1.0
• Remove dependency on five.z2monitor from opengever.maintenance.
• Drop monitor code from opengever.maintenance
• Add ftw.monitor as a dependency to opengever.core (or possibly opengever.maintenance)
• Switch deployments to 1 thread per instance
• HttpOk plugins will need to be removed for the switched deployments.
• Adjust HAProxy config for the switched deployments: https://github.com/4teamwork/ftw.monitor#haproxy-example
• Move all SaaS deployments to the new setup

[lukasgraf]

The work in ftw.monitor is finished and tech review is pending at 4teamwork/ftw.monitor#3

For the required buildout changes I created a new issue at #6045. Estimated remaining work: 5 SP

[phgross]

Fachliches Review: 👍

Issue für nächster Task (SaaS deployments) bereits erfasst #6097