Add monitoring for SQL redundancy

[cornelinux]

We could add some monitoring if master-master replication is working.

Either we could take a direct look or we could actually use privacyIDEA to verify this like:

1. create a token on the first node
2. check later, if this token exists on the 2nd node
3. delete token on the 2nd node
4. check later, if this token does not exist anymore on the 1st node.

This would be a successful check without any database specifics involved. So this could also work for other redundant setups.

[cornelinux]

The top level requirement for such monitoring should be, that it can be done, without the need to login on the shell.

So this could be done via either:

A) A nagios script, that provides the information for a 3rd party monitoring tool.

This coud be done on the DB level, checking the replication state on both nodes.

B) Displaying information in the privacyIDEA WebUI

This looks convenient in the first place. A systemlevel monitoring script could write the current DBsync state in a monitoring table, so that privacyIDEA could display it in the WebUI.
However, I am not sure, if redundancy can fail in only one direction and on node thinks, everything is fine and the other node is broken. So if admins only log in to the first node, then they again would not be notified.

C) A script, that can run in a cron job and send emails (or notifications) to recipients

We could actually use the same script as in A) and then use the privacyIDEA notification mechanisms/code.
Or we could on purpose not use privacyIDEA code. In case the redundancy is broken, maybe privacyIDEA - and thus the notification - does not work anymore correctly.