Seeking input on metrics collection for Uyuni Health Check Tool

[ycedres]

In the context of the RFC for the Health Check Tool for Uyuni, we're currently at the stage of developing the disconnected solution( see Draft PR).

One component of this tool is the Uyuni-Health-Exporter. This component reads from a system troubleshooting information tarball created by the supportconfig utility and generates metrics that are considered to be useful in the process of investigating an issue. These metrics will be presented to the person doing the issue research (using Grafana) in a way that is helpful so that the process of finding the solution is accelerated.

Another component of the Health Check Tool for Uyuni is the Loki component. Loki is a log aggregation system optimised for log storage and querying. Loki uses Promtail as an agent to scrape, collect, and forward logs from various sources to Loki for processing. In this case, the only source would be the supportconfig generated tarball.

I would like to gather opinions regarding what metrics or useful information that should be interesting to collect from the supportconfig tarball. For example, at this moment, we're collecting these metrics for the Uyuni-Health-Exporter:

• Salt Minion Keys
• Salt Jobs
• Salt Configuration File

What other information (either metrics or log entries) generated by supportconfig would be useful to collect and present to anybody investigating an issue?

---

About supportconfig:

The supportconfig tool makes it easier for administrators and support teams to resolve issues by providing a comprehensive system report. It is provided by the supportutils package and can integrate plug-ins that extend the functionality of the base tool enabling it to collect specific data.

[vzhestkov]

I think it could be useful to get the number of log messages matching the following from the salt master log:

Authentication attempt from %MINON-ID% failed, the public keys did not match.
Event with bad payload received from '%MINION-ID%':
Received minion error from [%MINION-ID%]:
An extra return was detected from minion %MINION-ID%, please verify the minion, this could be a replay attack
Failed to send msg SaltReqTimeoutError('Message timed out',)

Some of the customers are ignoring such messages, but they are the signs that something wrong with certain minions or master is overloaded.

[mcalmer]

Check for "Out of Memory" errors, oom-killer and Java Heap out of memory.
Typically in Java log files and the system journal aka /var/log/messages

Check also general minimal system requirements:

• CPUs (Cores)
• RAM
• diskspace (especially free space in /var/spacewalk, /var/lib/pgsql - in the container world /var/lib/containers/storage/volumes/...)

SQLGrammarException in java logfiles are also notable.
Looking into postgresql logfiles and search for errors can also help

[rjmateus]

We could also implement the check for these rules, specially for large scale: https://documentation.suse.com/suma/4.3/en/suse-manager/specialized-guides/large-deployments/tuning.html

[martinjedamzik]

You should take the input from basic-health-check.txt from supportconfig. It gives statistics about the top 10 memory users, CPU and so on. Always my second file in supportconfig to have a look on. Always a good starting point if you look out for issues ;-)

[mcalmer]

Look also in the plugin-.... files. In plugin-susemanager.txt we have some specific checks for values which should be shown when they are not in a kind of expected range.
E.g. we check the size of the task queue. The expected value i 0 or at least very low. If it is a high number we know already that something is broken and we need to look why tasks are faster created and closed.

[martinjedamzik]

maybe the output of "/usr/lib/susemanager/bin/susemanager-connection-check" may give further hints

[martinjedamzik]

salt-key -L | sed '/Denied Keys/,/Unaccepted Keys/!d'
salt-key -L | sed '/Rejected/,/$/!d'

[martinjedamzik]

salt-key -L | sed '/Denied Keys/,/Unaccepted Keys/!d' salt-key -L | sed '/Rejected/,/$/!d'

means:
sed '/Denied Keys/,/Unaccepted Keys/!d' plugin-saltminionskeys.tx

[martinjedamzik]

Parsing spacewalk-debug/rhn-logs/rhn/reposync.logs for errors ?