Add logrotate to the Docker image

[ksuderman]

Add logrotate to the Docker image build.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. Build the Docker image
  2. Launch Galaxy using the new image
  3. Login to one of Galaxy pods
  4. Ensure which logrotate returns /usr/sbin/logrotate and the /etc/logrotate.d directory is empty.

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[ksuderman]

I am not sure why, but including logrotate in with the previous RUN apt-get install step causes the adduser step to fail, possibly due to the --no-install-recommends? Placing the install step prior to the adduser step also causes the adduser step to fail. However, placing it after the adduser step works. Color me confused.

[nuwang]

Is this because some logs are getting saved in the container? If so, I think we should stop that from happening. If any logs are getting saved in the container, the logs will be lost whenever the container restarts. Instead, the logging needs to be reconfigured so it goes into the console logs, which can then be inspected with docker log or managed with log shipping tools.

[ksuderman]

The maintence.sh script sends output to a log file that I have been directing to /galaxy/server/database. The problems I've encountered with the console logs is they get deleted with the pod. Since the maintenance.sh script runs as a CronJob the pod is long gone before I can look at the logs. Of all the ways to solve that problem (re-writing the maintenance.sh script, shipping the logs, etc.) the easiest, for now, would be to simply rotate that log file. Getting this into the 24.0 release at least gives us that option.

[nuwang]

I still don't think logrotate should be in this container (among other things, we would have to violate the single-responsibility principle and manage more than one process in the container). Some options we could use:

1. Increase log-retention k8s wide through kubelet configuration - https://kubernetes.io/docs/concepts/cluster-administration/logging/#log-rotation
2. Ue a side-car container, something like: https://github.com/blacklabelops/logrotate that can be optionally enabled. Disabled by default and provided as an option for basic management of logs.
3. Install cluster-level loggers like fluent-bit, fluentd etc. - which we could recommend as the production option.

[nuwang]

Thinking about this a bit more, perhaps we should remove both gxadmin and logrotate, and package them into a utility container for use in maintenance, logging etc. Neither of them need to be in the galaxy container? Looks like gxadmin calls pgcleanup, so it may indeed need to live in this container after all. Alternatively, we could extend the base container and layer gxadmin+logrotate on top in a separate utility container.

[ksuderman]

I still don't think logrotate should be in this container (among other things, we would have to violate the single-responsibility principle and manage more than one process in the container).

All good points. However logrotate would not run as a process in the Galaxy pods, it is just that the CronJobs use the Galaxy image so that is where I stuck it. Having a separate image based on BusyBox or Alpine is something I considered and is likely a better solution if we don't want logrotate in the Galaxy image. I'll close this PR and investigate those options.