gitlab: loose coupling to postgresql and redis #1105
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
This reduces the strong degree of coupling between Gitlab and its supporting 3rd party services redis and postgresql.
Background:
When Gitlab and its database postgresql are running on the same system, its systemd services are strongly coupled via
binds-to
relations to ensure start/stop/restart of the individual services propagates to the other services.We already had some slight issues with this in the past and fixed the reliability of coupled restarts in NixOS#245240. But since then, we still have seen some additional hard-to-reproduce race conditions that show the approach of strongly-coupled service restarts is brittle here. In one case when postgresql needed to restart, Gitlab was shut down while trying to start and it stayed inactive.
As a consequence, I propose to drop the approach of strong coupling between these services altogether and let the services postgresql or redis restart without forcing a restart of gitlab.service. This just causes the running service to temporarily loose its database connections, but regain them after a successful restart of the db services. Given the long startup times of gitlab.service, this actually even reduces the downtime caused by a database restart in many cases.
Reasoning/ why this decoupling is fine:
By design, both redis and postgresql can be hosted on another host than the gitlab service itself. The NixOS module actually supports this for postgresql, but not for redis (as a dependency of sidekiq). Still, Gitlab and Sidekiq are designed to cope with temporary unavailabilities of redis and postgresql and do recover from such situations automatically.
Pl-131811
Things done
sandbox = true
set innix.conf
? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
. Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/
)