Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gitlab: loose coupling to postgresql and redis #1105

Closed
wants to merge 0 commits into from

Conversation

osnyx
Copy link
Member

@osnyx osnyx commented Jan 8, 2024

Description of changes

This reduces the strong degree of coupling between Gitlab and its supporting 3rd party services redis and postgresql.

Background:
When Gitlab and its database postgresql are running on the same system, its systemd services are strongly coupled via binds-to relations to ensure start/stop/restart of the individual services propagates to the other services.
We already had some slight issues with this in the past and fixed the reliability of coupled restarts in NixOS#245240. But since then, we still have seen some additional hard-to-reproduce race conditions that show the approach of strongly-coupled service restarts is brittle here. In one case when postgresql needed to restart, Gitlab was shut down while trying to start and it stayed inactive.
As a consequence, I propose to drop the approach of strong coupling between these services altogether and let the services postgresql or redis restart without forcing a restart of gitlab.service. This just causes the running service to temporarily loose its database connections, but regain them after a successful restart of the db services. Given the long startup times of gitlab.service, this actually even reduces the downtime caused by a database restart in many cases.

Reasoning/ why this decoupling is fine:
By design, both redis and postgresql can be hosted on another host than the gitlab service itself. The NixOS module actually supports this for postgresql, but not for redis (as a dependency of sidekiq). Still, Gitlab and Sidekiq are designed to cope with temporary unavailabilities of redis and postgresql and do recover from such situations automatically.

Pl-131811

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@dpausp dpausp force-pushed the nixos-23.11 branch 4 times, most recently from 50ff595 to fa1270a Compare January 16, 2024 12:44
@dpausp dpausp force-pushed the nixos-23.11 branch 2 times, most recently from 159aa07 to c1d153e Compare February 5, 2024 09:17
@osnyx osnyx force-pushed the PL-131811-gitlab-loose-coupling branch from 14394a2 to adbc656 Compare February 5, 2024 16:06
@dpausp dpausp force-pushed the nixos-23.11 branch 2 times, most recently from 159aa07 to 2c79fa4 Compare February 6, 2024 09:01
@osnyx
Copy link
Member Author

osnyx commented Feb 7, 2024

Superseded by NixOS#286532

@osnyx osnyx closed this Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant