Out with the Nix, in with the Docker

[Riscky]

This PR throws our wonky, broken Nix setup out of the window, and replaces it with Docker.
Now I hear you thinking: "these are not even the same category of software products", and you would be right.
Docker doesn't replace Nix in terms of dependency management, instead we go back to good old apt+bundler+yarn.
But instead of making developers install these pieces of software themselves, we create a nice Docker image that contains all of these.
This makes sure that every developer runs (mostly) the same versions of the dependencies.
It makes setting up Koala a lot easier.

Note that this PR is very much Work In Progress.
The production container is building just fine, but:

• The webpack-dev-server container is not doing its job (if you enable it, Koala tries to connect to it, but fails)
• A container for sidekiq is still missing
• The CI pipeline is broken I fixed the CI pipeline, and added a second one that can build and publish the docker container. The latter needs testing
• Documentation is lacking (how do you run database commands against the container, how do you update dependencies, that kind of stuff)
• We need an accompanying PR in https://github.com/svsticky/sadserver to set this up on the servers

@Mstiekema, @TobiasDeBruijn, you both expressed interest in helping me out with this PR, so here goes :)

[SilasPeters]

@Riscky aside from the points mentioned above, could you give a list of things that need to be done in order for this PR to be ready for review/deploy? I would like to know what the next steps are that need to be taken, and how they are best taken.

[SilasPeters]

Could you explain why the secret key can't be used here?

[Riscky]

We cannot bake the (fake) secret in the build, because we need an actual secret on the server. Since we host the build in a public place, we cannot bake the actual secret in the build (and it would make rotating the secret much harder too).
The correct solution is to put the fake secret in the .env file and revert the change in this file. I probably had my reasons while developing this, but I don't recall exactly.

[Riscky]

@Riscky aside from the points mentioned above, could you give a list of things that need to be done in order for this PR to be ready for review/deploy? I would like to know what the next steps are that need to be taken, and how they are best taken.

I think the most important question to solve is how to bring rails and other service commands in scope. Putting everything in a Docker containers is nice for running the application, but when trying to run service commands (eg. rails db:setup), it requires you to run something like docker-compose exec koala-development -- rails db:migrate against a running container. A big drawback of using docker-compose exec (or the docker equivalent) is there is more things that can break, and feedback is rather minimal.
The 'simple' solution is to install everything on the host directly. This was the approach we took before we started using Nix, and made setup on a developer machine quite tedious.
If we want to truly replace Nix, we need a development shell, which, like Nix, brings the necessary dependencies in scope.
We also need a way of building the application so we can put it on the server batteries included.
Docker only covers the latter (as does Nix).
We also need an easy way of starting all services in a development environment, which in this PR is covered by docker-compose, and isn't covered by Nix.
We can use Procfile's and some runner like hivemind for that too, so we don't need to use containers to get to that goal.

Simply put, in the current state this PR solves some of the problems we have (Nix is hard to learn, we don't have a single entry point to run the stack), but introduces new problems (no easy way to run service commands).

What is the best way forward?
I think we should keep Nix as a developer shell, as it makes it very easy to get the right tooling to all developers.
We should however rip out the stuff that doesn't work, like pinning all NodeJS dependencies and gems with Nix.
That means devs have to run bundle install still to get their shells working, after starting a Nix shell.
We should keep Docker for running postgres en redis (like we already did), and for running the webpack dev server and Koala itself.
So: service commands in nix shell, running the stack with docker-compose, shipping to the server as a Docker container.

This keeps the Nix setup to a very manageable level, makes it easy to run service commands and the stack, and makes the build self-contained.

To get there, based on this PR, we'll need to:

• Reintroduce default.nix with just the toplevel tooling. Basically just the things that are now in propagatedBuildInputs + Docker + docker-compose
• Make sure Koala can connect to webpack-dev-server
• Add a container for sidekiq and connect Koala to it
• Update the documentation to reflect the new situation
• Test the build CI pipeline
• Make a PR in https://github.com/svsticky/sadserver to put Docker on it
• Update the Koala role in https://github.com/svsticky/sadserver to fetch the Docker build and run that instead of the Nix build

[Riscky]

Had some time to look at this again, short summary of the current status:

• We currently use host networking because we couldn't get Koala to connect with Redis and Postgres. We probably shouldn't use host networking;
• webpack-dev-server is not dockerized, because it seems to hate us. We don't need it;
• We should leave in Nix for now, while we create a PR in sadserver. devs can start developing in Docker before we use it on the server;
• We still need to dockerize sidekiq;
• and write some more documentation;
• and test the build CI pipeline.

[stickyPiston]

Apparently, docker's network_mode host doesn't work on windows and mac, so we probably definitely need to switch to normal networks again (https://docs.docker.com/network/drivers/host/)

[SamStaijen]

Apparently, docker's network_mode host doesn't work on windows and mac, so we probably definitely need to switch to normal networks again (https://docs.docker.com/network/drivers/host/)

What is the status on this? I have this issue in some other repo's as wel (or well, i technically had no issues but people running mag/win had, but minor detail). Would love to hear the result if a solution has been found! :)