Allow users to customize outputs list

[Ten0]

I'm considering using this project.

IIUC, I currently have two ways to generate an "output" for the CLI:

1. use a builtin, and that will generate an arbirary number of targets depending on some magic in makes' own code
2. use an extension, which always means creating a folder for the target, and having a main.nix in that folder.

What I want to achieve is the following directory structure:

project1
        /service1/<main or makes>.nix
        /service2/<main or makes>.nix
project2
        /service1/<main or makes>.nix
makes/service.nix

and I want my possible "OUTPUT"s in the CLI to look like this:

/project1/service1/build
/project1/service1/test
/project1/service1/run
/project1/service1/deploy
/project1/service2/build
/project1/service2/test
/project1/service2/run
/project1/service2/deploy
/project2/service1/build
/project2/service1/test
/project2/service1/run
/project2/service1/deploy

where the build/test/run/deploy outputs behavior are all defined by my /makes/service.nix

That means I need to be able to customize:

• The list of targets that are made available from a <main or makes>.nixs (so that with a single nix file in a service folder I can declare the multiple targets that I need)
• The arguments that are passed to the <main or makes>.nixs (to be able to pass import ./makes/service.nix)

My understanding is that this is not possible currently, and instead any such service.nix would have to be declared as a builtin in makes itself, and then I would put only the configuration of that in some makes.nix, which also means I could only instantiate one of each of those per makes.nix, whereas my ideal syntax would probably be:

# /project1/service1/<main or makes>.nix
{ args, service, pkgs, ... }: service { name = "a"; }

# /makes/service.nix
{ args, pkgs, makeScript, deployContainerImage, outputAttrSet, ... }:
let
  build = { /* ... */ };
  docker = pkgs.dockerTools.buildLayeredImage { /* ... */ };
  # Where args are the arguments pased as [ARGS...] to the CLI and may be used here
in
outputAttrSet {
  inherit build;
  deploy = deployContainerImage {
    images = [{
      src = docker;
      registry = { /* ... */ };
    }];
  };
  test = { /* ... */};
  run = makeScript { /* ... */ };
}

(where outputAttrSet would convert from the attrset of output to probably [{ name, derivation }], allowing for further customization if required)

Is it actually already possible to achieve something like this? If so, how? If not, is this something that could maybe be made possible in the future or are there any clear blockers?

Thanks,

[dsalaza4]

Hey @Ten0,

For main.nix files, you can set extendingMakesDirs = ["/"]; as described in the documentation. With this setting Makes will name CLI jobs according to absolute paths within the repository. That is, for file /project1/service1/run/main.nix, the CLI output would be /project1/service1/run.

With makes.nix things are a little more complicated. makes.nix is an interface between:

1. evaluators: default configurations that builtins receive (This is what you declare in makes.nix.
2. arguments: Makes' builtins.

This is why, when declaring builtins from makes.nix files, CLI outputs do not follow a naming pattern based on directory structure like main.nix files but rather group all jobs for the same builtin.

Example:

For the following makes.nix file

{
  formatPython,
  ...
}: {
  formatPython = {
    project1.targets = ["/project1"];
    project2.targets = ["/project2"];
    project3.targets = ["/project3"];
  };
}

We get the following CLI outputs:

/formatPython/project1
/formatPython/project2
/formatPython/project3

With some boilerplate code you could still use all builtins from main.nix files, granting directory-based CLI outputs.

Example:

/project1/service1/format/main.nix

{
  formatNix,
  makeScript,
  projectPath,
  ...
}: let
  bin = formatNix {
    name = "project1-service1";
    targets = [(projectPath "/project1/service1")];
  };
in
  makeScript {
    entrypoint = "format-nix-for-project1-service1";
    name = "project1-service1-format";
    searchPaths.bin = [bin];
  }

From that main.nix you can do anything that nix allows so you can orchestrate everything from a makes/services.nix file:

/project1/service1/format/main.nix

{
  formatNix,
  makeScript, 
  projectPath,
  ...
}: let
  config = import (projectPath "/makes/services.nix") {
    inherit formatNix;
    inherit makeScript;
    inherit projectPath;
  };
in
# customFormatNix is a more generic function that runs formatNix as explained before
config.customFormatNix {
  name = config.project1.service1.format.name; 
  targets = config.project1.service1.targets;
}

[dsalaza4]

You could even declare everything in the services.nix file and simply import it from the main.nix:

makesArgs: let
  config = import (makesArgs.projectPath "/makes/services.nix") {inherit makesArgs;};
in
config.project1.service1.format.run

[Ten0]

Thank you for your answer! 🙂

So it looks like my understanding was correct: to generate target paths, I'm tied by either the builtins (which I can't customize) or the "one folder with a main.nix per output" approach (which would force me to create one directory per output with a main file that would specify a bunch of boilerplate and finally .target_name(=folder_name), which woudn't fit at all with my current directory structure or be one I'd like anyway) 🙁

Is there any technical constraint that prevents building a system that has a more flexible interface similar to the one I described above? Having the root expression that evaluates to just at-most-one-incantation-of-each-type-of-builtin doesn't allow for flexible program structure compared to something along the lines of [{ name, thing_to_do }].

At first glance it looks like we could evaluate only the part of the current folder's expressions that are related to outputs names when starting the CLI, and then only when actually building, evaluate the derivation attribute of each { name, derivation } that the root expression outputs, but there may be big drawbacks that I'm missing.

Would a design where we add an outputs builtin that takes as parameter [{ name, derivation }] work or is there some constraint that prevents passing such complex inputs to builtins? (like, they have to get evaluated before they get passed or something along those lines)

To be very clear: I love the idea of the project of having a CI system globally powered by nix, with a CLI that enables choosing the targets we're interested in, especially making full use of caching across all users of the organization for every single derivation. The potential of it looks insane! However, if it seems that the only way to efficiently add outputs to the CLI is to make incantations in the makes.nix after PR-ing a new builtin to this repository, then it seems pretty clear to me that that doesn't scale unless you happen to also be the company that manages the Makes project: we can't have every company use this and PR every single customization they need as a global builtin in the project:

• Some builtins would be too custom for integration.
• The integration friction would be too high.
• That's too many global builtins and it becomes an absolute mess. (Actually there's already too many global builtins dedicated to tools and languages I don't use for my taste)

Thanks a lot 🙂

[dsalaza4]

I will start working on a first iteration that allows to declare jobs = {} within makes.nix so we can begin moving away from main.nix files.

[dsalaza4]

@Ten0 Hey! If you're still interested, take a look at the new jobs = {} essential. It basically allows to describe jobs directly in makes.nix files. https://makes.fluidattacks.tech/configuration/essentials/#jobs