Multiple processes of the same service (auto-spawning)

[ronaldtse]

Some processes can be started more than once independently, such as worker processes.

systemd can do this with a minor "hack", where you specify a foo.target and a [email protected]. When you enable 3 foos, do this:

systemctl enable [email protected]

Example below.

foo.target

[Unit]
Description=foobar
After=syslog.target network.target remote-fs.target nss-lookup.target

[Install]
WantedBy=multi-user.target

[email protected]

[Unit]
Description=foobar Process %I
;PartOf=foobar.service
;ReloadPropagatedFrom=foobar.service

# Stop this process when foobar.target is stopped
StopWhenUnneeded=yes

# start us only once the network and logging subsystems are available,
# consider adding redis-server.service if Redis is local and systemd-managed.
Requires=export-systemd-envs.service
After=syslog.target network.target nss-lookup.target export-systemd-envs.service

[Service]
Type=simple
WorkingDirectory=/opt/ribose/src/indigo
PrivateTmp=yes
User=root
Group=root
UMask=0002

# Run ExecStartPre with root-permissions
PermissionsStartOnly=true
ExecStartPre=-/usr/bin/mkdir -p /var/log/foobar /var/run/foobar
ExecStartPre=/usr/bin/chown -R nobody:nobody /var/log/foobar /var/run/foobar
PIDFile=/var/run/foobar/foobar-%I.pid

EnvironmentFile=/etc/ribose-service.env

# foobar reads RAILS_ENV
ExecStart=/usr/bin/bash -c "foobarctl start /var/run/foobar/foobar-%I.pid -L /var/log/foobar/foobar-%I.log"

# Use default 8 second timeout with SIGTERM
ExecStop=/usr/bin/bash -c "foobarctl stop /var/run/foobar/foobar-%I.pid 10 >> /var/log/foobar/foobar-%I.log 2>&1"

# if we crash, restart
RestartSec=5
Restart=on-failure

# output goes to /var/log/syslog
StandardOutput=syslog
StandardError=syslog

# This will default to "bundler" if we don't specify it
SyslogIdentifier=foobar

# The pixie dust that allows multiple processes to be started
[Install]
WantedBy=foobar.target

[drystone]

In order of increasing complexity:

applications = [
 foo
 foo
 foo
]

or

instances=n

Maybe both or something else? In systemd the @ works a bit like templates so the processes can vary slightly. Perhaps we could have something like:

applications = [
    foo@1
    foo@2
    foo@baz
]

And in the application section we could have something like env INSTANCE=%I

[ronaldtse]

Terraform uses the count = X syntax: https://www.terraform.io/docs/configuration/interpolation.html

1. instances = n is something we should do, for identical processes.

2. Interpolation per instance -- in Terraform they use a separate variable array/map to do this. For example we could do:

let application_configs = {
  1 = {
    env { ID = "${self.instance_count}-foo" }
    env_file = "/etc/foo/first-instance"
  }
  2 = { env { ID = "${self.instance_count}-bar" } }
  3 = { env { ID = "${self.instance_count}-foobar" } }
}

application "foo"  {
  instances = application_configs
}

It really seems that we should come up with a proper language to handle these things? @drystone thoughts?

[drystone]

That seems a good solution - this config would sit well in the application-group stanza:

application-group "foo" {
    applications [
        application "bar" {
            instances = 4
        }
        application "baz" {
            instance "1" {
                # per-instance overrides
                env {
                    "ID" = "1"
                }
                pidfile = "/var/run/baz.1.pid"
            }
            instance "2" {
                # ...
            }
        }
    }
}

Currently ${self.instance_count} isn't supported. It would be possible to add some form of reflection so substitutions could access things like node-name, node-offset in parent, node-count etc, I think, especially as Riffol configs are small static datasets and these values can be explicitly created, we could add this to the list of enhancements.

[ronaldtse]

@drystone the latest suggestion seems to be too enclosed, perhaps we can replace it with "relational" objects like this?

application-group "foo" {
  applications [
    "${application.baz}",
    "${application.bar}",
  ]
}

application "baz" {
  instance "1" {
    # per-instance overrides
    env {
    "ID" = "1"
    }
    pidfile = "/var/run/baz.1.pid"
  }
  
  instance "2" {
    # ...
    }
  }
}

application "bar" {
  instances = 4
}

[drystone]

@ronaldtse, with the current nereon syntax we can do the following (except concat() doesn't exist yet...):

let(baz, {
    pidfile concat(/var/run/baz., arg(0), .pid)
    env set ID arg(0)
})

let(bar, {
})

application-group foo {
    applications [
        baz1, baz2
        bar1, bar2, bar3, bar4
    ]
}

application baz1 apply(baz, 1)
application baz2 apply(baz, 2)
application bar1 $bar
application bar2 $bar
application bar3 $bar
application bar4 $bar

Do you think would be sufficient to close this issue?