multiple instances of hltd service

[smorovic]

There are requirements to save on hardware resources by running, for example, multiple minidaq systems while using common hardware. Since hltd is a system service, in current form it does not support instantiation.

This schema is proposed:

• running multiple instances of hltd is possible on BU. FU only runs one instance at the time.
• instances are set in an input file (later possibly HwCfg database) shipped with meta rpm, which takes care of setting up the software.
• list of installed instances is written to /etc/hltd.instances by the meta rpm.
• "main" instance, if installed, keeps using same structure as current version of hltd
• other instances use modified config file: /etc/hltd-$instance.conf and logs go to /var/log/hltd/$instance. pid and lock files also have a separate name,
• port is modified (in range [9000,9010]) based on the input file.
• non-main instance reads from ramdisk subdirectory which is a loopback file system mouting an image file with predefined size. in this way, ramdisk quota of each instance is enforced.
  -FUs have instance name specified in hltd.conf and read input from remote ramdisk subdirectory. boxinfo files are written to the same subdirectory. output is written either to normal BU NFS output, or a subdirectory in NFS mount (this is not yet decided).
• except in these described differences, FUs behave like a main instance. A single FU is assigned to one and only one BU hltd instance, also as described in input config. file.
• local nor central elasticsearch clusters are not divided into instances. This assumes no duplicate run numbers between the systems (i.e. using same RunInfo DB).
• init scripts needs improvement to allow starting/stopping selected instance.

Experimental support for this feature is implemented here:
https://github.com/smorovic/hltd/tree/multi-instances%2Bcloud