Multiple post services per node - multiple ATX

[pigmej]

Motivation

To support:

• multiple drives - one POST service per drive
• lower needs of maintenance of multiple go-spacemesh nodes

We need to support multiple post services per node.

This epic describes the state with supporting multiple keys in single go-spacemesh instance, there is separate discussion with research how we could make protocol changes to use single consensus key, single atx but multiple niposts.

Tasks

After #259 is completed (or in an acceptable enough state) we will need the following:

• beacon: support multiple smeshers go-spacemesh#5088
• hare3: support multiple smeshers go-spacemesh#5085
• cert: support multiple smeshers go-spacemesh#5086
• configuration to load multiple smeshers from disk go-spacemesh#5089
• tortoise: support multiple smeshers go-spacemesh#5087
• activation: support multiple smeshers go-spacemesh#5149

Details are in the tasks already.

[HVNC1]

a single ath carries the risk of losing rewards for all post files in case of failure, while an ath for everyone seems safer at first glance

[HVNC1]

it would also be good to consider the balance of processor power if the files are of different sizes and for example start the phase using the CPU first for the largest files so that they have the most time to read proofs

[Stizerg]

I will add my entire suggestion from https://discord.com/channels/623195163510046732/1151465823710494740

1. Configurable delay for k2pow stage. If we know for how long the CPU will be busy by another node we can set it in config.
   e.g. "pow_delay": 3600
   ~
2. We need each node to indicate what it's doing atm, it can be done a similar way as --filelock.
   For example, if we have 3 nodes:
   node1: 5TiB on e:
   node2: 4TiB on f:
   node3: 1TiB on e:
   ~
   node1 starts it poet gap's routine, it sets "pow_lock_node1"
   node2 and node3 are waiting (considering they have "pow_delay" set)
   node1 starts to read POS, releases "pow_lock_node1" and sets "read_lock_node1"
   node2 detects it and starts it's routine, setting up "pow_lock_node2"
   node3 is waiting
   node2 starts to read POS, releases "pow_lock_node2" and sets "read_lock_node2"
   node3 detects it and starts it's routine, setting up "pow_lock_node3"
   when node3 is ready to start reading POS it will wait for "read_lock_node1" to be released

Depending on configuration the logic can be different.

So, the target is to use resources of computer the most effective way.

[pigmej]

This feature debuted with the 1.4 series.

There is a demo available at: https://github.com/spacemeshos/multiple_post_services_demo and docs at https://docs.spacemesh.io/docs/start/smesher/post_1n

The remaining functionality would be to dynamically reconfigure the identities: spacemeshos/go-spacemesh#5090