Reduce wait time to deploy

[brandonpille]

Once #473 will be implemented nodes will be shut down whenever possible. This currently means that we at least keep one manager node per LAN per farm. That node will bring the other nodes up when needed. So the node will bring another node up if the user creates a capacity reservation contract that requires more resources than available on the nodes that are currently up. This brings us to the topic of this discussion. Whenever a new node has to be brought up the user will have to wait some time. We could reduce this wait time by keeping one extra node up that is not being used in any contracts (no used resources). Then, whenever a new contract is created requiring us to start using that node we bring another node up which will take its place (no used resources). This will go on until all nodes are up. Bringing nodes down will be the same but the other way around. Then we still have the choice of the node that we bring up in the case mentioned before. I would suggest choosing the node that has the most resources so that in most cases the contract will fit on that node.

@DylanVerstraete, @DylanVerstraete, @coesensbert, @muhamadazmy, @LeeSmet any thoughts?

[coesensbert]

My concern is mainly from the users side. Correct me if I'm wrong, but I imagine a scenario like this.

A user is given a node or manually selects a node that has been shut down. He deploys a workload:
~ 2 to 3 min for POST
~ 2 to 3 min ZOS img download and boot
~ 1 to 2 min ZOS setting it's networking
~ 20 to 30 sec for workload flist download
~ 1 to 2 min for actual deployment and boot of the workload

So best case that would be: ~6 min 30 sec for a deployment to be ready
A worst case scenario could be: ~10min 30 sec

So I fear a worse user experience in exchange for lower power consumption.

[brandonpille]

I still see an issue when choosing which node to keep up: there are cases where some nodes have more of a specific type of resource and less of another type of resources (more cores but less ram for example). To solve that issue we could keep a node up (that is not being used) for each type of resource, where that node has the most amount of that type of resource. So in the worst case we keep 4 nodes up (on top of the ones being used) and in the best case we keep 1 node up (on top of the ones being used).

We properly communicate this to the farmers to minimize the amount of nodes that we keep up (for nothing):

• try to create nodes that have the same resources
• make sure that the types of resources (cu, su, hu, mu) are in proportion on each node (avoid having nodes with a lot of cores but very little ram for example)

[DylanVerstraete]

try to create nodes that have the same resources

We cannot enforce this

make sure that the types of resources (cu, su, hu, mu) are in proportion on each node (avoid having nodes with a lot of cores but very little ram for example)

Current tokenomics punish such configuration by rewarding less tokens. Farmers by now know how they should build a node to have the maximal amount of tokens rewarded which is in line with a healthy amount of ram / ssd / cpu per node.

Here is another suggestion, if for example one out of x nodes is up in a farm. Whenever this node fills up like 75%, we power on an additional node. This way, a farmer is not punished when his farm is not used (because it means 2 nodes will be up the entire time).

[brandonpille]

We thought of that too. But how do you define 75%? Is it 75% of cpu, 75% of ssd, 75% of ram and 75% of hdd? Or is it 75% of all 4 resources (no matter what type)? It's a great idea but I think we will have much more cases where the user has to wait till the machine is powered on with this idea.

[brandonpille]

I guess we have to choose between minimal time to wait by the client and minimal amount of nodes that are ON without being used.

[DylanVerstraete]

We can calculate cu/su and when these 2 are filled for 75% we boot another node.