Detailed documentation about cluster deployment

[oculos]

Description:

The documentation of Synapse workers are very centered about workers on the same host, and about increasing the number of workers to improve performance.

The issue is that it is left to interpretation on how that translates to deployment into a cluster like Kubernetes.

The documentation is clear about:

• workers which should not be duplicated
• workers that can have simple load balancing (client, federation - though apparently not all types)

All the examples I find are about specifically creating a new worker, for example, federation-sender1 and federation-sender2, instead of worker replicas, which is more of a pattern to Kubernetes.

Provided that care is taken when doing load balancing for the special endpoints like sync, initial sync, etc, is it acceptable to run a few workers (for example, multiple federation workers for incoming federation) as replicas, thus being seing/named on the main configuration as just one worker (since the replicas would be back a service), or is there a reason why one would need to deploy separate, equal but differently named workers?

If this is possible, shouldn't it be mentioned on the documentation?

Right now, it is unclear that, if I want to add multiple workers for the federation endpoint, I need to create multiple workers and add each of them on an instance map, or if I could simple add the service hostname on the instance map, and add as many replicas I see fit.

I'm not sure if this is possible in case the main worker needs to address each individual worker node, but if so, the code to provide scalability for some worker pods would be greatly simplified.

So instead of having these general workers:

• federation_worker1
• federation_worker2

I'd simply have federation_worker, which is the hostname of a service that has lots of pods with that worker.