NATS JetStream Sharding

[c1moore]

I don't know if this is a suggestion or a discussion topic, but I've been trying to determine ways to shard streams across a cluster to spread out load across the cluster.

According to the community, NATS itself does not try to spread streams across different instances. It basically chooses the instances that respond the fastest, which may or may not implicitly spread streams out. It seems to me that for this use case (sharding), that wouldn't happen since chances are you're creating all the streams around the same time and there probably aren't any consumers yet.

So the next suggestion is to leverage tags. What I'd like to end up with is something like this

However, I haven't found a way to consistently generate tags in an automated fashion in Kubernetes, especially using the official NATS helmchart, that doesn't require an external service/state just to coordinate the instances (e.g. "you are the second instance in AZ 1, you will be assigned group 2"). This becomes much more complicated if you start considering the potential for instances to go down and multi-AZ replication. In the end, I'd rather not maintain something external just to tag NATS servers.

So, what's the recommended strategy for having a setup like this? Alternatively, the problem goes away if NATS could direct streams to instances that have the fewest streams already by default (and let this be overridden by tags).