Sharding Jetstream

[maxpert]

First and foremost I am in instant love with the project. I have been implementing a Marmot and one of things I was spending a lot of time on was distributed reliable streams of change log. While NATS solve most of my problems, I am still struggling with two things:

• For a given JetStream when I am setting it up how can I make sure that when I am publishing something on the stream only the nodes are responsible for storing the stream information and replicas are involved. And it doesn't add any extra load to Raft leader? Another way to ask the question this would be for publishing/committing a log entry to a JS is the Primary only involved in holding meta information about who to talk for primary and replica committing or primary is required throughout the process?
• What is the best recommended way of sharding JetStreams? Right now I am manually creating specific JetStreams with specific subjects attached to them, and then only publishing to those subjects. The main problem I am trying to solve is sharding change data log coming out of SQLite among nats cluster nodes.

[derekcollison]

If you want only jetstream to receive a message then you can restrict the subject (authz) for all other users/apps.

The leader of the stream receives the message directly from the app and is in charge of replicating it to the followers. Once a quorum of replies has been received by the leader it will respond to the application.

In terms of sharding, are you sure you need it? A single stream, even an R3 can run ~250k msgs/sec with proper hardware and setup. If you feel you do, you can have messages in your account be sharded automatically, but the streams would need to be setup to receive this partitioned messages.

I thought docs had that but a simple search could not find it so looping in @jnmoyne and @gcolliso and @bruth. Search was subject mapping which got me to a section but no results, then searched for partitioning which directed me to STAN which is not what I was looking for..

[maxpert]

Thanks for taking time to answer. From sharding perspective yes I do need it because I am designing Marmot for WAN and cheap commodity hardware (My current production setup costs me $14 for a year for single node, with TBs of bandwidth). I did the testing already on my local-hardware, and you are right it can go pretty fast. I would keep you posted with Demo once I've uploaded it here. Thanks for awesome work you guys have done, I really like NATS want to also put a badge on my repo saying something along lines of "Powered by NATS".

[bruth]

Hi @maxpert. The partitioning section is on the subject mapping page. The general strategy is that you choose N partitions and then define a subject mapping in the server config.

You can test out the mapping using the NATS CLI:

$ nats server mappings "test.*" "test.{{wildcard(1)}}.{{partition(7,1)}}" "test.5"
test.5.3

You can see the behavior over a number subjects.

for i in $(seq 1 10); do
  nats server mappings "test.*" "test.{{wildcard(1)}}.{{partition(7,1)}}" "test.$i"
done

test.1.2
test.2.3
test.3.5
test.4.1
test.5.3
test.6.4
test.7.6
test.8.5
test.9.0
test.10.5

From here you would define a stream per partition, by defining the filter subject like test.*.0 for the first partition.