Message prioritization to certain consumers

[plaisted]

I have a data processing application that runs in a kubernetes cluster that uses NATS Jetstream work queues and pull subscriptions. As it stands now the consumer that gets a message is relatively random (combination of round robin from NATS and timing of an open worker). I'm looking for a way to promote data locality, in this case when a message is published it would prefer to be read by a worker on the same node (so that a local cache would be used to read the results of the last step) but if there are no local workers available it could be pulled by a worker on another node.

I could publish to node specific topics (eg. work.{node name}) and have the stream read all the topics for the work queue but don't see a way to get it routed locally if possible without writing some sort of custom message broker to route the work around. Is there some existing functionality that could support a use case like this that I've missed?

[derekcollison]

Can you describe the state that is being read, the one you want locality for? Is it a file read? A Key Value pair?

[plaisted]

They are files persisted to s3 but also cached locally on the node SSD. The messages include the s3 key but grab the local cached file if it exists.

[derekcollison]

Maybe consider using JetStream KV or Object store and have realtime mirrors on all the nodes.

These can be configured to expire after a certain amount of time or when new entries are added.

[plaisted]

Interesting suggestion, my original intent for the prioritization to certain consumers was to avoid copying the data around (relatively large files) and paying for AWS network traffic / S3 traffic. However, if NATS is moving the data locally to an availability zone that's free and fast. Don't think it's a great fit for our use case (files need to be mmap'd so would need to be copied locally again anyway) but good to think about.