Make it possible to keep workflows in worker cache also if they're only queried

[recht]

Is your feature request related to a problem? Please describe.

Currently, when sending a query to a workflow that is in running state but not active on any worker, a worker will receive the request, replay history, process the query, and forget everything again. On next query the same thing happens. This is in contrast to starting the workflow from scratch without restarting the worker. Then the workflow is in the worker cache, ready to serve queries. Same if the worker has restarted, but workflow state has changed so that the workflow got loaded into the cache.

Because replays can get a bit expensive, especially if encryption/offloading codecs are used (which we do), it would be nice if querying a workflow would add it to the worker sticky cache.

Describe the solution you'd like
A config property that allows us to specify that workflows that are queried should also be inserted into the cache.

While debugging locally I did this:

diff --git a/internal/internal_task_handlers.go b/internal/internal_task_handlers.go
index 8312587..342f7f4 100644
--- a/internal/internal_task_handlers.go
+++ b/internal/internal_task_handlers.go
@@ -812,7 +812,7 @@ func (wth *workflowTaskHandlerImpl) GetOrCreateWorkflowContext(
                        return
                }
 
-               if wth.cache.MaxWorkflowCacheSize() > 0 && task.Query == nil {
+               if wth.cache.MaxWorkflowCacheSize() > 0 {
                        workflowContext, _ = wth.cache.putWorkflowContext(runID, workflowContext)
                        workflowContext.Lock()
                        workflowContext.cached = true

It's unclear if that's the right solution or if that has other unintended consequences, but at least it keeps the workflow in the sticky worker cache, making queries much faster.

Describe alternatives you've considered
Doing less queries, or faster codecs. Regardless, it seems wasteful to have to replay on every single query.

[Quinn-With-Two-Ns]

I believe this is a limitation on the server. Query only tasks cannot set the sticky cache attribute so the worker cache would not actually be used even if the SDK did keep it in cache. There is an issue to add this feature on the server side, once this is closed the SDK could keep the workflow in cache.

[recht]

Ok, thanks - so I guess the only reason why it made any difference for me was that I was only running a single worker so queries always went to that worker?

[Quinn-With-Two-Ns]

There are multiple ways a query can be delivered to a workflow, if the worker was already in the cache it will be used. The problem is direct queries cannot add the workflow to the sticky task queue so the server would always send the whole history down again causing a replay.

[recht]

Ok, but with the change I tried there was no replay after the first query - it's possible that the worker receives all the data, but it doesn't actually replay (checked by adding print statements in the workflow code and observing with and without the change)