Performance of Reads on Draft

[kevglass]

As part of an SDK I'm working on I provide a draft version of user provided data structure back to them to be updated. I want to maintain the immutable status of the original data so that I can compare it safely later on.

However, in this case the user has provided a reasonably large data structure that represents a physics engine. The performance of making changes to the mutative draft is understandably not as fast as a raw object - however, the performance of reads of properties of the draft seem to be impacted too.

To test this out I've put together a simple standalone test case over here: https://github.com/kevglass/mutative-performance-sample/ - it's important to note that the create() is intentionally inside the loop since in the real system the draft is created every frame.

It simulates a collection of balls (30) on a table moving in random directions and colliding. The performance test can be run in two ways - either with writing to the draft object (the same as it would be in a real physics engine) or in read only mode where the simulation is just calculating some values based on the contents of the draft objects.

I feel like I must be doing something wrong but I can't quite understand what it is. The results on my M1 for read only access to the draft object looks like this:

2024-03-11T21:23:43.254Z
Iterations=5000 Balls=30 ReadOnly=true

RAW     : 5000 iterations @12ms  (0.0024 per loop)
RAW+COPY: 5000 iterations @254ms  (0.0508 per loop)
MUTATIVE: 5000 iterations @3709ms  (0.7418 per loop)
IMMER   : 5000 iterations @4309ms  (0.8618 per loop)

Where RAW is a simple JS object, RAW+COPY is taking a copy of the object at each stage (parse/stringify). Mutative is the lovely library here and Immer for comparison.

I hadn't expected the impact of reading from the draft to be so high, so i'm guessing I've done something very wrong.

Any thoughts or directions appreciated.

For completeness heres my read/write results from my M1:

RAW     : 5000 iterations @14ms  (0.0028 per loop)
RAW+COPY: 5000 iterations @270ms  (0.054 per loop)
MUTATIVE: 5000 iterations @4813ms  (0.9626 per loop)
IMMER   : 5000 iterations @5430ms  (1.086 per loop)

[unadlib]

hi @kevglass , thanks for your feedback. In fact, even though Mutative has implemented lazy drafts, its access mechanism is still based on ES6 Proxy. As a result, achieving true caching during large-scale draft reads can be quite challenging.

When it comes to read-only operations, we strongly recommend using current(draft) to obtain the non-draft values for tasks such as iterating over original values. This approach will significantly boost performance.

In my tests, the comparison between not using current() and using current() is:

RAW     : 5000 iterations @14ms  (0.0028 per loop)
RAW+COPY: 5000 iterations @228ms  (0.0456 per loop)
MUTATIVE: 5000 iterations @3454ms  (0.6908 per loop)
IMMER   : 5000 iterations @4007ms  (0.8014 per loop)

VS

RAW     : 5000 iterations @13ms  (0.0026 per loop)
RAW+COPY: 5000 iterations @223ms  (0.0446 per loop)
MUTATIVE: 5000 iterations @54ms  (0.0108 per loop)
IMMER   : 5000 iterations @4285ms  (0.857 per loop)

[unadlib]

If we were to implement a simple draft WeakMap cache, then indeed, the read performance could be improved by about 40%. I am currently considering whether to implement such an optimization (there may be other factors that need to be considered as well).

[kevglass]

Hey, thanks a lot for the quick reply!

In the case I'm working on we won't know whether the developer will be making read only use of the draft or writing to certain pieces. It's likely they'll be doing a lot of reads and some small number of modifications (at least thats the expectations). I don't think I can pass them a current(draft) in this case. I'll have a think about it too.

[unadlib]

In the same code example, I implemented a pure object tree based on deep Proxies, which still performs slowly without any other draft code involved. It possibly executed the Proxy getter function 40 million times. Therefore, the performance of Mutative draft reads is challenging to match that of RAW+COPY.

RAW      : 5000 iterations @14ms  (0.0028 per loop)
RAW+COPY : 5000 iterations @244ms  (0.0488 per loop)
MUTATIVE : 5000 iterations @2293ms  (0.4586 per loop)
RAW+PROXY: 5000 iterations @1514ms  (0.3028 per loop)
IMMER    : 5000 iterations @3850ms  (0.77 per loop)

[bfelbo]

When it comes to read-only operations, we strongly recommend using current(draft) to obtain the non-draft values for tasks such as iterating over original values. This approach will significantly boost performance.

This seems very promising, thanks for providing a comparison that shows the substantial performance difference!

There's not that much info in the README on current(draft). Are there any downsides or anything to be aware of with this approach if we're using it for read-only operations?

[unadlib]

hi @bfelbo , the current() function returns the current state of a modified draft.

• For any draft where a child node has been modified, the state obtained by executing current() each time will be a new reference object.
• For a draft where no child nodes have been modified, executing current() will always return the original state.

Therefore, we recommend minimizing the number of times current() is executed if using read-only operations, ideally executing it only once.

create({ a: { b: { c: 1 } }, d: { f: 1 } }, (draft) => {
  draft.a.b.c = 2;
  // The node `a` has been modified.
  expect(current(draft.a) === current(draft.a)).toBeFalsy();
  // The node `d` has not been modified.
  expect(current(draft.d) === current(draft.d)).toBeTruthy();
});

[unadlib]

You can find more documentation on the current() API here.

[bfelbo]

Thanks @unadlib, really appreciate the detailed explanation and docs!

[bfelbo]

In the same code example, I implemented a pure object tree based on deep Proxies, which still performs slowly without any other draft code involved. It possibly executed the Proxy getter function 40 million times.

This makes sense. I find it quite interesting though. Would you be able to share your code?

(no need to clean it up or anything like that)

[unadlib]

hi @bfelbo , based on the implementation of this PR(#42), the performance comparison is as follows.

Before

RAW     : 5000 iterations @13ms  (0.0026 per loop)
RAW+COPY: 5000 iterations @231ms  (0.0462 per loop)
MUTATIVE: 5000 iterations @3270ms  (0.654 per loop)
RAW+PROXY: 5000 iterations @1504ms  (0.3008 per loop)
IMMER   : 5000 iterations @3806ms  (0.7612 per loop)

After

RAW     : 5000 iterations @13ms  (0.0026 per loop)
RAW+COPY: 5000 iterations @231ms  (0.0462 per loop)
MUTATIVE: 5000 iterations @2093ms  (0.4186 per loop)
RAW+PROXY: 5000 iterations @1547ms  (0.3094 per loop)
IMMER   : 5000 iterations @3803ms  (0.7606 per loop)

[bfelbo]

Thanks, and cool that you improved performance further!

[unadlib]

We have released Mutative v1.0.6. Feel free to use it.