Pull dedupe out of queue and into personality mixin #333
Conversation
mhutchinson
force-pushed
the
b331-dedupe
branch
3 times, most recently
from
41b7091
to
f8f3244
Compare
This allows the deduplication behaviour to be more precisely controlled by the personality. Previously deduping was always performed within a single batch flushed from the queue, and this could not be disabled. Now, deduping can be disabled, or made to apply across batches if a large enough cache size is set. The conformance binaries have been updated to use this new mixin, as this is the closest to a no-op change I can make. We may want to look at making the cache size configurable and see how this affects performance. This fixes transparency-dev#331.
mhutchinson
force-pushed
the
b331-dedupe
branch
from
f8f3244
to
1807426
Compare
|
There's already some deduplication code that was written for CT, it's Best Effort deduplication that eventually converges on always returning the smallest index possible for a submitted leaf. The plan is to take what we can out of the static-ct repo later on, and put it back in this tessera repo. This PR will change how to reason about this, both for in terms of how we align the existing and new deduplication APIs, and what happens if both deduplications are chained together (I think it will make convergence properties slightly different). Sounds good to me to move forward with it if you want this out before Alpha, and we can re-think how the personality mixins work with one another later? |
Great I'll merge this if I can get an approval. Until we hit 1.0 all parts of the API are still open to change. Ideally the core parts will change less, and the mixins are more open to change. So sounds like we're aligned that this mixin works for now but can be changed if needed. FWIW I really like the approach I've taken here of making the mixins that modify the |
This allows the deduplication behaviour to be more precisely controlled by the personality. Previously deduping was always performed within a single batch flushed from the queue, and this could not be disabled. Now, deduping can be disabled, or made to apply across batches if a large enough cache size is set.
The conformance binaries have been updated to use this new mixin, as this is the closest to a no-op change I can make. We may want to look at making the cache size configurable and see how this affects performance.
This fixes #331.