Feature/filter converged

[mdbenito]

Description

This PR closes #303

Changes

• Adds an additional parameter to compute_generic_semivalues that allows skipping samples for indices that have already converged
• Improves documentation and warns about that parameter (it's tricky)

Unrelated:

• Deprecates a custom attribute in StoppingCriterion in favour of __str__

Checklist

• Wrote Unit tests (if necessary)
• Updated Documentation (if necessary)
• Updated Changelog
• If notebooks were added/changed, added boilerplate cells are tagged with "tags": ["hide"] or "tags": ["hide-input"]

[kosmitive]

Can you elaborate, why this parameter is necessary? Instead of doing that, we can also try to exclude marginal evaluations for converged indices, by removing them from the task queue.

[mdbenito]

Can you elaborate, why this parameter is necessary? Instead of doing that, we can also try to exclude marginal evaluations for converged indices, by removing them from the task queue.

Yes, sorry. This is exactly what is done. The parameter is necessary because just making it the default would change the behaviour for users. Also, very often values would converge only because of "unexpected" additional marginal computations.

This is something that happens often: we do AbsoluteStandardError(0.02) | MaxUpdates(1000). Then some values might fulfill the first criterion really fast, despite being very bad estimates. Because the other indices take much longer to have low stderr and done(result) is a global check, the "converged" ones keep being updated and end up being good estimates. If we stopped updating them, the values would be off by a lot. The way to fix this is by making the stderr threshold much lower, e.g. AbsoluteStandardError(1e-3) | MaxUpdates(1000). With skip_converged=True this more stringent check still takes less time than the first one.

[mdbenito]

Thanks @kosmitive for the comments. Implementing a couple

[mdbenito]

This is something that happens often: we do AbsoluteStandardError(0.02) | MaxUpdates(1000). Then some values might fulfill the first criterion really fast, despite being very bad estimates. Because the other indices take much longer to have low stderr and done(result) is a global check, the "converged" ones keep being updated and end up being good estimates. If we stopped updating them, the values would be off by a lot. The way to fix this is by making the stderr threshold much lower, e.g. AbsoluteStandardError(1e-3) | MaxUpdates(1000). With skip_converged=True this more stringent check still takes less time than the first one.

Added something along these lines to the docs for stopping.py