Retry policy source of truth: op definition vs RetryRequested object

[dagsir[bot]]

The question was originally asked in Dagster Slack. Reposting it here for posterity and discoverability.

The above Slack thread uncovered a confusing aspect of the API around op retry policy. There is an ambiguity between the max_retries argument set on RetryRequested exception objects and the RetryPolicy set in an op definition. Present behavior is that manually created RetryRequested objects ignore the op's RetryPolicy (which is used for automatically created RetryRequested objects). This is potentially confusing-- should we change this behavior?

cc @alangenfeld

[cbini]

Now that I know RetryRequested has it's own params, I think the priority should go:

1. RetryRequested manual settings
2. Op RetryPolicy manual settings
3. RetryRequested defaults
4. Op RetryPolicy defaults

3 & 4, I'm not sure about. Depends on which is more intuitive

[cbini]

Tangent: in RetryRequested the param is called seconds_to_wait, and in RetryPolicy the param is called delay. These should match.

Also, RetryRequested doesn't appear to have backoff and jitter params, so it's unclear if they inherit them from the Op RetryPolicy or not.