Dynamically bridged provider replay tests cleverer log matching strategy

[VenelinMartinov]

What happened?

The replay tests for dynamically bridged providers use a simple strategy of matching logs to requests by assuming the order is consistent.

We can do something cleverer if we need to like matching the inputs.

See #2484 (comment)

Example

.

Output of pulumi about

.

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

I would push on this. It does not have to be clever, the default strategy can match on request inputs. The current strategy matches requests based on the request type and random ordering position is not acceptable and will lead to lost time failing to reproduce issues and debugging non-deterministic replay request ordering.

For more complicated cases there probably should be a hook, CustomMatchers or some such, for the owner of a concrete test scenario to add some code to customize how the logs are matched to the incoming requests. This seems to be a common refrain in libraries that are build this way, see for example:

https://github.com/dnaeon/go-vcr?tab=readme-ov-file#custom-request-matching

[VenelinMartinov]

Yeah, that sound sensible. Do we have an example of when this will be needed? Something around multiple resources created in parallel perhaps?

[VenelinMartinov]

I would also like to see the framework get any usage, frankly, before investing more here. It's currently only there as a proof of concept

[t0yv0]

This will be needed any time you have anything non-trivial. The requests aren't coming in-order.

[t0yv0]

ACK.