Feature Request: Individual Consumer Test Case per Async Message Interaction

[timvahlbrock]

TLDR: Having a single test case for multiple message types to the same parser cause fast failing of the test and make it hard to determine, which message was faulty. It would be easier to have one test per interaction, which is currently only possible when defining that manually.

Assuming I got that right, the currently recommended way to pass multiple interactions of the same message parser by referencing all interactions described by @Pact using the pactMethods annotation. The messages will then be passed as a list to the @PactFor method. This has two major consequences for failed parsing.

1. The test is stopped as soon as the first message fails to parse. This can be circumvented by Framework tools like JUnit's assertAll, but isn't very nice nevertheless.
2. It is very hard to detect which message failed to parse, as long as that isn't inferable from the stack trace. There is also very little information within the @PactFor method which one of the parsed messages corresponds to which of the interactions. The only point of reference is the current index of the loop on the message list. The content cannot be used to infer information, as it already has been converted to a byte array.

The only way to circumvent this currently (to my knowledge) is to define a @PactFor method for each of the interactions. However, those methods would only differ in their name and their value for pactMethod.

Solution Approach:
Use a similar solution to the Provider Test implementation relying on the @TestTemplate annotation. I haven't work with this so far, so I don't know if that possible or which changes would be required.

An example based on https://github.com/pact-foundation/pact-jvm/blob/master/consumer/junit5/src/test/groovy/au/com/dius/pact/consumer/junit5/V4AsyncMessageTest.groovy

@ExtendWith(PactConsumerTestExt)
@PactTestFor(providerName = 'MessageProvider', providerType = ProviderType.ASYNCH, pactVersion = PactSpecVersion.V4)
class V4AsyncMessageTest {
  // Example message handler
  static class MessageHandler {
    static ProcessedMessage process(byte[] data) {
      def json = new JsonSlurper().parse(data) as Map
      new ProcessedMessage(json)
    }
  }

  // Example processed message
  @Canonical
  static class ProcessedMessage {
    String testParam1
    String testParam2
  }

  /**
   * Set the first message interaction (with matching rules)
   */
  @Pact(consumer = 'test_consumer_v4')
  V4Pact createPact(MessagePactBuilder builder) {
    PactDslJsonBody body = new PactDslJsonBody()
    body.stringMatcher('testParam1', '\\w+', 'value1')
    body.stringValue('testParam2', 'value2')

    Map<String, Object> metadata = [destination: Matchers.regexp('\\w+\\d+', 'X001')]

    builder.given('SomeProviderState')
      .expectsToReceive('a test message')
      .withMetadata(metadata)
      .withContent(body)
      .toPact()
  }

  /**
   * Setup the second message interaction (with plain data)
   */
  @Pact(provider = 'MessageProvider', consumer = 'test_consumer_v4')
  V4Pact createPact2(MessagePactBuilder builder) {
    PactDslJsonBody body = new PactDslJsonBody()
    body.stringValue('testParam1', 'value3')
    body.stringValue('testParam2', 'value4')

    Map<String, String> metadata = ['Content-Type': 'application/json']

    builder.given('SomeProviderState2')
      .expectsToReceive('a test message')
      .withMetadata(metadata)
      .withContent(body)
      .toPact()
  }

  /**
   * This is the new, comfortable part. the method would be invoked for each interaction. The test name could be generated from the value that is passed to `expectsToReceive` on the interaction, like it's done in both sync and async message provider tests.
   */
  @TestTemplate
  @ExtendWith(PactVerificationSpringProvider::class)
  @WithMockUser
  void test(PactInteractionContext context) {
      assertDoesNotThrow(() -> {
          new MessageHandler().process(context.getMessage().getAsString())
      })
  }
}

Edit: Fixed test template assertions to be actually applicable to both interactions.

[timvahlbrock]

Just saw that the example https://github.com/pact-foundation/pact-jvm/blob/master/consumer/junit5/src/test/groovy/au/com/dius/pact/consumer/junit5/V4AsyncMessageTest.groovy mentioned above the second test takes V4Pact as an argument instead of a message. For what I can tell from a glance at the implementation, it should be possible to provide context to failed assertions from this. I think that having one test case per message would be nice nevertheless.

[timvahlbrock]

I gave this a little try by defining a TestTemplateInvocationContextProvider myself and using the resolveParameter method on the Pact extension to get the messages without re-implementing the collection. Currently the main issue seems to be that the different extension contexts that a TestTemplateInvocationContextProvider receives versus a regular test extension. As each test uses its own test class instance, testInstance is null on the extension context passed to the TestTemplateInvocationContextProvider. However, testInstance is required by pact to be able to invoke the message builder methods. This means for this to work the message descriptions would need to be static or in another class while Pact would need to receive implementations to extract message descriptions without requireing an extension context with a testInstance.

[rholshausen]

I did have a brief look at this, but it looks very challenging. It will require a refactor so that the TestTemplateInvocationContextProvider and the PactVerificationExtension can work together correctly, or all the functionality replicated there.

[timvahlbrock]

Fiddled around with this a bit to create a POC, added as #1815 . Implementation details are commented there.