Encoding issue when generating pact containing accented characters

[arochat]

In a Java project using pact junit5 4.2.11, and I have encoding issues.
When generating a Pact, the response body is read from a json file containing accented characters (é, è, ü, etc)
When the pact is generated, those characters are replaced with their unicode encoding.
Then on the producer side, when running the test, the pact verification fails due to those encoded characters.

This is the test code :

@ExtendWith(PactConsumerTestExt.class)
@SpringBootTest
public class ProductWithAccentPactTest {

   @Autowired
   private ObjectMapper objectMapper;

   @Pact(provider = "one-catalog_product_with_accents", consumer = "product-display")
   public RequestResponsePact createPact(PactDslWithProvider builder) {

      return builder.given("an hybrid product")
         .uponReceiving("a request for product 100001789")
         .path("/one-catalog/public/v1/catalog-items")
         .query(
            "ids=100001090&warehouseId=1&ongoingOfferDate=2022-04-19T00:00&fields=(id, product)&activeOffersOnly=true")
         .method("GET")
         .willRespondWith()
         .status(200)
         .body(jsonAsString("/pact/product-with-accents.json"))
         .headers(Map.of("content-type", "application/json"))
         .toPact();

   }


   @PactTestFor(providerName = "one-catalog_product_with_accents")
   @Test
   void validatePact(MockServer mockServer) throws IOException {

      // when
      var actual = oneCatalogClient(mockServer.getUrl()).findOneCatalogItems(
         ProductFilter.builder().uids(List.of(100001090l)).build(),
         OfferFilter.builder()
            .storeType(StoreType.ONLINE)
            .warehouseId(1)
            .ongoingOfferDate(LocalDateTime.of(2022, 4, 19, 0, 0, 0))
            .build(), "(id, product)");

      // then
      var expected = objectMapper.readValue(
         ResourceReader.readResource("pact/product-with-accents.json"),
         OneCatalogItem[].class);

      assertThat(actual).hasSize(1).first().usingRecursiveComparison().isEqualTo(
         Arrays.stream(expected).findFirst().get());
   }

   private String jsonAsString(String path) {
      return removeWhiteSpaces(IOUtils.toString(
         this.getClass().getResourceAsStream(path),
         StandardCharsets.UTF_8));
   }

   private String removeWhiteSpaces(String input) {
      return input.replaceAll("\r", "").replaceAll("\n", "").replaceAll("\t", "");
   }

   private RestOneCatalogClient oneCatalogClient(String uri) {
      return new RestOneCatalogClient(URI.create(uri), new RestTemplateBuilder());
   }

   @SpringBootConfiguration
   public static class TestContext {

      @Bean
      public ObjectMapper objectMapper() {
         return new ObjectMapper()
            .registerModule(new Jdk8Module())
            .registerModule(new JavaTimeModule())
            .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
            .setSerializationInclusion(JsonInclude.Include.NON_EMPTY);
      }
   }
}

And here is an extract of the generated pact with encoded characters :

{
  "consumer": {
    "name": "product-display"
  },
  "interactions": [
    {
      "description": "a request for product 100001789",
      "providerStates": [
        {
          "name": "an hybrid product"
        }
      ],
      "request": {
        "method": "GET",
        "path": "/one-catalog/public/v1/catalog-items",
        "query": {
          "activeOffersOnly": [
            "true"
          ],
          "fields": [
            "(id, product)"
          ],
          "ids": [
            "100001090"
          ],
          "ongoingOfferDate": [
            "2022-04-19T00:00"
          ],
          "warehouseId": [
            "1"
          ]
        }
      },
      "response": {
        "body": [
          {
            "id": 100001090,
            "product": {
              "brand": "Jura-Sel",
              "categoryBreadcrumbs": [
                {
                  "categoryId": "3850252",
                  "categoryName": {
                    "de": "Salzige Lebensmittel",
                    "en": "Salty Groceries",
                    "fr": "Epicerie sal\u00E9e",
                    "it": "Alimenti salati"
                  },
                  "level": 2,
                  "slug": {
                    "de": "salzige-lebensmittel",
                    "en": "salty-groceries",
                    "fr": "epicerie-salee",
                    "it": "alimenti-salati"
                  }
                },
                {
                  "categoryId": "3860189",
                  "categoryName": {
                    "de": "Gew\u00FCrze & Saucen",
                    "en": "Spices & Sauces",
                    "fr": "Condiments & Sauces",
                    "it": "Spezie e salse"
                  },
                  "level": 3,
                  "slug": {
                    "de": "gewurze-saucen",
                    "en": "spices-sauces",
                    "fr": "condiments-sauces",
                    "it": "spezie-e-salse"
                  }
                },
                {
                  "categoryId": "3860200",
                  "categoryName": {
                    "de": "Salz, Pfeffer & W\u00FCrze",
                    "en": "Spices, Salt & Pepper",
                    "fr": "Sels, Poivres & Assaisonnements",
                    "it": "Sale, pepe e condimenti"
                  },
                  "level": 4,
                  "slug": {
                    "de": "salz-pfeffer-wurze",
                    "en": "spices-salt-pepper",
                    "fr": "sels-poivres-assaisonnements",
                    "it": "sale-pepe-e-condimenti"
                  }
                }
              ],

While debugging I can see that until this line

pact-jvm/core/model/src/main/kotlin/au/com/dius/pact/core/model/PactWriter.kt

Line 66 in 4712d32

Json.prettyPrint(pact.sortInteractions().toMap(pactSpecVersion))

the characters look fine.
Therefore I guess something happens in the Json.prettyPrint() function, but I can’t debug inside.

[rholshausen]

How is the pact verification failing? Can you provide the error messages? The UTF escape codes should still represent the same characters.

[timvahlbrock]

How is the pact verification failing? Can you provide the error messages? The UTF escape codes should still represent the same characters.

This issue hasn't seen a lot of activity recently. However, I'm having a similar issue. Any non-ASCII characters are escaped with their unicode code, even though the pact file is UTF-8. I'm running into problems when the pact cli is trying to upload it: https://www.linen.dev/s/pact-foundation/t/23419250/hello-i-got-a-problem-with-the-regex-generator-and-the-pact- .

[timvahlbrock]

If i get it correctly, Pact JVM had sometime made changes to not escape unicode chars: #395

[rholshausen]

That change was a long time ago. It does escape all characters now.

[rholshausen]

It uses the Apache Commons Text library to correctly encode the strings, and I doubt that library does not do the correct thing. So this is an issue with the pact cli.

[rholshausen]

Although, looking at the code, it is using the ESCAPE_JAVA translator which has a high code point of 0x7f, while there is an ESCAPE_JSON translator that has a high code point of 0x7e. That only differs by 1, and I have no idea what the difference would be.

[timvahlbrock]

Although, looking at the code, it is using the ESCAPE_JAVA translator which has a high code point of 0x7f, while there is an ESCAPE_JSON translator that has a high code point of 0x7e. That only differs by 1, and I have no idea what the difference would be.

Despite some research, I don't really get what that means, but I can say that the generated escape sequences match the original symbols and with some characters like the german "ö" the pact cli doesn't seem to have a problem. So this does in fact look like an issue with the CLI. I suspect that the problem that the original author of this issue had might have been similar.