Reverse scrubbing?

[JulianRooze]

I just got introduced to this great library and I have a question regarding scrubbing:

When I saw scrubbing was an option, I thought the workflow for a new test was:

• I call Verify on a result object
• I get a failure because I don't have a *.verified file yet.
• I check the *.received file, and if I like it I copy it to *.verified
• I replace everything in *.verified that differs between calls (like IDs) with placeholders myself
• I run it again and IDs in the *.received file that are covered by placeholders are ignored

But this isn't how it actually works turns out (unless I'm missing something) 🙃

From what I understand, scrubbing works primarily on cleaning the received file, by intercepting the serialization. I think I understand why it works the way it does, because then comparison is mostly just a string comparison exercise between old and new, but is there perhaps also potential here for a reverse scrubbing workflow, where the received file is left 'dirty' and matched against a manually scrubbed verified file and anything that is covered by a placeholder is ignored?

For our use case, the existing scrubbing functionality is hard to get right as our objects look like this (for example):

{
  Address: {
    Address1: Sesame Street,
    City: Oatmeal Town,
    CountryID: NL,
    FirstName: Big,
    HouseNumber: 00,
    ID: 1718498579581017,
    LastName: Bird,
    Region: Bakewell,
    ZipCode: 1010NA,
    TypeID: 4
  }
}

So the ID property differs every call, so must be scrubbed. However, the TypeID property is always the same and is something we want to verify. Initially I figured I would check if something looked like an ID and could be parsed as a long, but that would include TypeID here and ultimately I don't think we have a reliable way to programmatically ignore/scrub the right members and not ignore the ones we don't want to ignore, so we would have to do it manually (which is fine).

What I would like to be able to do is manually edit the verified file to look something like this:

{
  Address: {
    Address1: Sesame Street,
    City: Oatmeal Town,
    CountryID: NL,
    FirstName: Big,
    HouseNumber: 00,
    ID: @@ADDRESS_ID@@,
    LastName: Bird,
    Region: Bakewell,
    ZipCode: 1010NA,
    TypeID: 4
  }
}

And at comparison time, any members in the received file that are a matched on a placeholder value are ignored.

Is this something that would be feasible to add, or is this perhaps already supported somehow, or even a better way to do what I want?

[SimonCropp]

@JulianRooze not sure about "reverse scrubbing" that sounds difficult to implement. But you should be able to easily scrub the id using .ScrubMember("ID") https://github.com/VerifyTests/Verify/blob/main/docs/serializer-settings.md#scrub-member-by-name

[JulianRooze]

@SimonCropp Well this is just a contrived example, in reality there's many properties, like ProductID/UserID/etc. While they do follow a naming convention, there's two things:

• I don't believe there's a way to scrub members based on a pattern, is there? There's an overload that accepts an Expression, but that seems to be a type-safe way to return the name of a single member, you can't do member => member.EndsWith("ID"). Would be really nice if there was also a predicate based scrubbing option to deal with patterns in one fell swoop.
• For us this is going to lead to some false positives as there are also quite a few properties called TypeID that are bound to a (on the backend, will be using this library for integration testing) hardcoded enum that mustn't be scrubbed, but also properties called TypeID that are "random" varying IDs.

I think the second issue is something we can work around or just accept. As for the first, I think that's something I could fairly easily add to public class SerializationSettings, at least that seems to be the place to add it. What I would add is a Func<MemberInfo, bool> to return if a member should be scrubbed or ignored and check that in TryGetScrubOrIgnore. Does that sound like a good idea to you?

[SimonCropp]

how about this as a solution

    [ModuleInitializer]
    public static void Initialize()
    {
        foreach (var type in typeof(Address).Assembly.GetTypes())
        {
            foreach (var property in type.GetProperties())
            {
                if (property.Name.EndsWith("ID"))
                {
                    VerifierSettings.ScrubMember(type, property.Name);
                }
            }
        }
    }

[JulianRooze]

@SimonCropp 👌 that works I suppose.

Are you opposed to the predicate overload to Ignore/Scrub methods? I had a go at implementing it, I was conflicted whether the predicate should receive the whole MemberInfo or just the name, as you don't have a MemberInfo available everywhere you can scrub/ignore, sometimes you only have a name (like dictionary keys), but having a MemberInfo is nice as you can make decisions based on the property/field type as well, but a string predicate is more widely applicable.

[SimonCropp]

Are you opposed to the predicate overload to Ignore/Scrub methods

exposing MemberInfo via a predicate would add more reflection overhead during serialization. IMO it i better to do that once at startup

[JulianRooze]

@SimonCropp My proposal would be executed at the same time as the current Ignore/ScrubMember options, but instead of doing is member name in dictionary then scrub it would do if callback for member returns true then scrub. I don't think it would add any runtime overhead and potentially save some startup overhead as you only register one callback function instead of adding thousands of strings to a dictionary.

[SimonCropp]

happy to consider a PR if you include benchmarks

[benmccallum]

This is timely! I was just looking at finally updating Verify on a project that's stuck back on an old version when scrubbing of "NumericIds" was still a thing. That feature used to replace the values with say Id_1, Id_2, etc but was likely removed because the counter potentially replaced occurrences of the same id value with the same replacement value, and that could be a problem in integration tests with dynamic ids depending on what you subset of tests you ran and if ids overlapped across tables for instance.

PR looks good, the string overload would be good enough for me and the MemberInfo is a nice to have but since that's already been reflected wouldn't seem to impact perf.