fix: remediator ignores no resource/kind match

[karlkfi]

• Update remediator to record NoMatchErrors as a resource conflict.
  These are reported as metrics and cause retry, but are still not
  reported as sync errors, for now.
• Update comments in TestCRDDeleteBeforeRemoveCustomResourceV1
  to explain why a conflict is expected and which ones might be
  encountered due to race condition.

[nan-yu]

The intent of the check here is to confirm the remediator detects the drift, but couldn't correct it.

Instead of checking no conflicts, what about adding a new metric for the remediator error, and validating a remediator error is recorded in the metrics? We can still check no conflicts, but it is just no enough.

[karlkfi]

I added handling of NoMatchErrors as conflicts using the existing metric. This should make it report the error metric consistently in this test.

If we want to add a new metric, it'll need a one-pager, because we also need to separate management conflicts from resource conflict, and list all the cases to make sure they're handled correctly. The management conflict reporting is spotty at best today. It only works for RootSyncs and only sometimes. I don't really have time to do this now, due to the watch filter project.

[sdowell]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdowell

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sdowell]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment