Bug in evaluation script during training

[KevinEloff]

There are some inconsistencies in how arguments are passed to the various evaluation functions. This might be causing amino acid precision and recall to be switched.

In the validation step, the peptide precision is calculated as follows:

        aa_precision, _, pep_precision = evaluate.aa_match_metrics(
            *evaluate.aa_match_batch(
                peptides_pred, peptides_true, self.decoder._peptide_mass.masses
            )
        )

The evaluate.aa_match_metrics expects the arguments in the order of first n_aa_true and then n_aa_pred.

def aa_match_metrics(
    aa_matches_batch: List[Tuple[np.ndarray, bool]],
    n_aa_true: int,
    n_aa_pred: int,
) -> Tuple[float, float, float]:

The evaluate.aa_match_batch function returns n_aa1 and n_aa2 in the same order as peptides1 and peptides2

def aa_match_batch(
    peptides1: Iterable,
    peptides2: Iterable,
    aa_dict: Dict[str, float],
    cum_mass_threshold: float = 0.5,
    ind_mass_threshold: float = 0.1,
    mode: str = "best",
) -> Tuple[List[Tuple[np.ndarray, bool]], int, int]:
    ...
    return aa_matches_batch, n_aa1, n_aa2

The problem comes in that in the validation step, we pass peptides1=peptides_pred and peptides2=peptides_true, thereby passing n_aa_true as the number of number of predicted amino acids, and vice-versa for n_aa_pred.

This shouldn't cause an issue to the current peptide-level precision calculation, but it may affect older versions of the codebase.

[bittremieux]

Thanks, excellent catch.

The good thing is that this bug only influenced the validation metrics reported by Casanovo directly (e.g. during training/validation). The results and figures included in the papers were obtained with external evaluation scripts, so normally those should still be correct.