Newly added ProwJobs don't honor skip_if_changed, etc.. configurations

[smg247]

When a new check is added to prow, it is added to (and triggered on) all currently opened PRs in the repo regardless of the skip_if_changed, and only_run_if_changed configs. This causes massive disruption in some cases where there are many opened PRs in the repo. When a new PR is opened after the ProwJob has been added, the configurations are honored.

[smg247]

/kind bug

[petr-muller]

Long time ago this had an issue in k/test-infra, but it probably was not migrated (or maybe rot before the migration even started). This is caused by the difference in how the jobs are triggered when they run "normally" on a GH event, and when they are triggered by the status-reconciler after job configuration changed. I think status-reconciler wants to save GH tokens or something and only inspects the open PRs to trigger newly added jobs for, but does not check out the code and test the conditional trigger clauses against the PR content.

Not sure what is the best way to address this - either improve status-reconciler to fully evaluate each PR for trigger feasibility, or come up with some alternative.