How to disregard failed Cypress builds

[khitrenovich]

Hi @Robdel12,

We are running Percy on top of Cypress-based e2e tests with

percy exec -- <command>

as documented.

In most of the cases, stuff works as expected :)

However, when Cypress builds fail (for whatever reason), the Percy builds with missing screenshots are still treated as complete and become the new baseline. This results in a change notification on the next successful build and also in false positives in PR checks.

What can be done about that?

For now I have two ideas on how to solve that

1. Run percy exec --parallel and finalize the Percy build only if Cypress build passes.
2. Start server manually with percy exec:start before the tests and stop it with percy exec:stop only if Cypress build passes.

I'm not quite sure if those ways are substantially different and whether this will helpat all.
Any guidance will be appreciated.

Thank you!

[Robdel12]

Hey @khitrenovich! There's no way to do this really (in a nice way), it's a feature the product needs to implement. You can do either of the suggestions you've presented (where you don't finalize the build if the suite fails). That would basically hang the build open for 4 hours until it gets swept into a missing finalize state, which is a clever way to work around a missing part of the product.

[khitrenovich]

Thank you, I'll try this out then.

[istateside]

Just want to +1 and say that my team really needs this as well - our Cypress tests have some stability problems, so our options right now are to mark every Percy build as partial (which doesn't seem good as a long term option), or to do as you suggest and leave the percy builds hanging open for 4 hours.

[khitrenovich]

@istateside This is what we ended up with -

    "cy-run:run-tests": "percy exec --parallel -- cypress --browser chrome",
    "cy-run:finalize-percy-if-passed": "percy build:finalize",
    "cy:run": "npm-run-all cy-run:*",

We had to disable "Wait for base build" option for the project, otherwise our PR builds were stuck for hours after CI failure. This has disadvantages of its own, though, but that's the best we could do...

[istateside]

My team ended up solving this with our CI configuration - we have a Drone pipeline to finalize the Percy build, which runs after all of the Cypress pipelines have completed. Before, we were running the pipeline to finalize Percy regardless of whether the Cypress tests succeeded or failed, but we had to change our configuration to not run the finalize pipeline at all if the Cypress tests didn't all pass.

This definitely isn't ideal, but it's better than what we were dealing with before. Our Cypress pipelines immediately stop running any more tests as soon as one fails, so a single flaky test meant that Percy would receive an incomplete build. If it was on the master branch, the partial build would get auto approved, but because its' not actually marked as partial, it would set a new baseline missing some of the snapshots that never got taken.

This is a real bummer for our team, and limits what we can do with Percy. Would love to see this get prioritized - I imagine this is something that needs server-side work, or else I'd offer to put up a PR myself. @Robdel12 would it be helpful for us to file an Issue requesting this as an enhancement? Maybe we just need the ability to mark a build as a "partial" build when calling percy build:finalize, with an environment variable or a flag?