Random test failures #106
Comments
|
I ran into this too. I agree enforcing a stable iOS version is a good thing to try. I had also noticed that some of the versions of the reusable GitHub Action stages we leverage were reported as deprecated, but it looks like you already fixed that. Separate issue, but I also noticed that the pipeline isn't accurately noticing when the build stage fails. This makes the output confusing, because the pipeline carries on to the subsequent test stages which complain about the app bundle not existing. Example: https://github.com/soundscape-community/soundscape/actions/runs/10754605735/job/29825643227 |
* test on iOS 18.0 * return the correct exit code from the build command
|
It looks like this is still a problem (the most recent commit still needed to be run twice before it passed). I dug into the output log and have a theory, though not a fix. A common property of the inconsistently failing tests is triggering speech. Based on some discussions online (e.g. https://stackoverflow.com/a/73886077, https://forums.developer.apple.com/forums/thread/708114), as of iOS 16, the simulator no longer comes with voices installed by default -- you need to manually open the Settings app within the simulator and install voices individually. This is consistent with what I observed after an upgrade some time within the last year, when all the voices disappeared from inside the simulator on my Mac. I'm not sure how/if this can be done in an automated way inside a pipeline. Nor can I explain why the failures are only inconsistent. Below is an excerpt of the output from a failing test: |
|
Apparently using the tts api triggers a voice to be installed. I suspect it succeeds the second time because GitHub is caching the macos instance and the voice has already been downloaded We could possibly add a test that runs first to trigger a voice download and waits for it to be availible. |
Perhaps extending the wait time gives TTS voices time to download.
steinbro
added
bug
|
Unrelated to the voice issue, but it is failing again with the latest macos image on github actions. It worked with version 20241022.361, but failed with version 20241108.442. It doesn't seem to be accepting the target specification: According to the readme for the image the iOS 18 simulator is only available with Xcode 16, and 18.1 with Xcode 16.1. I'm not sure there's much we can do other than to manually update the workflow each time a version becomes unsupported. |
|
Seems to be okay after updating the workflow to iOS 18.1 / Xcode 16.1 |
|
Should we revert to the action using the latest, possibly beta iOS version? Hard-coding the iOS version didn't fix the problem here, and I'd rather have potential future failures reported earlier than be stuck on a single version until someone manually changes it. |
|
There's the Xcode version to deal with as well. I have it working with the latest stable Xcode, iOS version determined automatically by Xcode in my fork. The run is here: https://github.com/RDMurray/soundscape/actions/runs/11818261318/job/32925422972#step:3:1 |
Sometimes the tests in GitHub are failing and I don't know why. The latest run failed but then succeeded when I re-ran it.
It does seem to be running the tests on iOS 18.1 which I think is still in beta. Perhaps we should force it to use 18.0 or the latest version of iOS 17 for now?
The text was updated successfully, but these errors were encountered: