Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SauceLabs tests to CI #137

Merged
merged 57 commits into from
Nov 13, 2023
Merged

Add SauceLabs tests to CI #137

merged 57 commits into from
Nov 13, 2023

Conversation

seemk
Copy link
Contributor

@seemk seemk commented Apr 16, 2023

There's now a new project, SauceLabsTestApp, which runs through a few (mostly) networking tests. The app sets up its own HTTP server which is then used to route the spans to tests for verification.

I tried to bring in some order to the tests, all tests derive from TestCase, which sets up some central bookkeeping (handles timeouts, attaches test names to spans, routes test results).

When creating new tests, only 2 methods need to be overridden: execute - sets up the test, e.g. start an HTTP request and verify which is called with the received spans.

Since it's a bit annoying to verify the results in Sauce Labs, I just created a label that changes the text to success when all tests pass. Sauce Labs waits for this label.

Areas that need more work:

  • iOS 11-13 compatibility. Initially opted for SwiftUI for the test app, but later found out it's not really supported on iOS 11 and 12. The UI part of the app is really small, so perhaps it would make sense to write it in the old way. iOS 13, the app compiles and tests run fine, but the Sauce Labs script can't find the UI elements necessary to validate tests.
  • More tests (crash reporting, additional networking libraries, UI clicks)?

@seemk seemk requested a review from johnbley as a code owner April 16, 2023 20:55
@seemk seemk marked this pull request as draft April 16, 2023 20:56
@seemk seemk changed the title [DRAFT] Testing SauceLabs CI Add SauceLabs tests to CI May 30, 2023
@seemk seemk marked this pull request as ready for review May 30, 2023 11:17
@seemk seemk requested review from a team as code owners November 1, 2023 12:19
strategy:
fail-fast: false
matrix:
os: ['14', '15', '16']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great. I really like that the test matrix is in github rather than in SauceLabs UI/config.

Copy link

github-actions bot commented Nov 13, 2023

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@seemk
Copy link
Contributor Author

seemk commented Nov 13, 2023

I have read the CLA Document and I hereby sign the CLA

srv-gh-o11y-gdi-cla added a commit to splunk/cla-agreement that referenced this pull request Nov 13, 2023
@seemk seemk merged commit cb9fdea into main Nov 13, 2023
8 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Nov 13, 2023
@Kielek Kielek deleted the saucelabs-tests branch November 14, 2023 05:35
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants