Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BCPT Test Report #619

Merged
merged 137 commits into from
Jun 21, 2024
Merged

BCPT Test Report #619

merged 137 commits into from
Jun 21, 2024

Conversation

freddydk
Copy link
Contributor

@freddydk freddydk commented Jul 24, 2023

This PR will add a BCPT Test Report to the build summary, looking like this:
image

You can also add a bcptBaseLine.json to the project in order to establish a baseline for the performance tests.
It looks like this:
image

TODOs:

  • Add tests
  • Add thresholds to project settings
  • Determine how thresholds should work? threshold on very small items like (enter account no.) doesn't make much sense. It absolutely makes sense to have threshold on scenarios.
  • Determine sorting of test results? (codeunitID, codeunitName or ???)
  • Issue GitHub warnings and errors when thresholds are exceeded
  • Is durationMin in milliseconds, seconds or what? how many decimal digits should be displayed?
  • Get BCPT Backend Team signoff that BCPT Test Results are correctly understood and compared
  • Add scenario documentation

Example of bcpt tests with failures and warnings:
image

@freddydk freddydk changed the title BCPT Test Report [DRAFT] BCPT Test Report Jul 24, 2023
mazhelez
mazhelez previously approved these changes Jun 3, 2024
Copy link
Collaborator

@mazhelez mazhelez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay to me.

Should there be any part of the E2E tests that check any of the new functionality?

Scenarios/AddAPerformanceTestApp.md Outdated Show resolved Hide resolved
Actions/AnalyzeTests/TestResultAnalyzer.ps1 Show resolved Hide resolved
@freddydk freddydk requested a review from mazhelez June 7, 2024 08:05
Actions/AnalyzeTests/AnalyzeTests.ps1 Show resolved Hide resolved
RELEASENOTES.md Outdated Show resolved Hide resolved
Actions/AnalyzeTests/TestResultAnalyzer.ps1 Show resolved Hide resolved
@freddydk freddydk requested a review from aholstrup1 June 17, 2024 18:14
aholstrup1
aholstrup1 previously approved these changes Jun 18, 2024
Copy link
Collaborator

@aholstrup1 aholstrup1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't fully agree with adding the bcptThresholds.json file but the rest of the PR looks good and will be a really nice improvement to AL-Go 🥳

mazhelez
mazhelez previously approved these changes Jun 19, 2024
Copy link
Collaborator

@mazhelez mazhelez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expect for the MD generation, the PR looks okay.

Tests are also hard to understand. Maybe using a template for the markdown will help there as well.

Actions/AnalyzeTests/TestResultAnalyzer.ps1 Show resolved Hide resolved
Tests/AnalyzeTests.Test.ps1 Outdated Show resolved Hide resolved
Actions/AnalyzeTests/TestResultAnalyzer.ps1 Outdated Show resolved Hide resolved
@freddydk freddydk dismissed stale reviews from mazhelez and aholstrup1 via 954bfda June 20, 2024 11:24
@freddydk freddydk requested review from aholstrup1 and mazhelez June 20, 2024 12:09
mazhelez
mazhelez previously approved these changes Jun 20, 2024
Actions/AnalyzeTests/TestResultAnalyzer.ps1 Outdated Show resolved Hide resolved
@freddydk freddydk merged commit dd7d7a6 into microsoft:main Jun 21, 2024
4 checks passed
@freddydk freddydk deleted the bcptreport branch June 24, 2024 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BCPT Test Results Viewer
3 participants