Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rules unit tests: Do static analysis instead of fuzz testing #242

Open
MatthewMarinets opened this issue Jul 19, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@MatthewMarinets
Copy link

MatthewMarinets commented Jul 19, 2024

What task needs to be completed?

The new test that checks all items used in rules are marked progression is a fuzz test, which unfortunately means it's very slow and probably shouldn't be run with the other unit tests.

Following a discussion in the discord (starting around here), we evaluated some options and suggested that the best course of action to get an automated test with reasonable sanity-checking and performance would be a test that looks at the contents of the file. At a basic level, the changes should:

  • Move defense-rating dicts from items.py to rules.py
  • Make a test that scans the contents of rules.py, searching for instances of item_names.(\w+) and checking that the associated item is progression
    • To be really fancy, this parser probably needs to account for comments, which also means accounting for strings. May not be necessary given item_names.(\w+) probably won't appear too much in strings and comments, and we can just check that anyways
  • Make tests that also scan the contents of the starting_unit / basic_unit lists in items.py

Other possibilities would include

  • some kind of intelligent depth-first search algorithm to explore every path of every rule (a lot of work, very complicated for a test)
  • Reduce the repetitions on the fuzz test and just rely on it running frequently (would potentially lead to flaky tests, though)
  • Something slower / manually initiated, run before releases / milestones?
@MatthewMarinets MatthewMarinets added the enhancement New feature or request label Jul 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant