Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: add experimental features flag to be only accessible in core config #1003

Merged
merged 2 commits into from
Nov 27, 2024

Conversation

leondz
Copy link
Collaborator

@leondz leondz commented Nov 15, 2024

add facility to unlock experimental CLI options if a core config flag is set

Verification

  • run python -m garak --help, experimental features should not be enabled
  • change system.enable_experimental to true in garak/resources/garak.core.yaml
  • run python -m garak --help, there should be a message between the usage summary and options that experimental features are enabled

Question - would this feature be better off in a site config? I'm concerned about garak.core.yaml's value getting accidentally clobbered in a development commit

@leondz leondz added architecture Architectural upgrades cli Command-line interface functions labels Nov 15, 2024
@leondz leondz requested a review from jmartin-tech November 15, 2024 10:29
@jmartin-tech
Copy link
Collaborator

Question - would this feature be better off in a site config? I'm concerned about garak.core.yaml's value getting accidentally clobbered in a development commit

One option is to add garak.core.yaml to the .gitignore, changes to it could still committed they would just require an additional flag force adding any file changes to a commit.

I could see allowing this to be in garak.site.yaml as well, depending on what level of accessibility we want exposed. The banner combined with the serialized system config in the report log makes it clear if the option is turned on.

I am wondering if we should make this more of a feature flag set, maybe we should accept a list of experimental features to enable vs boolean value. That may be overkill for now though.

@leondz
Copy link
Collaborator Author

leondz commented Nov 20, 2024

Alright. I think for now retaining the checks predicated on garak.core.yaml are more important than the odd mis-commit, so will proceed without a .gitignore. Site config is a great place for this flag but putting it there means changing the order of loading in a way that affects garak.cli processing, and I haven't taken a call on whether that's a good idea yet. So let's proceed as is

@leondz leondz merged commit 5c34717 into main Nov 27, 2024
11 checks passed
@leondz leondz deleted the feature/experimental_setting branch November 27, 2024 12:16
@github-actions github-actions bot locked and limited conversation to collaborators Nov 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
architecture Architectural upgrades cli Command-line interface functions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants