Feature: enable configuration updates

[jmartin-tech]

fix #889

Provides a path for forward migration of configuration when plugins or other configuration based parameter change between versions.

As an initial capability this acts similar to the packer fix command by accepting input of outdated configuration to produce an equivalent configuration for the current version of garak. To support fixing a previous configuration tooling is added to create Migration classes to be detected and run in order to modify and entires in the configuration based on structural or naming changes in garak.

The following process can be used to add new Migration to the applied set used to fix a configuration:

1. Add a new Mirgation file garak/resources/fixer/ with a filename that formatted
   as YYYYMMDD_some_descriptive_name.py
2. Implement a new class that extends garak.resources.fixer.Migration with an apply() method that accepts an existing dictionary and returns a new modified dictionary without any side-effects to the original passed dictionary.
3. If the migration impacts the name of a plugin an alias should be added to the plugin class attributes.

When a user calls garak with the --fix cli parameter the configuration change detected as requiring update will be applied to and an equivalent updated configuration will be output.

Expected action when --fix is specified:

• passed one of --*_option_file or --*_options, json output will be generated if any change is required.
• passed --config or no additional configuration based flags yaml will be output if changes are required for the user supplied configuration as well as garak.site.yaml if found in the env.

Additional changes:

• expose options and option_file support for all plugin types

Verification

Example usage and outputs:

% python -m garak --fix
garak LLM vulnerability scanner v0.10.0.post1 ( https://github.com/NVIDIA/garak ) at 2024-11-27T12:00:09.690263
No revisions applied please verify options provided for `--fix`

old.yaml:

plugins:
  probe_spec: lmrc,continuation,replay,tap,knownbadsignatures.EICAR",
  probes:
    continuation:
      ContinueSlursReclaimedSlursMini:
        source_resource_filename: fake_data_file.json
  generators:
    huggingface:
      hf_args:
        torch_dtype: float32
        device: cpu

% python -m garak --fix --config old.yaml
garak LLM vulnerability scanner v0.10.0.post1 ( https://github.com/NVIDIA/garak ) at 2024-11-27T11:50:47.931551
Updated old.yaml:
plugins:
  generators:
    huggingface:
      hf_args:
        device: cpu
        torch_dtype: float32
  probe_spec: lmrc,continuation,divergence,tap,av_spam_scanning.EICAR",
  probes:
    continuation:
      ContinueSlursReclaimedSlursMini:
        source_resource_filename: fake_data_file.json

garak.site.yaml:

plugins:
  probe_spec: replay
  generators:
    huggingface:
      hf_args:
        torch_dtype: float32
        device: cpu

% python -m garak --fix
garak LLM vulnerability scanner v0.10.0.post1 ( https://github.com/NVIDIA/garak ) at 2024-11-27T11:51:31.658398
Updated /Users/jemartin/.config/garak/garak.site.yaml:
plugins:
  generators:
    huggingface:
      hf_args:
        device: cpu
        torch_dtype: float32
  probe_spec: divergence

probe.json

{
  "continuation": {
    "ContinueSlursReclaimedSlursMini": {
      "source_resource_filename": "fake_data_file.json"
    }
  }
}

% python -m garak --fix --probe_option_file probe.json
garak LLM vulnerability scanner v0.10.0.post1 ( https://github.com/NVIDIA/garak ) at 2024-11-27T11:52:00.846171
Updated 'probe' configuration:
{
  "continuation": {
    "ContinueSlursReclaimedSlursMini": {
      "source_resource_filename": "fake_data_file.json"
    }
  }
}

• Verify garak.log reports application of each applicable fix for various configs.

[leondz]

coupla minor points, looks good in general

[leondz]

what uses aliases? same comment for other probes

[jmartin-tech]

Currently just for --plugin_info output, I could see skipping them or enabling interactive to use them later as hints for outdated command requests.

[erickgalinkin]

I'm into it -- it enables deprecation pretty easily too. Could warn on the use of aliases and fix it at some point, especially via interactive.

[leondz]

ok, that's cool. we might like to choose now how diligent we intend to be about this - are aliases only given as indications, or do we guarantee that they're exhaustive?

there's a duplication of info here (because it's already given in fixer modules) that i'm mindful of

[jmartin-tech]

Agreed I tried to note in the description this becomes a required step when adding a migration that impacts a plugin name however there is no enforcement/early warning mechanism currently in place, we could add a lint like action that checks for plugin file or class renames as a warning/hint on PRs.

[erickgalinkin]

Very into this as a clean way to help manage deprecation. Awesome work here, @jmartin-tech

[erickgalinkin]

I'm into it -- it enables deprecation pretty easily too. Could warn on the use of aliases and fix it at some point, especially via interactive.

[leondz]

What's going on with the unmatched ", at the end of the probe_spec line in old.yaml above? And I guess, seeing as it's retained, then fixer doesn't validate configs, only migrate what's already there. Maybe that is valid / maybe "valid" is undefined for us right now. This is OK, just making the behaviour explicit

[jmartin-tech]

What's going on with the unmatched ",

You have a good point, the --fix option does not currently validate the original or end config at this time, it only requires the inputs be valid yaml or json and will not correct items a garak run may consider syntax errors. I agree this should be known to consider for future troubleshooting purposes. Might be worth filing an issue documenting a possible future need for pre execution configuration validation support in the future.