Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

per-probe tags now adjustable based on payload selection #1031

Merged
merged 1 commit into from
Dec 2, 2024

Conversation

leondz
Copy link
Collaborator

@leondz leondz commented Dec 2, 2024

fix bug where encoding probes with payloads set in config would fail to load

Verification

try this config

encodingpayloads.yaml

run:
  generations: 1

plugins:
  probe_spec: encoding
  extended_detectors: true
  probes:
    encoding:
      payloads:
        - default
        - xss
        - slur_terms
  model_type: test

Before fix:

failed to load probe probes.encoding.InjectAscii85
failed to load probe probes.encoding.InjectBase16
failed to load probe probes.encoding.InjectBase2048
failed to load probe probes.encoding.InjectBase32
failed to load probe probes.encoding.InjectBase64
failed to load probe probes.encoding.InjectBraille
failed to load probe probes.encoding.InjectEcoji
failed to load probe probes.encoding.InjectHex
failed to load probe probes.encoding.InjectMorse
failed to load probe probes.encoding.InjectNato
failed to load probe probes.encoding.InjectROT13
failed to load probe probes.encoding.InjectUU
failed to load probe probes.encoding.InjectZalgo

After fix:

encoding.InjectAscii85                                                          encoding.DecodeMatch: PASS  ok on 1140/1140                                                         
encoding.InjectAscii85                                                         encoding.DecodeApprox: PASS  ok on 1140/1140
encoding.InjectBase16                                                           encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectBase16                                                          encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectBase2048                                                         encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectBase2048                                                        encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectBase32                                                           encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectBase32                                                          encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectBase64                                                           encoding.DecodeMatch: PASS  ok on 1105/1105                                                         
encoding.InjectBase64                                                          encoding.DecodeApprox: PASS  ok on 1105/1105
encoding.InjectBraille                                                          encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectBraille                                                         encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectEcoji                                                            encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectEcoji                                                           encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectHex                                                              encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectHex                                                             encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectMorse                                                            encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectMorse                                                           encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectNato                                                             encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectNato                                                            encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectROT13                                                            encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectROT13                                                           encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectUU                                                               encoding.DecodeMatch: PASS  ok on  570/ 570                                                         
encoding.InjectUU                                                              encoding.DecodeApprox: PASS  ok on  570/ 570
encoding.InjectZalgo                                                            encoding.DecodeMatch: PASS  ok on  588/ 588                                                         
encoding.InjectZalgo                                                           encoding.DecodeApprox: PASS  ok on  588/ 588

side note - encoding.py is getting close to the point where it needs a tidy

@leondz leondz added the bug Something isn't working label Dec 2, 2024
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works for a short term improvement. I would like to see this whole method move to gather from self.payloads with an early call in __init__() of the BaseEncodingProbe to self._load_config(config_root) instead of sharing a package level value.

@leondz leondz merged commit c236753 into NVIDIA:main Dec 2, 2024
10 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Dec 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants