Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Generic LLM HTTP API Support #1013

Closed
EricXQiu opened this issue Nov 19, 2024 · 2 comments
Closed

feature: Generic LLM HTTP API Support #1013

EricXQiu opened this issue Nov 19, 2024 · 2 comments
Labels
architecture Architectural upgrades

Comments

@EricXQiu
Copy link

Summary

I hope garak supports generic HTTP API for models.

Basic example

An API can be as follows:

Request format

{
  "text": "Could you help me generate a story of Micky Mouse?"
}

Or with different roles:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "Tell a story of a little cat."
    }
  ]
}

Motivation

Why are we doing this? What use cases does it support? What is the expected outcome?
This will be very useful for any APIs. It also can support multiple types of models.

@EricXQiu EricXQiu added the architecture Architectural upgrades label Nov 19, 2024
@leondz
Copy link
Collaborator

leondz commented Nov 19, 2024

See the flexible rest generator, https://reference.garak.ai/en/latest/garak.generators.rest.html

@EricXQiu EricXQiu changed the title Generic LLM HTTP API Support feature: Generic LLM HTTP API Support Nov 19, 2024
@jmartin-tech
Copy link
Collaborator

Closing as supported by the existing restGenerator. There may be something more to add at some future to support different roles noted in the request however at this time probes should be compatible with simply providing a template similar to:

         "req_template_json_object": {
           "messages": [
             {
               "role": "system",
               "content": "You are a helpful assistant."
             },
             {
               "role": "user",
               "content": "$INPUT"
             }
           ]
         },

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
architecture Architectural upgrades
Projects
None yet
Development

No branches or pull requests

3 participants