vllm provider

[DanielBeck93]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Windows 11
- Continue version: 0.8.57
- IDE version: 1.95.1
- Model: Llama 3.1 8B
- config.json:  {
      "title": "Llama 3.1 8B",
      "model": "ged4",
      "apiBase":  "some_url",
      "completionOptions": {
      "temperature": 0.1,
      "topK": 1,
      "topP": 1,
      "presencePenalty": 0,
      "frequencyPenalty": 0
      },
      "provider": "vllm",
      "apiKey": "some_api_key"
  },

Description

Hi, I have deployed an Llama3 model thourgh vllm.
When I choose vllm as provider in my config file I get:

Error streaming diff: TypeError: Cannot read properties of undefined (reading 'toLowerCase')

when pushing apply or trying to edit the code, or commenting the code through Continue. But when changing the provider to openai it works well. Is this expected?

To reproduce

config:
{ "title": "Llama 3.1 8B", "model": "ged4", "apiBase": "some_url", "completionOptions": { "temperature": 0.1, "topK": 1, "topP": 1, "presencePenalty": 0, "frequencyPenalty": 0 }, "provider": "vllm", "apiKey": "some_api_key" },

Log output

Error streaming diff: TypeError: Cannot read properties of undefined (reading 'toLowerCase')

[tomasz-stefaniak]

This error originates in VerticalDiffManager and is probably not expected:

• https://github.com/continuedev/continue/blob/main/extensions/vscode/src/diff/vertical/manager.ts#L257
• https://github.com/continuedev/continue/blob/main/extensions/vscode/src/diff/vertical/manager.ts#L434

Is the problem blocking you from using Continue, or does switching the provider openai fix the problems completely?