adding support for llama2 (replicate), anthropic and cohere

[krrishdholakia]

Hi @JustinStitt ,

Noticed you're using a couple different llm providers in gpyt. I'm working on litellm (simple library to standardize LLM API Calls - https://github.com/BerriAI/litellm) and was wondering if we could be helpful.

Added support for Llama2, Anthropic and Cohere through LiteAssistant.

Would love to know if this helps. Also adding a screenshot of the code working post changes.

[JustinStitt]

Hey @krrishdholakia, thanks for the contribution.

I did not expect people to contribute to gpyt. I am really sorry for how bad the code is. I will fix it up at some point.

As for your patch, I noticed you didn't make any frontend changes so I will try to integrate your backend changes with the TUI (adding option to enable it, etc).

Thanks, I'll try to get this integrated and tested by next week.

[krrishdholakia]

Sweet - let me know if this helps!

[JustinStitt]

Hey, I've integrated the LITELLM backend into gpyt. I need you to checkout and test gpyt/litellm-review and setup an anthropic api key (which I don't have) and then run:

$ python -m gpyt --litellm claude-instant-1.

Be sure you are up-to-date with the dependencies by using Poetry $ poetry install (you may want a virtualenv first)

Send a screenshot of a basic prompt working and I'll resolve this PR.

[krrishdholakia]

@JustinStitt okay will do - thanks!

[krrishdholakia]

hey @JustinStitt just tested - everything works besides streaming. My streaming test works fine though - (you can see it in the test on liteassistant). I tried to fit it within your existing paradigm but let me know if i wrote the function incorrectly

    def get_response_stream(self, user_input: str) -> Generator:
        """
        Use OpenAI API to retrieve a ChatCompletion response from a GPT model.

        Memory can be configured so that the assistant forgets previous messages
        you or it has sent. (saves tokens ($$$) as well)
        """
        if not self.memory:
            self.clear_history()
        self.messages.append({"role": "user", "content": user_input})

        response = completion(  # type: ignore
            model=self.model,
            messages=self.messages,
            stream=True,
        )
        # print(type(response))
        if isinstance(response, dict):
            # fake stream output if model doesn't support it
            tmp_response = response
            text_response = response['choices'][0]['message']['content']
            tmp_response['choices'][0]['delta'] = {"content": ""}
            for i in range(0, len(text_response), 8):
                tmp_response['choices'][0]['delta']['content'] = text_response[i : i + 8]
                yield tmp_response

[krrishdholakia]

Hi @JustinStitt, what was litellm missing to be useful to you? Any feedback here would be helpful.