[R-313] Adding more LLM/Embedding Providers

[jjmachan]

A running issue to get feedback on which LLM or Embedding Providers we need to add for you

_R-313

[kent-william007]

If the output of my model is obtained through a method call, for example, def query_model(query): return response. Then how should I write the code to load the model? In addition, the Embedding model is also generated through a similar method call, providing methods for converting words into vectors and comparing vector similarities.

[jjmachan]

hey @kent-william007, in that case you will have to implement

ragas/src/ragas/llms/base.py

Lines 64 to 83 in 1e7121b

	@abstractmethod
	def generate_text(
	self,
	prompt: PromptValue,
	n: int = 1,
	temperature: float = 1e-8,
	stop: t.Optional[t.List[str]] = None,
	callbacks: Callbacks = None,
	) -> LLMResult: ...
	@abstractmethod
	async def agenerate_text(
	self,
	prompt: PromptValue,
	n: int = 1,
	temperature: t.Optional[float] = None,
	stop: t.Optional[t.List[str]] = None,
	callbacks: Callbacks = None,
	) -> LLMResult: ...

on your own and make sure make sure that the result is of the type https://python.langchain.com/api_reference/core/outputs/langchain_core.outputs.llm_result.LLMResult.html

similarly for https://github.com/explodinggradients/ragas/blob/main/src/ragas/embeddings/base.py

let me know if you have further doubts, we'll help you out 🙂

btw are you on discord?

[kent-william007]

Is it possible to support model invocation through an API? This way, we can leverage the model capabilities provided by the server without needing to worry about which underlying model is being used.

[matheusft]

Trying to use Ragas with Google's Gemini models always gives me some headache.

Whether I'm Evaluating Using Metrics or trying to Generate Synthetic Testset for RAG I keep getting errors such as The LLM generation was not completed. Please increase try increasing the max_tokens and try again.

Same result with different models e.g. gemini-1.5-pro-001,gemini-1.5-pro-002, gemini-1.5-flash.

I just commented on #1573 about getting the same thing.

Gemini support, please.
🟦🟥🟨🟩

[jjmachan]

@matheusft could you check this PR and docs: #1728 this should help you.

let me know if it solves your issue or I'll look further into it

[baptvit]

@jjmachan I was having the same issue using google`s api. Seens like the #1728 works here with the googles api.

[matheusft]

@matheusft could you check this PR and docs: #1728 this should help you.

let me know if it solves your issue or I'll look further into it

It does not work for me as LangchainLLMWrapper does not implement the .invoke() method, so it's not working with my LangChain RAG.

I'm now getting

Expected a Runnable, callable or dict.Instead got an unsupported type: <class 'ragas.llms.base.LangchainLLMWrapper'>

[tee-wealth001]

i get this error

AttributeError: 'GoogleGenerativeAIEmbeddings' object has no attribute 'set_run_config'
I used the LangchainLLMWrapper for the llm, but there seems to be an issue with embeddings.
I don't know why most libraries strictly enforces OPENAI when not all uses Open AI
Note: I am using gemini llm and embeddings

[jjmachan]

@matheusft could you check this PR and docs: #1728 this should help you.
let me know if it solves your issue or I'll look further into it

It does not work for me as LangchainLLMWrapper does not implement the .invoke() method, so it's not working with my LangChain RAG.

I'm now getting

Expected a Runnable, callable or dict.Instead got an unsupported type: <class 'ragas.llms.base.LangchainLLMWrapper'>

wrapped classes are only intended to be used inside Ragas, it is recommended to not use it anywhere else

[matheusft]

Trying to use Ragas with Google's Gemini models always gives me some headache.

Whether I'm Evaluating Using Metrics or trying to Generate Synthetic Testset for RAG I keep getting errors such as The LLM generation was not completed. Please increase try increasing the max_tokens and try again.

Same result with different models e.g. gemini-1.5-pro-001,gemini-1.5-pro-002, gemini-1.5-flash.

I just commented on #1573 about getting the same thing.

Gemini support, please. 🟦🟥🟨🟩

So would you have any suggestion on how to use Ragas with Gemini and Langchain that actually works?

[jjmachan]

hey @matheusft let me check this and get back to you. We had a community PR that should have helped fix this but I think that didn't solve the issue your facing? let me setup the google gemini workflow and get back to you by today 🙂

[tee-wealth001]

Does anyone know how else on can evaluate a rag system without ragas?

[jjmachan]

hey @matheusft could you check this PR out: #1759

I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂

[matheusft]

hey @matheusft could you check this PR out: #1759

I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂

Fantastic!
I’ll give it a go this week and will let you know the outcome.

Thanks in advance.