Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-lingual support for question generation and evaluation in Ragas v2 #1732

Closed
stilkin-pxl opened this issue Dec 5, 2024 · 3 comments
Closed
Labels
enhancement New feature or request module-testsetgen Module testset generation

Comments

@stilkin-pxl
Copy link

Describe the Feature
A way to generate test sets (and run evaluations) in languages other than English.

Why is the feature important for you?
I do not live in an English-speaking country.
The source material I am using is multi-lingual

Additional context
When using Ragas v1, there was an option to "adapt" the generator to different languages:

Example v1 code:
generator.adapt('french', evolutions=[simple, reasoning, conditional, multi_context], cache_dir=cache_dir)

I am not able to use this code in the new version of Ragas
(this generates the following error: AttributeError: 'TestsetGenerator' object has no attribute 'adapt')

Add any other context about the feature you want to share with us.
When using non-English source material the resulting question-answer pairs are a mix between the source language and English.

I also cannot seem to find how to do this in the v2 documentation.
If there is no translation feature present, could it be possible to append something to the generation / evaluation prompt?

@stilkin-pxl stilkin-pxl added the enhancement New feature or request label Dec 5, 2024
@dosubot dosubot bot added the module-testsetgen Module testset generation label Dec 5, 2024
@shahules786
Copy link
Member

Hey @stilkin-pxl We are working on this. Will try to ship it with this week's release it possible. Meanwhile, we have shipped items that enables you to create custom single and multi hop queries, refer docs https://docs.ragas.io/en/latest/howtos/customizations/

@shahules786
Copy link
Member

Hey @stilkin-pxl Just shipped it here #1734

@shahules786
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request module-testsetgen Module testset generation
Projects
None yet
Development

No branches or pull requests

2 participants