Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to pass an evaluation criteria to a SimpleCriteriaScore #1763

Open
vklonin opened this issue Dec 14, 2024 · 0 comments
Open

Unable to pass an evaluation criteria to a SimpleCriteriaScore #1763

vklonin opened this issue Dec 14, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@vklonin
Copy link

vklonin commented Dec 14, 2024

[ X] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
Unable to pass an evaluation criteria to a SimpleCriteriaScore

Ragas version: 0.2.8
Python version: 3.12

Expected behavior
I want to set up my own simple validation in SimpleCriteriaScore but I am unable to, because in a code it is rewrited with some hardcoded validation criteria

self._definition = definition
self.single_turn_prompt = single_turn_prompt or SingleTurnSimpleCriteriaPrompt() - i may pass my instruction here
self.multi_turn_prompt = multi_turn_prompt or MultiTurnSimpleCriteriaPrompt() - i may pass my instruction here

update the instruction for the prompts with the definition
instruction = f"Evaluate the Input based on the criterial defined. Give a score between 0 and 5.\nCriteria Definition: {self._definition}"
self.single_turn_prompt.instruction = instruction
self.multi_turn_prompt.instruction = instruction

I also may pass my instruction in definition but it is also not working because of this instruction = f"Evaluate...

@vklonin vklonin added the bug Something isn't working label Dec 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant