Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ragas v0.2.8 error: prompt_trace.outputs.get("output", {})[0] TypeError: 'float' object is not subscriptable #1751

Closed
minmin-intel opened this issue Dec 11, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@minmin-intel
Copy link

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
When using ragas v0.2.8, I got the error: prompt_trace.outputs.get("output", {})[0] TypeError: 'float' object is not subscriptable.
I printed out the prompt_trace.outputs, the error was due to the score in prompt_trace not being a list. See printouts below.

This PR changed the code: #1743. If I revert back to ragas v0.2.6 where the code was "output": prompt_trace.outputs.get("output", {}). The error is solved.

{'output': [SentencesSimplified(sentences=[SentenceComponents(sentence_index=0, simpler_statements=['There are 38 schools with an average score in Math over
560 in the SAT test.', 'All 38 schools with an average score in Math over 560 in the SAT test are located in the Bay Area.'])])]}
{'output': [SentencesSimplified(sentences=[])]}
{'output': [ClassificationWithReason(TP=[], FP=[StatementsWithReason(statement='There are 38 schools with an average score in Math over 560 in the SAT test.'
, reason='This statement is not supported by the ground truth as there is no information provided.'), StatementsWithReason(statement='All 38 schools with an
average score in Math over 560 in the SAT test are located in the Bay Area.', reason='This statement is not supported by the ground truth as there is no info
rmation provided.')], FN=[])]}
{'output': 0.39574024755065995}

Ragas version: 0.2.8
Python version: 3.10

Code to Reproduce
Share code to reproduce the issue

Error trace
File "/home/user/GenAIEval/evals/metrics/ragas/ragas.py", line 150, in measure
self.score = evaluate(
File "/usr/local/lib/python3.10/dist-packages/ragas/_analytics.py", line 205, in wrapper
result = func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py", line 336, in evaluate
result = EvaluationResult(
File "", line 10, in init
File "/usr/local/lib/python3.10/dist-packages/ragas/dataset_schema.py", line 410, in post_init
self.traces = parse_run_traces(self.ragas_traces, run_id)
File "/usr/local/lib/python3.10/dist-packages/ragas/callbacks.py", line 167, in parse_run_traces
"output": prompt_trace.outputs.get("output", {})[0],

Expected behavior
No such error should occur.

Additional context
Add any other context about the problem here.

@minmin-intel minmin-intel added the bug Something isn't working label Dec 11, 2024
@ibi111
Copy link

ibi111 commented Dec 12, 2024

I encountered the same bug while using the following setup:

Name: ragas
Version: 0.2.8

After investigation, I identified the issue as being related to the AnswerCorrectness metric.

@Sergio-Zavota
Copy link

I had the same issue with ragas version 0.2.8 with FactualCorrectness metric.
I bumped down the version to 0.2.7 which doesn't return the error.

@shahules786
Copy link
Member

shahules786 commented Dec 13, 2024

Hey, We have raised and merged a fix for this. #1756

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants