Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug Fix: Custom code evaluation aggregation #1347

Merged
merged 7 commits into from
Feb 7, 2024
Merged

Bug Fix: Custom code evaluation aggregation #1347

Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
69f795c
Update - refactor evaluator_results aggregation logic
aybruhm Feb 7, 2024
68f527d
:art: Format - ran black
aybruhm Feb 7, 2024
5979886
Update - format black using the latest version
aybruhm Feb 7, 2024
7a4731c
Feat - created aggregation_service and implemented aggregation logics
aybruhm Feb 7, 2024
26512a4
Update - refactor aggregate_evaluator_results function
aybruhm Feb 7, 2024
2baa7d4
Update - resolve concerns in aggregation logic
aybruhm Feb 7, 2024
09019ed
Update evaluations.py
mmabrouk Feb 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions agenta-backend/agenta_backend/tasks/evaluations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,8 +338,13 @@ async def aggregate_evaluator_results(
)

else:
# Handle boolean values for auto_regex_test and other evaluators
if all(isinstance(result.value, bool) for result in results):
if evaluator_key == "auto_regex_test" and all(
isinstance(result.value, bool) for result in results
):
average_value = sum(result.value for result in results) / len(
aybruhm marked this conversation as resolved.
Show resolved Hide resolved
results
)
elif evaluator_key not in ["auto_ai_critique", "auto_regex_test"]:
average_value = sum(result.value for result in results) / len(
results
)
Expand Down
Loading