-
Notifications
You must be signed in to change notification settings - Fork 235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Evaluation parallele bug fix #1406
Evaluation parallele bug fix #1406
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the PR, @vishalvanpariya!
Unfortunately, it still doesn't solve the issue:
The LLM app invocation occurs for all data points before any evaluation begins. As a result, we must wait for the complete invocation process to finish across all data points before initiating any evaluations. This leads to significant delays in obtaining evaluation results, particularly for large test sets.
What we want to achieve is to start receiving evaluation outputs immediately after the first data point is processed.
@vishalvanpariya Thanks a lot for the PR! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks again for the PR @vishalvanpariya !
If I understand correctly, we are adding nest_asynco to be able to nest run_batch calls. Is there a way to change the logic so that we don't have the problem and we don't need to use nest_asyncio? It seems nest_asyncio.apply() patches asyncio which I am not very keen to as it might have side effects (what are your thoughts @aybruhm ?)
Yes, you're right. I am also concerned about why we are patching the event loop as it may result in unpredictable behaviour. @vishalvanpariya here's what I suggest doing: async def run_batch(start_idx: int):
# ... (existing code up to the loop)
tasks = [] # Store the tasks for parallel execution
for index in range(start_idx, end_idx):
task = asyncio.create_task(run_with_retry(
uri,
testset_data[index],
parameters,
max_retries,
retry_delay,
openapi_parameters,
))
tasks.append(task)
# Gather results of all tasks
results = await asyncio.gather(*tasks)
for result in results:
list_of_app_outputs.append(result)
print(f"Adding outputs to batch {start_idx}") The |
Thank you a lot @vishalvanpariya ! We have fixed the bug in a different PR. It should be live in the next oss release in an hour or so. Sorry it took so long |
@all-contributors please add @vishalvanpariya for code |
I've put up a pull request to add @vishalvanpariya! 🎉 |
Thanks @mmabrouk |
Evaluation batch parallel processing bug fixed
Bug: #1358
Additional Information
New backend dependency: nest-asyncio