Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improvement: Resolves OpenAI rate-limiting issue when running large test sets in the playground/evaluation #1007

Closed
wants to merge 4 commits into from
Closed

Improvement: Resolves OpenAI rate-limiting issue when running large test sets in the playground/evaluation #1007

wants to merge 4 commits into from

Conversation

aybruhm
Copy link
Member

@aybruhm aybruhm commented Dec 6, 2023 •
edited
Loading

Description

This PR improves the handling process of large testsets by sequentially executing requests with a backoff mechanism to alleviate the openai rate limiting issue.

Related Issue

Closes C_188

@aybruhm aybruhm changed the title Improvement: Resolving the OpenAI rate-limiting issue when running large test sets in the playground/evaluation Improvement: Resolves OpenAI rate-limiting issue when running large test sets in the playground/evaluation Dec 6, 2023
@aakrem aakrem removed the request for review from mmabrouk December 7, 2023 07:34
@aakrem
Copy link
Collaborator

aakrem commented Dec 7, 2023

https://platform.openai.com/docs/guides/rate-limits/usage-tiers?context=tier-one

@MohammedMaaz MohammedMaaz mentioned this pull request Dec 7, 2023
Closed
@MohammedMaaz
Copy link
Contributor

Thanks for the PR @aybruhm

I've created a new PR (#1014) to cater the retrying and batching logic in a more generic way inspired by your approach.

We can continue to address TPM rate limiting in the new PR.

cc @aakrem

@aybruhm aybruhm closed this Dec 7, 2023
@MohammedMaaz MohammedMaaz mentioned this pull request Dec 11, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MohammedMaaz MohammedMaaz Awaiting requested review from MohammedMaaz

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@aybruhm @aakrem @MohammedMaaz