fix: Fix DB unittest reliability #4548
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
the
size:S
anticorrelator
force-pushed
the
dustin/lifespan-testing
branch
from
b3cfce7 to
fced295
Compare
anticorrelator
changed the title
axiomofjoy
reviewed
Sep 12, 2024
| @@ -248,21 +248,21 @@ dependencies = [ | |||
| ] | |||
|
|
|||
| [tool.hatch.envs.default.scripts] | |||
| tests = "pytest -n auto {args}" | |||
| tests = "pytest {args}" | |||
There was a problem hiding this comment.
Do we no longer need pytest-xdist?
There was a problem hiding this comment.
how many cores are we using if we don't specify this option? do we still need pytest-xdist?
| @@ -116,7 +111,7 @@ def openai_api_key(monkeypatch: pytest.MonkeyPatch) -> str: | |||
| postgresql_connection = factories.postgresql("postgresql_proc") | |||
|
|
|||
|
|
|||
| @pytest.fixture() | |||
| @pytest.fixture(scope="function") | |||
There was a problem hiding this comment.
Isn't "function" the default scope?
There was a problem hiding this comment.
yes, but it's better to be explicit in case the default changes
There was a problem hiding this comment.
okay. fwiw i doubt pytest will change that default since it would be a very disruptive change.
axiomofjoy
approved these changes
Sep 19, 2024
| @@ -248,21 +248,21 @@ dependencies = [ | |||
| ] | |||
|
|
|||
| [tool.hatch.envs.default.scripts] | |||
| tests = "pytest -n auto {args}" | |||
| tests = "pytest {args}" | |||
There was a problem hiding this comment.
how many cores are we using if we don't specify this option? do we still need pytest-xdist?
anticorrelator
force-pushed
the
dustin/lifespan-testing
branch
from
3f481ca to
da69b6c
Compare
anticorrelator
force-pushed
the
dustin/lifespan-testing
branch
from
da69b6c to
07b218a
Compare
RogerHYang
pushed a commit
that referenced
this pull request
Sep 21, 2024
* Remove busywait * Ruff 🐶 * Use closure loop * Use nest-asyncio for nested asgi fixture management * Wait for db insertions before reading in test * Ensure the entire experiment has run * Experiment with locks * xfail unstable tests * Use asyncio.sleep before querying database after client interactions * Ruff 🐶 * Reduce number of evaluators to make tests more reliable * Only bypass lock for unittests * Convert to an integration test * Set default loop scope for unit tests * Remove loop policy * xfail tests where evals do not reliably write to the database * Ensure databases are function scoped * Ensure inmemory sqlite testing * Ruff 🐶 * Wipe DBs between tests * Continue github actions on error * Use async sleep in spans test * Remove needless import * Refactor engine setup to potentially reduce deadlock risk * Wait for evaluations for more stable tests * Don't continue on failure * Ruff 🐶 * BulkInsterters insert immediately in tests * Remove xdist * Increase timeout to 30 * Xfail test * Use shared cache * Use tempfile based sqlite db * Use tempdirs for windows compatibility * Xfail test again * Wait a waiter to llm eval test * Skip flaky tests only on windows and mac
This was referenced Sep 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates our DB management strategy for unittests, this resolves the majority of stability issues we were seeing in CI.