Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix time for runs in convert_runs_test #1338

Merged
merged 1 commit into from
Dec 17, 2024
Merged

fix time for runs in convert_runs_test #1338

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions python/langsmith/beta/_evals.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Beta utility functions to assist in common eval workflows.

Check notice on line 1 in python/langsmith/beta/_evals.py

View workflow job for this annotation

GitHub Actions / benchmark

Benchmark results 

...........
WARNING: the benchmark result may be unstable
* the standard deviation (84.7 ms) is 12% of the mean (693 ms)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

create_5_000_run_trees: Mean +- std dev: 693 ms +- 85 ms
...........
WARNING: the benchmark result may be unstable
* the standard deviation (200 ms) is 15% of the mean (1.38 sec)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

create_10_000_run_trees: Mean +- std dev: 1.38 sec +- 0.20 sec
...........
WARNING: the benchmark result may be unstable
* the standard deviation (213 ms) is 15% of the mean (1.47 sec)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

create_20_000_run_trees: Mean +- std dev: 1.47 sec +- 0.21 sec
...........
dumps_class_nested_py_branch_and_leaf_200x400: Mean +- std dev: 695 us +- 11 us
...........
dumps_class_nested_py_leaf_50x100: Mean +- std dev: 25.2 ms +- 0.3 ms
...........
dumps_class_nested_py_leaf_100x200: Mean +- std dev: 104 ms +- 2 ms
...........
dumps_dataclass_nested_50x100: Mean +- std dev: 25.5 ms +- 0.2 ms
...........
WARNING: the benchmark result may be unstable
* the standard deviation (17.6 ms) is 23% of the mean (74.9 ms)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

dumps_pydantic_nested_50x100: Mean +- std dev: 74.9 ms +- 17.6 ms
...........
dumps_pydanticv1_nested_50x100: Mean +- std dev: 202 ms +- 3 ms

Check notice on line 1 in python/langsmith/beta/_evals.py

View workflow job for this annotation

GitHub Actions / benchmark

Comparison against main 

+-----------------------------------------------+----------+------------------------+
| Benchmark                                     | main     | changes                |
+===============================================+==========+========================+
| dumps_pydanticv1_nested_50x100                | 217 ms   | 202 ms: 1.07x faster   |
+-----------------------------------------------+----------+------------------------+
| create_5_000_run_trees                        | 721 ms   | 693 ms: 1.04x faster   |
+-----------------------------------------------+----------+------------------------+
| dumps_class_nested_py_branch_and_leaf_200x400 | 695 us   | 695 us: 1.00x faster   |
+-----------------------------------------------+----------+------------------------+
| dumps_class_nested_py_leaf_100x200            | 103 ms   | 104 ms: 1.00x slower   |
+-----------------------------------------------+----------+------------------------+
| create_10_000_run_trees                       | 1.37 sec | 1.38 sec: 1.01x slower |
+-----------------------------------------------+----------+------------------------+
| dumps_dataclass_nested_50x100                 | 25.2 ms  | 25.5 ms: 1.01x slower  |
+-----------------------------------------------+----------+------------------------+
| dumps_class_nested_py_leaf_50x100             | 24.8 ms  | 25.2 ms: 1.01x slower  |
+-----------------------------------------------+----------+------------------------+
| create_20_000_run_trees                       | 1.36 sec | 1.47 sec: 1.07x slower |
+-----------------------------------------------+----------+------------------------+
| dumps_pydantic_nested_50x100                  | 64.9 ms  | 74.9 ms: 1.16x slower  |
+-----------------------------------------------+----------+------------------------+
| Geometric mean                                | (ref)    | 1.02x slower           |
+-----------------------------------------------+----------+------------------------+

These functions may change in the future.
"""
Expand Down Expand Up @@ -163,6 +163,9 @@
)

for new_run in to_create:
latency = new_run["end_time"] - new_run["start_time"]
new_run["start_time"] = datetime.datetime.now(tz=datetime.timezone.utc)
new_run["end_time"] = new_run["start_time"] + latency
client.create_run(**new_run, project_name=test_project_name)

_ = client.update_project(
Expand Down
Loading