Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG][playground] unhandled error in subscription is not thrown in onError instead causes onCompleted to run when running on dataset #5441

Open
Tracked by #3435
Parker-Stafford opened this issue Nov 20, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@Parker-Stafford
Copy link
Contributor

Parker-Stafford commented Nov 20, 2024

Describe the bug
When running the playground on a dataset if you hit an unhandled error the subscription onCompleted callback runs but not the onError (this is not the case with the mutation).

To Reproduce
Steps to reproduce the behavior:

  1. Go to playground
  2. select a dataset
  3. switch to an anthropic model
  4. don't set the max tokens (required, will throw)
  5. click run and see subscription complete with no info

Expected behavior
onError would run (since unhandled) and user would be notified or error

Additional context
onCompleted is running when the error is hit

error stack from server for one of the examples in the dataset, this same stack is logged for all examples in dataset

Required parameter max_tokens not provided
Traceback (most recent call last):
  File "/Users/parkerstafford/Library/Application Support/uv/tools/arize-phoenix/lib/python3.12/site-packages/phoenix/server/api/subscriptions.py", line 290, in chat_completion_over_dataset
    yield task.result()
          ^^^^^^^^^^^^^
  File "/Users/parkerstafford/Library/Application Support/uv/python/cpython-3.12.4-macos-aarch64-none/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/Users/parkerstafford/Library/Application Support/uv/tools/arize-phoenix/lib/python3.12/site-packages/phoenix/server/api/subscriptions.py", line 435, in _as_coroutine
    return await iterable.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/parkerstafford/Library/Application Support/uv/tools/arize-phoenix/lib/python3.12/site-packages/phoenix/server/api/subscriptions.py", line 326, in _stream_chat_completion_over_dataset_example
    invocation_parameters = llm_client.construct_invocation_parameters(input.invocation_parameters)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/parkerstafford/Library/Application Support/uv/tools/arize-phoenix/lib/python3.12/site-packages/phoenix/server/api/helpers/playground_clients.py", line 209, in construct_invocation_parameters
    validate_invocation_parameters(supported_params, formatted_invocation_parameters)
  File "/Users/parkerstafford/Library/Application Support/uv/tools/arize-phoenix/lib/python3.12/site-packages/phoenix/server/api/input_types/InvocationParameters.py", line 147, in validate_invocation_parameters
    raise ValueError(f"Required parameter {param_def.invocation_name} not provided")
@Parker-Stafford Parker-Stafford added bug Something isn't working triage issues that need triage labels Nov 20, 2024
@github-project-automation github-project-automation bot moved this to 📘 Todo in phoenix Nov 20, 2024
@RogerHYang RogerHYang removed the triage issues that need triage label Nov 21, 2024
@axiomofjoy
Copy link
Contributor

Raising an error from subscription would stop the entire experiment run, e.g., if one error occurs one a single dataset example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 📘 Todo
Development

No branches or pull requests

3 participants