Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor to reuse stream parsing across ChatModels #380

Merged
merged 42 commits into from
Nov 29, 2024

Conversation

jackmpcollins
Copy link
Owner

@jackmpcollins jackmpcollins commented Nov 29, 2024

Big refactor. Replaces near-duplicate generator functions with better abstractions.

To parse an LLM streamed response now requires

  • StreamParser to identify string vs tool and parse these
  • StreamState to keep track of current message snapshot, usage, etc. for a streamed response

This removes the assumption that the LLM returns either string or tool calls, in preparation for #232

Improved parsing of output types. So now FunctionCall or ParallelFunctionCall is required in the return type of the prompt-function rather than allowing these to be returned when not in the type.

Also added new error types FunctionCallNotAllowedError, ObjectNotAllowedError, UnknownToolError which can be added to the LLM-assisted retry logic in future.

@jackmpcollins jackmpcollins merged commit a9d513f into main Nov 29, 2024
1 check passed
@jackmpcollins jackmpcollins deleted the use-streaming-events branch November 29, 2024 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant