-
-
Notifications
You must be signed in to change notification settings - Fork 341
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix prefix removal when streaming inline completions #879
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@krassowski This looks good aside from some minor typos which I've called out below, thank you for making this change! Do you recall the previous reason why we were only trimming the prefix conditionally when the response began/ended with ```? Want to verify that we're not making a regression unexpectedly.
This was just a premature optimization. Briefly, the idea was that instead of storing two versions of the So with the previous logic (where we wanted to store only one version) if we got the following reply:
in four chunks:
then the old logic (if we did not have the conditional) would trim the first chunk away, but accept second and third chunk (and possibly the fourth - I do not remember), so the result would be:
or (if the code was only trimming the closing fence if it detected an opening fence - which I think a version of the code did)
In either case, we would not want this to happen, the intended result is just:
Now after this PR we are reapplying post-processing after each fragment and do not need to worry about these. While this has a negligibly higher cost, it is even more appropriate because the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@krassowski Thanks for lending context & addressing my feedback! Proceeding to merge.
Co-authored-by: david qiu <[email protected]>
85a3f31
to
267a8b9
Compare
* Fix removal of prefix when streaming * Fix typo Co-authored-by: david qiu <[email protected]> --------- Co-authored-by: david qiu <[email protected]>
Fixes #841