Modifiy post processing in chat

[srdas]

Fixes #686

Removed spurious fragments from the suggestion, like backticks

    While most models (especially instruct and infill models do not require
    any pre-processing), some models such as gpt-4 which only have chat APIs
    may require removing spurious fragments (such as backticks).
    This function uses heuristics and request data to remove such fragments.
    It comments out all lines that are not code, irrespective of programming language.

Changes made to packages/jupyter-ai/jupyter_ai/tests/completions/test_handlers.py

1. All lines with backticks have been removed.
2. All text outside code blocks is commented out.

Result looks like the following:

[krassowski]

There are many languages where comments are written differently. This would generate invalid code for these languages.

Why not just modify the previous logic to strip the triple backtick suffix? Why do you want to ignore language? There is a very different use case of generating suggestions for markdown where having triple backtick is desired and of generating Python (or other) code.

[krassowski]

In general, if an LLM wants to generate comments, it should generate comments as valid language-specific syntax. It is not responsibility of jupyter-ai to transform any spurious text to comments. If jupyter-ai were to do that, one would need to write (and maintain) a very extensive multi-language logic to handle it. Instead, a LLM should be prompted differently (to only return non-code as comments with syntax matching given language). If a given LLM cannot do that, user should be advised that this is not a good LLM for code completions (ideally we would have two lists, one for chat and one for code completion; some models would be on both, but many are fine tuned for one or the other task).

[srdas]

Closing this PR as discussed here: #726 (review)