v0.34.0

What's Changed

Add StreamedResponse and AsyncStreamedResponse to enable parsing responses that contain both text and tool calls. See PR #383 or the new docs (copied below) https://magentic.dev/streaming/#StreamedResponse for more details.

⚡ StreamedResponse

Some LLMs have the ability to generate text output and make tool calls in the same response. This allows them to perform chain-of-thought reasoning or provide additional context to the user. In magentic, the StreamedResponse (or AsyncStreamedResponse) class can be used to request this type of output. This object is an iterable of StreamedStr (or AsyncStreamedStr) and FunctionCall instances.

!!! warning "Consuming StreamedStr"

The StreamedStr object must be iterated over before the next item in the `StreamedResponse` is processed, otherwise the string output will be lost. This is because the `StreamedResponse` and `StreamedStr` share the same underlying generator, so advancing the `StreamedResponse` iterator skips over the `StreamedStr` items. The `StreamedStr` object has internal caching so after iterating over it once the chunks will remain available.

In the example below, we request that the LLM generates a greeting and then calls a function to get the weather for two cities. The StreamedResponse object is then iterated over to print the output, and the StreamedStr and FunctionCall items are processed separately.

from magentic import prompt, FunctionCall, StreamedResponse, StreamedStr


def get_weather(city: str) -> str:
    return f"The weather in {city} is 20°C."


@prompt(
    "Say hello, then get the weather for: {cities}",
    functions=[get_weather],
)
def describe_weather(cities: list[str]) -> StreamedResponse: ...


response = describe_weather(["Cape Town", "San Francisco"])
for item in response:
    if isinstance(item, StreamedStr):
        for chunk in item:
            # print the chunks as they are received
            print(chunk, sep="", end="")
        print()
    if isinstance(item, FunctionCall):
        # print the function call, then call it and print the result
        print(item)
        print(item())

# Hello! I'll get the weather for Cape Town and San Francisco for you.
# FunctionCall(<function get_weather at 0x1109825c0>, 'Cape Town')
# The weather in Cape Town is 20°C.
# FunctionCall(<function get_weather at 0x1109825c0>, 'San Francisco')
# The weather in San Francisco is 20°C.

PRs

• Test Ollama via OpenaiChatModel by @jackmpcollins in #281
• Rename test to test_openai_chat_model_acomplete_ollama by @jackmpcollins in #381
• Add (Async)StreamedResponse for multi-part responses by @jackmpcollins in #383

Full Changelog: v0.33.0...v0.34.0