Adding support for llama3.2-vision

[chaos369]

llama3.2-vision is good enough in many scenarios, would you please add it to magentic?

[jackmpcollins]

@chaos369 This is possible now using using Ollama via the OpenaiChatModel. The code below (adapted from https://magentic.dev/vision/) worked for me (though it did take 3 minutes to run so you might want to test with a smaller image). Let me know if this works for you!

ollama pull llama3.2-vision

import requests
from pydantic import BaseModel, Field

from magentic import chatprompt, UserMessage, Placeholder, OpenaiChatModel
from magentic.vision import UserImageMessage


IMAGE_URL_WOODEN_BOARDWALK = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"


def url_to_bytes(url: str) -> bytes:
    """Get the content of a URL as bytes."""

    # A custom user-agent is necessary to comply with Wikimedia user-agent policy
    # https://meta.wikimedia.org/wiki/User-Agent_policy
    headers = {"User-Agent": "MagenticExampleBot (https://magentic.dev/)"}
    return requests.get(url, headers=headers, timeout=10).content


@chatprompt(
    UserMessage("Describe the following image in one sentence."),
    UserImageMessage(Placeholder(bytes, "image_bytes")),
    model=OpenaiChatModel("llama3.2-vision", base_url="http://localhost:11434/v1/")
)
def describe_image(image_bytes: bytes) -> str: ...


image_bytes = url_to_bytes(IMAGE_URL_WOODEN_BOARDWALK)
describe_image(image_bytes)