Tracking usage for all calls made

[djaddis]

I can only see a way of tracking token usage / cost for the chat models, is this possible with the other regular prompts too?

Great package though!

[jackmpcollins]

Hi @djaddis , the @prompt/decorator syntax doesn't give much room for surfacing information from the query beyond the return type of the function. Additionally, magentic uses streaming for all queries to the LLM in order to support the streamed return types, and usage info doesn't become available until the end of the stream. This means for a prompt-function like make_heroes() -> Iterable[SuperHero], there is no usage info after running heroes = make_heroes() until heroes has been consumed.

Based on that, it seems the only way to solve this generally would be to allow providing a callback function that is called when the usage is received for a message. When experimenting with this previously I found this was limited because often you want to use the AssistantMessage in the callback but this creates an issue where the for x in message.content line running at the same time as the callback trying to use message causes conflict.

For the moment, if your responses are not streamed you could create a wrapper ChatModel that does something with the usage after each LLM query, like described here #74 (comment) This could be used with @prompt and other decorators by passing it as the model argument. Let me know if this is enough for your use case, or what you would need here. Thanks