Skip to content

Commit

Permalink
Expanded register_models() docs for async
Browse files Browse the repository at this point in the history
  • Loading branch information
simonw committed Nov 14, 2024
1 parent 6322040 commit e677e2c
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 1 deletion.
3 changes: 3 additions & 0 deletions docs/plugins/advanced-model-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ The {ref}`model plugin tutorial <tutorial-model-plugin>` covers the basics of de

This document covers more advanced topics.

(advanced-model-plugins-async)=

## Async models

Plugins can optionally provide an asynchronous version of their model, suitable for use with Python [asyncio](https://docs.python.org/3/library/asyncio.html). This is particularly useful for remote models accessible by an HTTP API.
Expand Down Expand Up @@ -52,6 +54,7 @@ def register_models(register):
```

(advanced-model-plugins-attachments)=

## Attachments for multi-modal models

Models such as GPT-4o, Claude 3.5 Sonnet and Google's Gemini 1.5 are multi-modal: they accept input in the form of images and maybe even audio, video and other formats.
Expand Down
17 changes: 16 additions & 1 deletion docs/plugins/plugin-hooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,20 @@ class HelloWorld(llm.Model):
def execute(self, prompt, stream, response):
return ["hello world"]
```
If your model includes an async version, you can register that too:

```python
class AsyncHelloWorld(llm.AsyncModel):
model_id = "helloworld"

async def execute(self, prompt, stream, response):
return ["hello world"]

@llm.hookimpl
def register_models(register):
register(HelloWorld(), AsyncHelloWorld(), aliases=("hw",))
```
This demonstrates how to register a model with both sync and async versions, and how to specify an alias for that model.

The {ref}`model plugin tutorial <tutorial-model-plugin>` describes how to use this hook in detail. Asynchronous models {ref}`are described here <advanced-model-plugins-async>`.

{ref}`tutorial-model-plugin` describes how to use this hook in detail.

0 comments on commit e677e2c

Please sign in to comment.