Integration with transformers

[deepdiffuser]

It would be interesting to be able to use loom with open source LLMs such as GPT-Neo-X, FLAN-UL2, and LLaMA. The transformers library by Huggingface has support for almost every open source LLM through a standardized interface.

One approach to accomplish this could be direct integration. Another approach, to keep the loom client thin, could be to develop (maybe this already exists?) a shim that adapts the OpenAI API shape to a transformers backend

[mirrorcult]

https://github.com/oobabooga/text-generation-webui, which just runs transformers (and has support for LLaMA etc) has an API which is pretty similar to the openAI shape--probably not hard to adapt loom to post requests to the local API, I was planning on doing this

[deepdiffuser]

I just discovered https://github.com/hyperonym/basaran which should accomplish this!

[mirrorcult]

interesting, that looks much better

[kirill-fedyanin]

@mirrorcult Have you succeed to launch loom on llama?

[Technologicat]

I'd like to second this.

Having read the blog posts on Loom, this thing borders on the legendary... so would be very nice to get it working with a local LLM. Even though oobabooga's text-generation-webui nowadays provides an OpenAI-compatible API, Loom seems hardcoded to call OpenAI's servers.

Most LLM tools are missing the multiverse features, which are kind of the point of this.

SillyTavern with its Timelines extension comes close, for chat mode, but it'd be really nice to play around with base models with a tool like this.