Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(community) : Upgrade node-llama-cpp to be compatible with version 3 #7135

Merged
merged 16 commits into from
Nov 12, 2024

Conversation

rd4cake
Copy link
Contributor

@rd4cake rd4cake commented Nov 1, 2024

[Work in Progress]

Draft PR to address the following issue: #6994
Using asynchronous function to load models from node-llama-cpp breaks adherence to the default way of instantiating components ( chatmodels, embeddings, llms, etc ).

Here is an example:

before with node-llama-cpp v2

const llamaPath = "/path/to/your/model.gguf"; 

const model = new LlamaCpp({ modelPath: llamaPath });

after with node-llama-cpp v3

const llamaPath = "/path/to/your/model.gguf"; 

const model = await LlamaCpp.llmInit({ modelPath: llamaPath });

Question for the maintainers:
Our implementation changes the way LlamaCpp model is instantiated. Is it this fine moving forward?

Copy link

vercel bot commented Nov 1, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
langchainjs-docs 🛑 Canceled (Inspect) Nov 11, 2024 10:04pm
1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
langchainjs-api-refs ⬜️ Ignored (Inspect) Nov 11, 2024 10:04pm

@Jacky3003
Copy link
Contributor

@jacoblee93 was wondering if it was possible for you to take a look at what we have so far for our implementation, it would be greatly appreciated.

@@ -2,7 +2,7 @@ import { LlamaCppEmbeddings } from "@langchain/community/embeddings/llama_cpp";

const llamaPath = "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin";

const embeddings = new LlamaCppEmbeddings({
const embeddings = await LlamaCppEmbeddings.embeddingsInit({
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer these all to be named .initialize for consistency

_llmType() {
return "llama2_cpp";
return "llama3_cpp";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just call it llama_cpp

@@ -87,20 +91,35 @@ export class ChatLlamaCpp extends SimpleChatModel<LlamaCppCallOptions> {
return "ChatLlamaCpp";
}

constructor(inputs: LlamaCppInputs) {
private constructor(inputs: LlamaCppInputs) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this make inheritance impossible? Should it be protected instead?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And is there no sync API anymore?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, there isn't a synchronous API anymore so the current implementation is the best workaround right now. This is both outlined in the docs and the source code. And yea it does make inheritance impossible, so will change to protected, as long as the person using the library is unable to access the constructor directly, all should be fine.

Copy link
Collaborator

@jacoblee93 jacoblee93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't run through it yet but looks great overall! Some comments on naming - can you clean up?

@jacoblee93 jacoblee93 added the close PRs that need one or two touch-ups to be ready label Nov 5, 2024
@rd4cake rd4cake marked this pull request as ready for review November 7, 2024 01:53
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. auto:improvement Medium size change to existing code to handle new use-cases auto:question A specific question about the codebase, product, project, or how to use a feature labels Nov 7, 2024
Copy link
Collaborator

@jacoblee93 jacoblee93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to run for me, thank you!

@dosubot dosubot bot added the lgtm PRs that are ready to be merged as-is label Nov 11, 2024
@jacoblee93
Copy link
Collaborator

CC @nigel-daniels

Updating Forked Version With Main
@jacoblee93
Copy link
Collaborator

Just one merge to main should be fine! If you keep doing it, it requires CI to run again.

@jacoblee93 jacoblee93 changed the title [WIP] fix(community) : Upgrade node-llama-cpp to be compatible with version 3 fix(community) : Upgrade node-llama-cpp to be compatible with version 3 Nov 11, 2024
@jacoblee93 jacoblee93 merged commit 2a7a2b8 into langchain-ai:main Nov 12, 2024
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto:improvement Medium size change to existing code to handle new use-cases auto:question A specific question about the codebase, product, project, or how to use a feature close PRs that need one or two touch-ups to be ready lgtm PRs that are ready to be merged as-is size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants