Add Ollama #646

jtpio · 2024-02-16T08:52:34Z

Fixes #482

Ollama seems to be getting popular for running models locally, and looks like it would be good to have in Jupyter AI by default.

jtpio · 2024-02-16T08:53:33Z

I guess it would also be fine to have it as a custom model provider in a separate package, if we don't want to maintain it here in this repo.

dlqqq · 2024-02-17T04:47:38Z

Jeremy, thank you very much for working on this PR and for keeping up with the open source LLM ecosystem. I lack the time to engage there as much as I would like.

Let me know when this PR is ready, and I will approve & merge it once I run it locally. 👍

lalanikarim · 2024-02-20T02:58:35Z

Looking forward to this integration.
Is it feasible to have the list of supported models as configurable given that there are numerous finetunes out there that would be viable candidates with Ollama?

jtpio · 2024-02-20T10:51:18Z

Thanks @dlqqq and @lalanikarim!

Yeah I'll try to finish the PR soon. It was originally opened early to see if there was interest to have built-in support for Ollama in jupyter-ai.

Is it feasible to have the list of supported models as configurable given that there are numerous finetunes out there that would be viable candidates with Ollama?

Currently jupyter_ai seems to have support for allowing and blocking models:

jupyter-ai/packages/jupyter-ai/jupyter_ai/extension.py

Lines 61 to 75 in 642ac53

    
               allowed_models = List( 
        
                   Unicode(), 
        
                   default_value=None, 
        
                   help=""" 
        
                   Language models to allow, as a list of global model IDs in the format 
        
                   `<provider>:<local-model-id>`. If `None`, all are allowed. Defaults to 
        
                   `None`. 
        
                   Note: Currently, if `allowed_providers` is also set, then this field is 
        
                   ignored. This is subject to change in a future non-major release. Using 
        
                   both traits is considered to be undefined behavior at this time. 
        
                   """, 
        
                   allow_none=True, 
        
                   config=True, 
        
               )

Although not sure if that would be enough to use models that have not been added to the list of models in jupyter_ai.

startakovsky · 2024-02-24T16:33:33Z

🙇 @jtpio excited for this

lalanikarim · 2024-03-03T17:12:54Z

Since Ollama sports an OpenAI compatible Rest API, and jupyter-ai supports setting a base url to override the default OpenAI url, I created new local models based on models supported by Ollama and named them to match OpenAI models.

$ cat mistral-gpt-4.modelfile
FROM mistral

$ ollama create gpt-4 -f mistral-gpt-4.modelfile

This hack currently allows me to use Ollama with jupyter-ai.

Looking forward to integration to support local models hosted with Ollama.

triinity2221 · 2024-03-04T08:04:35Z

@lalanikarim - I am also replicating a similar setup. Just curious are you able to use the /learn, /generate commands in the chat

lalanikarim · 2024-03-05T03:15:17Z

@lalanikarim - I am also replicating a similar setup. Just curious are you able to use the /learn, /generate commands in the chat

I haven't had any luck with /generate. I run into pydantic errors.

pydantic.v1.error_wrappers.ValidationError: 1 validation error for Outline
sections
  field required (type=value_error.missing)

I haven't tried /learn yet.

lalanikarim · 2024-03-05T03:21:08Z

@jtpio I am wondering if it would make sense to provide model name in a free form text field for Ollama models and to require %%ai magic code to include model name for ollama models and limit models to a predefined list.

%%ai ollama:mistral

Thoughts?

jtpio · 2024-03-05T16:39:08Z

@jtpio I am wondering if it would make sense to provide model name in a free form text field for Ollama models and to require %%ai magic code to include model name for ollama models and limit models to a predefined list.

Yeah I guess given the number of available models it would be very difficult to pick just a few here in the jupyter-ai package. And that would also require updating Jupyter AI regularly, increasing maintenance load.

So yes having a way to let users configure the list of models sounds like a good idea.

siliconberry · 2024-03-11T12:21:29Z

Since Ollama sports an OpenAI compatible Rest API, and jupyter-ai supports setting a base url to override the default OpenAI url, I created new local models based on models supported by Ollama and named them to match OpenAI models.
$ cat mistral-gpt-4.modelfile
FROM mistral

$ ollama create gpt-4 -f mistral-gpt-4.modelfile
This hack currently allows me to use Ollama with jupyter-ai.

Looking forward to integration to support local models hosted with Ollama.

Hi, Thanks for the hack. Its working. But curious.. how/what did you set the OPENAI API key ? Its working inside the chat box (Jupyternaut), but not inside the jupyter notebook. and its asking for OPEN_API_KEY. can u pl help ?

Mrjaggu · 2024-03-13T11:38:28Z

Anything update on this pr for adding ollama to jupyter ai..

jtpio · 2024-03-13T13:35:48Z

I'll try to finish this PR soon, and provide a way to configure the list of models (+ docs).

dannongruver · 2024-04-02T03:41:17Z

@jtpio: Thank you for this ollama integration Jeremy! Also look forward to configurable models. Let me know where I can buy you a coffee.

lalanikarim · 2024-04-03T00:56:50Z

Hi, Thanks for the hack. Its working. But curious.. how/what did you set the OPENAI API key ? Its working inside the chat box (Jupyternaut), but not inside the jupyter notebook. and its asking for OPEN_API_KEY. can u pl help ?

@siliconberry You can put any value for OPANAI_API_KEY. It is needed for this hack to work but will be ignored by ollama.

dannongruver · 2024-04-03T04:29:43Z

@lalanikarim in answer to @siliconberry question, have you gotten both the chat/jupyternaut as well as notebook cell (ie magic %%ai) working with ollama?

I’m struggling now to get notebook working. When I set the OPENAI_API_KEY environment variable to a dummy key (sk-abcd1234), the notebook gives an error indicating that ChatGPT doesn’t accept the key. Jupyternaut chat works fine. It seems like either jupyter_ai’s notebook is not using the config.json like Jupyternaut or ollama does not support all ChatGPT api…? Idk, maybe I’m missing something

dannongruver · 2024-04-03T11:12:01Z

Nevermind @lalanikarim @siliconberry. The magic %%ai does work.

Note: The magic doesn’t use the config.json but uses environment vars for key and url.

cboettig · 2024-05-10T19:44:49Z

Just joining the chorus of folks who would be excited to see this, especially given the rate at which new open-weights models are regularly appearing for ollama (llama3, phi3, wizardlm2). ollama's use of local GPU seems a lot smoother too than gpt4all too.

orkutmuratyilmaz · 2024-05-13T08:15:37Z

I'm so excited for this! 😍😍😍

pedrogutobjj · 2024-06-15T12:42:43Z

any updates?

srdas · 2024-07-02T14:03:55Z

@jtpio Please see also recent PR #868 in case this includes the remaining ideas you planned to implement?

jtpio · 2024-07-03T06:40:38Z

Thanks @srdas for the heads-up 👍

Looks like #868 lists more models, so we could continue with this PR instead of this one. But it would still be interesting for end users to have a way to configure the models, to avoid having to update Jupyter AI and make a new release each time there is a new model out there.

srdas · 2024-07-03T08:11:41Z

Thanks @srdas for the heads-up 👍

Looks like #868 lists more models, so we could continue with this PR instead of this one. But it would still be interesting for end users to have a way to configure the models, to avoid having to update Jupyter AI and make a new release each time there is a new model out there.

Thanks @jtpio -- let me test the new PR locally and also summarize what remains from what you have proposed. The drop down list is becoming really long so keeping on adding new models at the pace they are being included in Ollama may not be ideal both from a maintenance point of view and from an interface one. It may be preferable to handle Ollama in the way HuggingFace models are treated in the drop-down listing, which may address your note above.

dlqqq · 2024-07-08T15:28:21Z

@jtpio Thank you for checking in on this again! Since your PR does not duplicate the LangChain provider implementation and is older than #868, I think this PR (#646) should take precedence. Here are the next steps towards getting this merged:

Rebase onto latest main, addressing conflicts
Add additional model IDs listed in supporting local model for jupyter-ai with ollama #868 but not here. Please co-author @767472021 on the commit for contribution credit.
Mark this PR as ready for review

Let me know if you need assistance!

dlqqq · 2024-07-08T15:30:19Z

But it would still be interesting for end users to have a way to configure the models, to avoid having to update Jupyter AI and make a new release each time there is a new model out there.

@jtpio I agree! This should probably be done in a future PR however, as it is a broader change that is not strictly necessary for Ollama support in Jupyter AI.

docs/source/users/index.md

srdas · 2024-07-10T21:25:57Z

@jtpio Awesome - the changes are working nicey. I tested it on the new embedding model list and it all seems to be working well.

Co-authored-by: Bc <[email protected]>

Co-authored-by: Piyush Jain <[email protected]> Co-authored-by: david qiu <[email protected]>

srdas · 2024-07-10T21:46:37Z

docs/source/users/index.md

+### Ollama usage
+
+To get started, follow the instructions on the [Ollama website](https://ollama.com/) to set up `ollama` and download the models locally. To select a model, enter the model name in the settings panel, for example `deepseek-coder-v2`.
+


Use Ollama capitalized in line 362 in "... to set up ollama ... "

Suggest adding a few more notes to the docs as follows:

Once Ollama is installed, download the model using the following command in a terminal:

ollama pull <model_name>

For example: ollama pull llama3. Do this for both LLM models and embedding models, as many as needed.

Once the models are downloaded, start up the Ollama server:

ollama serve

The models will then be available in jupyter-ai.

Good suggestion, but wouldn't block on this to merge. Also, this info can change based on ollama API, so we will be carrying instructions that are available in their docs and will have to update in case this changes.

@3coins Agree. I tested it again and it's all working so merging is predicated.

srdas

Thanks for adding the code to enable model choice.

3coins

@jtpio
Looks great!

jtpio · 2024-07-11T05:59:35Z

Thanks all for the reviews and the merge!

pedrogutobjj · 2024-07-12T17:49:56Z

I'm using the test version of 2.20, and I noticed that the line completion is not working, I tested it with the same model in VSCODE using the continue plugin and the code completion works, could it be a bug?

srdas · 2024-07-12T18:17:44Z

@pedrogutobjj - it is working at my end. Did you enable Inline completion in settings?

Example:

pedrogutobjj · 2024-07-12T18:19:05Z

my configs @srdas

pedrogutobjj · 2024-07-12T18:34:30Z

@srdas
i need use -instruct for completion model? Or can I use the same one?

Another question, if I enable the "Whether to fetch completions History provider." It doesn't provide suggestions for models, it provides suggestions for what I've used on other notebooks in the past.

srdas · 2024-07-12T18:44:39Z

You can use any model - I just used the instruct model as I could not find the version you are using on https://ollama.com/search. It shows this:

The run command gives a hint at what you need to name the model.
Also after you pull the Ollama model, you can run ollama list to see the exact name of the model.

I have to look into your second question some more.

pedrogutobjj · 2024-07-12T18:47:48Z

@srdas

pedrogutobjj · 2024-07-12T19:07:51Z

@srdas i need use -instruct for completion model? Or can I use the same one?

Another question, if I enable the "Whether to fetch completions History provider." It doesn't provide suggestions for models, it provides suggestions for what I've used on other notebooks in the past.

This happens when I activate the Whether to fetch completions History provider option, it brings codes that I have already used in the past, it has nothing to do with the model indications, as you mentioned in your post.

@srdas

pedrogutobjj · 2024-07-12T19:31:24Z

Another strange situation, even though there is no suggestion to complete the line, the gpu usage is at maximum, as if using a model, but nothing happens.

kilmarnock · 2024-09-28T11:55:48Z

for the records:

%%ai ollama:qwen2.5-coder:7b-instruct-q8_0 -m {"base_url":"http://192.168.x.y:11434"}
write some command in python

works.

Sort of

* Add Ollama * mistral:text for embeddings * Add gemma * Update the list of models Co-authored-by: Bc <[email protected]> * Mention Ollama in the docs * Apply suggestions from code review Co-authored-by: Piyush Jain <[email protected]> Co-authored-by: david qiu <[email protected]> --------- Co-authored-by: Bc <[email protected]> Co-authored-by: Piyush Jain <[email protected]> Co-authored-by: david qiu <[email protected]>

jtpio added the enhancement New feature or request label Feb 16, 2024

jtpio force-pushed the ollama branch from d862068 to b5df73c Compare June 20, 2024 17:40

srdas mentioned this pull request Jul 2, 2024

supporting local model for jupyter-ai with ollama #868

Closed

jtpio force-pushed the ollama branch 2 times, most recently from 697cf70 to bfb036a Compare July 10, 2024 08:42

3coins reviewed Jul 10, 2024

View reviewed changes

docs/source/users/index.md Outdated Show resolved Hide resolved

3coins reviewed Jul 10, 2024

View reviewed changes

docs/source/users/index.md Outdated Show resolved Hide resolved

jtpio and others added 6 commits July 10, 2024 14:45

Add Ollama

12f5bf4

mistral:text for embeddings

226670a

Add gemma

c36dce2

Update the list of models

88a80f8

Co-authored-by: Bc <[email protected]>

Mention Ollama in the docs

6843ce3

Apply suggestions from code review

224aac8

Co-authored-by: Piyush Jain <[email protected]> Co-authored-by: david qiu <[email protected]>

srdas reviewed Jul 10, 2024

View reviewed changes

3coins force-pushed the ollama branch from ee47992 to 224aac8 Compare July 10, 2024 21:47

srdas reviewed Jul 10, 2024

View reviewed changes

3coins approved these changes Jul 10, 2024

View reviewed changes

3coins merged commit 9f70ea7 into jupyterlab:main Jul 10, 2024
8 checks passed

jtpio deleted the ollama branch July 11, 2024 05:59

krassowski mentioned this pull request Jul 11, 2024

v2.19.0 release? #891

Closed

krassowski mentioned this pull request Jul 16, 2024

Ollama starcoder2:15b inline completions model does not work #896

Open

This was referenced Sep 22, 2024

Custom local LLMs #389

Open

add VLLM support? #1010

Open

		### Ollama usage

		To get started, follow the instructions on the [Ollama website](https://ollama.com/) to set up `ollama` and download the models locally. To select a model, enter the model name in the settings panel, for example `deepseek-coder-v2`.

Add Ollama #646

Add Ollama #646

Conversation

jtpio commented Feb 16, 2024 • edited Loading

jtpio commented Feb 16, 2024

dlqqq commented Feb 17, 2024

lalanikarim commented Feb 20, 2024

jtpio commented Feb 20, 2024

startakovsky commented Feb 24, 2024 • edited Loading

lalanikarim commented Mar 3, 2024

triinity2221 commented Mar 4, 2024

lalanikarim commented Mar 5, 2024

lalanikarim commented Mar 5, 2024

jtpio commented Mar 5, 2024

siliconberry commented Mar 11, 2024

Mrjaggu commented Mar 13, 2024 • edited Loading

jtpio commented Mar 13, 2024

dannongruver commented Apr 2, 2024

lalanikarim commented Apr 3, 2024

dannongruver commented Apr 3, 2024

dannongruver commented Apr 3, 2024

cboettig commented May 10, 2024

orkutmuratyilmaz commented May 13, 2024

pedrogutobjj commented Jun 15, 2024

srdas commented Jul 2, 2024

jtpio commented Jul 3, 2024

srdas commented Jul 3, 2024

dlqqq commented Jul 8, 2024 • edited Loading

dlqqq commented Jul 8, 2024

srdas commented Jul 10, 2024

srdas Jul 10, 2024 • edited Loading

Choose a reason for hiding this comment

3coins Jul 10, 2024

Choose a reason for hiding this comment

srdas Jul 10, 2024

Choose a reason for hiding this comment

srdas left a comment

Choose a reason for hiding this comment

3coins left a comment

Choose a reason for hiding this comment

jtpio commented Jul 11, 2024 • edited Loading

pedrogutobjj commented Jul 12, 2024

srdas commented Jul 12, 2024

pedrogutobjj commented Jul 12, 2024

pedrogutobjj commented Jul 12, 2024

srdas commented Jul 12, 2024

pedrogutobjj commented Jul 12, 2024

pedrogutobjj commented Jul 12, 2024

pedrogutobjj commented Jul 12, 2024

kilmarnock commented Sep 28, 2024 • edited Loading

jtpio commented Feb 16, 2024 •

edited

Loading

startakovsky commented Feb 24, 2024 •

edited

Loading

Mrjaggu commented Mar 13, 2024 •

edited

Loading

dlqqq commented Jul 8, 2024 •

edited

Loading

srdas Jul 10, 2024 •

edited

Loading

jtpio commented Jul 11, 2024 •

edited

Loading

kilmarnock commented Sep 28, 2024 •

edited

Loading