/learn - not enough values to unpack (expected 2, got 0)

[El-Don-Quijote]

Description

Hello,
i am using chat-gpt 3.5 with jupyter-ai and the basic chatting and also including selections works flawless. What however does not work is the \learn command. When specifiying and incorrect path, it correctly identifies that this path does not exist.

When I specify a correct path, this is what jupyternaut returns:

Sorry, something went wrong and I wasn't able to index that path.

  File "C:\Users\XXX\mambaforge\envs\XXX\Lib\site-packages\jupyter_ai\chat_handlers\base.py", line 38, in process_message
    await self._process_message(message)

  File "C:\Users\XXX\mambaforge\envs\XXX\Lib\site-packages\jupyter_ai\chat_handlers\learn.py", line 119, in _process_message
    await self.learn_dir(load_path, args.chunk_size, args.chunk_overlap)

  File "C:\Users\XXX\mambaforge\envs\XXX\Lib\site-packages\jupyter_ai\chat_handlers\learn.py", line 156, in learn_dir
    self.index.add_embeddings(*embedding_records)

  File "C:\Users\XXX\mambaforge\envs\XXX\Lib\site-packages\langchain\vectorstores\faiss.py", line 176, in add_embeddings
    texts, embeddings = zip(*text_embeddings)

    ^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 0)

Expected behavior

I would expect that Jupyter AI would start to learn about the specified folder and that I can then ask questions about it.
Any help much appreciated & thanks in advance!

Context

• Operating System and version: Windows 11
• Browser and version: 116.0.3
• JupyterLab version: jupyterlab 4.0.5
• Jupyter AI version: jupyter_ai 2.1.0

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[JasonWeill]

To be clear, the command is /learn with a forward slash — is that what you're using?

Also, which embedding model are you using in chat settings?

[Andrea23Romano]

In my experience this happens when the target folder is empty or contains only non supported file formats (like .pdf).

https://github.com/jupyterlab/jupyter-ai/blob/56c1f518afd09d0d09a43221f0767aa961e9430f/packages/jupyter-ai/jupyter_ai/chat_handlers/learn.py#L135C5-L149C49

here the split method seems to return an empty list instead of raising an error. This seems to be the reason why add_embedding fails.
I can open a PR next week to fix it if you want. I would also consider adding support for pdfs.

[slashzap]

Thank for this reply @Andrea23Romano. I was just experiencing this issue. I see from the code that only Python and markdown and Latex files are supported. Any thoughts on adding support for data files?

[srdas]

This error does occur if you use an unsupported file type. Currently the supported file types are:
.py .md .R .Rmd .jl .sh .ipynb .js. .ts .jsx .tsx .txt
See:

jupyter-ai/packages/jupyter-ai/jupyter_ai/document_loaders/directory.py

Line 27 in 144cc9b

SUPPORTED_EXTS = {

I am testing a modification for PDFs this week. Will keep you posted.