Previously working RAG pipeline stopped working - cannot import name 'can_be_positional' from 'pydantic._internal._utils'

[oboudry-mvp]

I've been successfully running a lanchain RAG pipeline since August 2024, but it stopped working a couple weeks ago, also there was no code changes. I did not notice immediately as it's not used a lot, and therefore cannot relate it to a specific open-webui, langchain, or pydantic update.

What is weird is that it runs from an interactive python shell in the same environment, but does not work when loaded from open-webui.

My repo is public and the full code for the pipeline is available here: https://github.com/marvinpac-it/ask-hr-policies-open-webui/blob/master/ask_hr_policies_pipeline.py

When I check the logs of the pipeline pod, I'm getting the following:

olivier@SINGLE-HOST-DOCKER:~/k8s/open-webui$ k logs open-webui-pipelines-769d89bc96-crwrt | tail
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: pip install --upgrade pip
Error loading module: ask_hr_policies_pipeline
cannot import name 'can_be_positional' from 'pydantic._internal._utils' (/usr/local/lib/python3.11/site-packages/pydantic/_internal/_utils.py)
WARNING:root:No Pipeline class found in ask_hr_policies_pipeline
INFO:     10.233.103.164:37564 - "POST /pipelines/upload HTTP/1.1" 200 OK
INFO:     10.233.103.164:46070 - "GET /pipelines HTTP/1.1" 200 OK
INFO:     10.233.103.164:46074 - "GET /models HTTP/1.1" 200 OK

I tried to exec the code directly inside the pipelines pod, hoping it would help me get better debug messages, but when I do that it runs perfectly well, as can be seen in the following session:

olivier@SINGLE-HOST-DOCKER:~/k8s/open-webui$ k exec -it open-webui-pipelines-769d89bc96-crwrt -- bash
root@open-webui-pipelines-769d89bc96-crwrt:/app# cd pipelines/
root@open-webui-pipelines-769d89bc96-crwrt:/app/pipelines# python
Python 3.11.10 (main, Nov 12 2024, 02:25:24) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from failed.ask_hr_policies_pipeline import Pipeline
>>> p = Pipeline()
>>> import asyncio
>>> asyncio.run(p.on_startup())
on_startup:failed.ask_hr_policies_pipeline
⚠️ It looks like you upgraded from a version below 0.6 and could benefit from vacuuming your database. Run chromadb utils vacuum --help for more information.
>>> result = p.pipe("What is the vacation policy?", "", [], {})
>>> for r in result:
...   r
...
pipe:failed.ask_hr_policies_pipeline
''
'The'
' vacation'
' policy'
' states'
' that'
' employees'
' are'
' entitled'
' to'
' '
'20'
' working'
' days'
' of'
' vacation'
' per'
' calendar'
' year'
--- stripped for sake of space ---

I deployed open-webui and pipelines using the open-webui helm chart from the helm.openwebui.com repo. I'm currently running the latest version.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: open-webui

helmCharts:
- name: open-webui
  repo: https://helm.openwebui.com/
  version: "4.1.0"
  releaseName: open-webui
  namespace: open-webui
  valuesFile: open-webui-values.yaml

resources:
- open-webui-namespace.yaml
- apis-sealed-secret.yaml
- chromadb-pvc.yaml
- langfuse-sealed-secret.yaml

The pipeline is defined in the values file of the helm chart:

pipelines:
  extraEnvVars:
    - name: OPENAI_API_KEY
      valueFrom:
        secretKeyRef:
          name: openai-config
          key: openai-key
    - name: PIPELINES_URLS
      value: "https://github.com/marvinpac-it/ask-hr-policies-open-webui/blob/master/ask_hr_policies_pipeline.py"
    - name: LANGFUSE_URL
      valueFrom:
        secretKeyRef:

Any help troubleshooting this further would be really appreciated.

[Cody-W-Tucker]

What is your Pydantic version? I set mine to 2.8.2 and the error went away.

# pip list | grep pydantic
pydantic                                 2.8.2
pydantic_core                            2.20.1