Migration to new Pipeline

[ZanSara]

Follow up of #2807

---

Goal

Now that we have an initial implementation of the new Pipeline, we should start migrating most nodes, stores, REST API and all other abstractions to support it.

Steps

Not in priority order unless specified.

• Migrate nodes. In priority order:

  • feat: port PromptNode to new Pipelines #4462
  • WebRetriever Implement WebRetriever for Pipeline v2 #4840
  • Retriever: one for each document store
    • Implement BM25 support forMemoryRetriever (v2) #4842 and Implement embedding retrieval for MemoryRetriever (2.0) #4843
  • A Reader
  • File converters (txt, pdf, other formats)
  • PreProcessor
  • FileTypeClassifier
  • All other nodes in order of "popular demand" (like Whisper, language classifiers, image related nodes, etc), like:
    • WhisperTranscriber: feat: WhisperTranscriber for v2 #4723

• Migrate Agent and tools. Agents will become a node and tools are probably going to be converted into nodes too. To be refined.

• Migrate Stores

  • Define a new structure for them: proposal: DocumentStores and Retrievers #4370
  • Add MemoryDocumentStore for new Pipelines #4446
    • Implement filters for MemoryDocumentStore : Improve MemoryDocumentStore filtering to ease retrieval #4838
      • feat: initial implementation of MemoryDocumentStore for new Pipelines #4447
      • bug: fix filtering in MemoryDocumentStore (v2) #4768
      • feat: allow filtering documents on all fields (v2) #4773
    • Implement Retrieval for MemoryDocumentStore (v2) #4839
  • Support one or two "modern" vector stores (Qdrant/Weaviate/Pinecone)
  • Support Elasticsearch 8
  • Migrate all other stores in order of "popular demand" (especially FAISS or SQL)

• Implement REST API

  • Implement REST API for new pipelines #4427
  • Make Docker images for them

• Introduce "standard pipelines", for example "QAPipeline", "IndexingPipeline", "RAGPipeline", "AgentPipeline". These pipelines are blocked by the implementation of the relevant nodes. To be refined

• Documentation and tutorials for everything

Tasks

Preview Give feedback

1. feat: port PromptNode to new Pipelines #4462
   2.x P2 +2
2. Implement WebRetriever for Pipeline v2 #4840
   2.x P2 +2
3. Implement BM25 support forMemoryRetriever (v2) #4842
   2.x P2 +2
   
4. Implement embedding retrieval for MemoryRetriever (2.0) #4843
   2.x P2 +2
   
5. feat: WhisperTranscriber for v2 #4723
   2.x topic:tests type:documentation +3
6. proposal: DocumentStores and Retrievers #4370
   proposal +1
7. Add MemoryDocumentStore for new Pipelines #4446
   +0
   
8. Improve MemoryDocumentStore filtering to ease retrieval #4838
   2.x +1
   
9. feat: initial implementation of MemoryDocumentStore for new Pipelines #4447
   +0
10. bug: fix filtering in MemoryDocumentStore (v2) #4768
    +0
    
11. feat: allow filtering documents on all fields (v2) #4773
    2.x topic:tests type:documentation +3
    
12. Implement Retrieval for MemoryDocumentStore (v2) #4839
    2.x P1 +2
13. Implement REST API for new pipelines #4427
    +0
    

[jingqiao]

Do you have a rough timeline for the release of v2 (including the new pipeline mentioned here)?

As to the vector store, I'd like to vote for Qdrant. It is also among the ones supported by OpenAI's latest retrieval plugins.

[julian-risch]

Let's close this epic and continue tracking of the pipeline migration in the follow-up epic for Q3 #5265 We'll continue the open tasks there too.

[julian-risch]

@jingqiao I can't communicate a timeline yet but we're focusing on this topic as our main priority for Q3. We will support Qdrant among many other document stores in v2. 🙂