Migrate Components to Pipeline v2

[julian-risch]

We are working on Haystack 2.0, with a major refactoring of pipelines and components.

Rationale

We need to prioritize the list of components and separately the list of document stores to migrate to pipelines v2. Most risky components and components essential to most pipelines should be migrated first. Let's also collect feedback on what components are most relevant to Sol (@sjrl) to enable them to give feedback early on based on real use cases. Let's also use telemetry data to see what components are most important to the community.

Use cases

List of the usecases to support, in priority order, with a list of the bare minimum components required for them to work. Note: every pipeline needs the components of all the pipelines above it in priority order in order to work.

Each "component type" links to another small epic where the specific component is broken down into a set of requirements, which might eventually be covered by one or more v2 components.

1. Document Search

Tasks

Preview Give feedback

1. MemoryRetriever components (v2) #5311
   3 of 3
   2.x +1
2. proposal: Embedders design #5390
   ignore-for-release-notes proposal showcase +3
   
3. Implement Embedders components (2.0) #5312
   4 of 4
   2.x +1
   
4. Elasticsearch support (2.x) #5326
   3 of 3
   2.x P1 epic +3
   

Note: Retrievers and Embedder's planning will follow the Docstores

2. Generative QA & Agent Pipelines

Tasks

Preview Give feedback

1. LLM support (2.x) #5330
   12 of 13
   2.x epic +2
   
2. Web Retrieval (2.x) #5614
   4 of 4
   2.x epic +2
   

3. Extractive QA

Tasks

Preview Give feedback

1. Reader Components (v2) #5430
   1 of 1
   2.x +1

4. Minimal Indexing

Tasks

Preview Give feedback

1. DocumentWriter (v2) #5339
   2.x +1
   
2. TextFileToDocument (v2) #5363
   +0
   
3. Preprocessing Components (2.x) #5581
   6 of 6
   2.x epic +2
   

6. General Indexing

Tasks

Preview Give feedback

1. File Classifier by Type (v2) #5362
   1 of 2
   2.x good first issue hacktoberfest +3
   
2. FileToDocument components (2.x) #5367
   7 of 7
   2.x epic +2
   
3. Multimodal transcribers (v2) #5366
   1 of 3
   2.x P3 epic +3

7. Advanced querying

Tasks

Preview Give feedback

1. TopP Sampler (v2) #5626
   2.x epic +2

Agent Pipelines

Agent pipelines will need a bit of exploration to get right. I expect their main enabler to be the LLM component: any other unforeseen component that might be needed here will be prioritized accordingly.

Other

Tasks

Preview Give feedback

1. Port existing preview components to Canals 0.3.x #5341
   +0
   
2. Port existing preview components to Canals 0.4.x #5429
   +0
   
3. chore: port preview components to Canals 0.7.0 #5672
   +0
   
4. proposal: Embedders design #5390
   ignore-for-release-notes proposal showcase +3
   
5. Copy LazyImport to v2 #5579
   2.x +1
6. Marshalling/unmarshalling of Haystack pipelines (v2) #5342
   2.x +1
   
7. DocumentWriter (v2) #5339
   2.x +1
   
8. MemoryRetriever components (v2) #5311
   3 of 3
   2.x +1
9. Reader Components (v2) #5430
   1 of 1
   2.x +1
10. Port Haystack REST API v2 to latest Haystack #5627
    2.x +1
11. Evaluation of Pipeline 2.0 #5628
    1 of 1
    2.x +1
12. Telemetry for Haystack (2.x) #5915
    6 of 6
    2.x epic +2
    
13. Finetuning of LLMs in Pipelines 2.0
14. Finetuning of Retriever & Reader models in Pipelines 2.0

Developer relations efforts

• Have initial demonstration and preview content on Pipelins v2: articles, demos and videos
• Set up the Haystack website to house both v1 and v2 content: tutorials, Integrations, articles

Context

• Notion page for large refinements: https://www.notion.so/Haystack-v2-All-notes-7f1ffa9eeaac4b49b89f3aeacf8df9ba
• Old roadmap item Migration to new Pipeline #4390
• Canals, a component orchestration engine by deepset
  • https://deepset-ai.github.io/canals/
  • https://github.com/deepset-ai/canals
  • https://github.com/deepset-ai/haystack/blob/main/proposals/text/4370-documentstores-and-retrievers.md

[masci]

Unfinished items were added to the roadmap for Q4, closing this one as complete