proposal: Embedders design

proposals/text/5390-embedders.md

@@ -0,0 +1,249 @@
+- Title: Embedders
+- Decision driver: @anakin87
+- Start Date: 2023-07-19
+- Proposal PR: https://github.com/deepset-ai/haystack/pull/5390
+
+# Summary
+
+As decided in the previous proposals ([Embedding Retriever](3558-embedding_retriever.md) and [DocumentStores and Retrievers](4370-documentstores-and-retrievers.md)), in Haystack V2 we want to introduce a new component: the Embedder.
+
+**Separation of concerns**
+- DocumentStores: store the Documents, their metadata and representations (vectors); they offer a CRUD API.
+- Retrievers: retrieve Documents from the DocumentStores; they are specific and aware of the used Store (e.g., MemoryRetriever for the MemoryDocumentStore). They will be commonly used in query pipelines (not in indexing pipelines).
+- **Embedders**: encode a list of data points (strings, images, etc.) into a list of vectors (i.e., the embeddings) using a model. They are used both in indexing pipelines (to encode the Documents) and query pipelines (to encode the query).
+
+*In the current implementation, the Embedder is part of Retriever, which is unintuitive and comes with several disadvantages (explained in the previous proposals).*
+
+**This proposal aims to define the Embedder design.**
+
+# Basic example
+
+*This code snippet is merely an example and may not be completely up-to-date.*
+
+
+```python
+from haystack import Pipeline
+from haystack.components import (
+    TxtConverter,
+    PreProcessor,
+    DocumentWriter,
+    OpenAITextEmbedder,
+    OpenAIDocumentEmbedder,
+    MemoryRetriever,
+    Reader,
+)
+from haystack.document_stores import MemoryDocumentStore
+docstore = MemoryDocumentStore()
+
+indexing_pipe = Pipeline()
+indexing_pipe.add_store("document_store", docstore)
+indexing_pipe.add_node("txt_converter", TxtConverter())
+indexing_pipe.add_node("preprocessor", PreProcessor())
+indexing_pipe.add_node("embedder", OpenAIDocumentEmbedder(model_name="text-embedding-ada-002"))
+indexing_pipe.add_node("writer", DocumentWriter(store="document_store"))
+indexing_pipe.connect("txt_converter", "preprocessor")
+indexing_pipe.connect("preprocessor", "embedder")
+indexing_pipe.connect("embedder", "writer")
+
+indexing_pipe.run(...)
+
+query_pipe = Pipeline()
+query_pipe.add_store("document_store", docstore)
+query_pipe.add_node("embedder", OpenAITextEmbedder(model_name="text-embedding-ada-002"))
+query_pipe.add_node("retriever", MemoryRetriever(store="document_store", retrieval_method="embedding"))
+query_pipe.add_node("reader", Reader(model_name="deepset/model-name"))
+query_pipe.connect("embedder", "retriever")
+query_pipe.connect("retriever", "reader")
+
+results = query_pipe.run(...)
+```
+
+- The `OpenAITextEmbedder` uses OpenAI models to convert a list of strings into a list of vectors. It is used in the query pipeline to embed the query.
+- The `OpenAIDocumentEmbedder` uses OpenAI models to enrich a list of Documents with the corresponding vectors (stored in the `embedding` field). It is used in the indexing pipeline to embed the Documents.
+- The Retriever is no longer needed in the indexing pipeline.
+
+# Motivation
+
+The motivations behind this change were already provided in the previous proposals ([Embedding Retriever](3558-embedding_retriever.md) and [DocumentStores and Retrievers](4370-document_stores_and_retrievers.md)). Here is a summary:
+- Retrievers should't be responsible for embedding Documents.
+- Currently, Retrievers have many parameters just to support and configure different underlying Encoders(≈Embedders).
+- Adding support for new embedding providers or strategies is difficult. It requires changing the Retriever code.
+
+# Detailed design
+
+## Handle queries and Documents
+This is the most critical aspect of the design.
+
+- When embedding queries, the Embedder component receives a list of strings in input that are transformed into a list of vectors returned as output.
+- When embedding documents, the Embedder component receives a list of `Document` objects in input; for each item in the list, the corresponding vectors are computed and stored in the `embedding` field of the item itself. The list is then returned as the component output.
+- When working with documents, there's the possibility to compute embeddings also for document's metadata. In this case, the Embedder will be responsible for performing any text-manipulation work needed in preparation of the actual embedding process.
+
+**Below, I will focus on the public API. The internal implementation is discussed in [Implementation details](#implementation-details).**
+
+```python
+@component
+class HFTextEmbedder:
+    ...
+
+    @component.output_types(result=List[np.ndarray])
+    def run(self, strings: List[str]):
+        ...
+        return {"result": list_of_computed_embeddings}
+
+
+@component
+class HFDocumentEmbedder:
+    ...
+
+    @component.output_types(result=List[Document])
+    def run(self, documents: List[Document]):
+        ...
+        return {"result": list_of_documents_with_embeddings}
+```
+
+## Different providers/strategies
+
+- We can define different embedder components depending on the models or services providing the actual embeddings: `OpenAIEmbedder`, `CohereEmbedder`, `HuggingFaceEmbedder`, `SentenceTransformersEmbedder`, etc.
+- Additionally, we could define different classes depending on the embedding strategy if necessary.
+While this is not a prominent use case, there are scenarios where [new strategies](https://github.com/deepset-ai/haystack/issues/5242) are introduced, requiring different libraries (`InstructorEmbedder`) or involving a different string preparation (`E5Embedder`). Supporting these scenarios with minimal effort would be nice.
+
+## Different models in the same embedding/retrieval task
+
+As you can observe from the [current implementation](https://github.com/deepset-ai/haystack/blob/main/haystack/nodes/retriever/dense.py), some embedding/retrieval tasks require the usage of different models.
+
+This is not the most popular approach today, compared to what we call Embedding Retrieval (based on a single model). But it still has some relevant applications.
+
+Some examples:
+- In Dense Passage Retrieval, you need a model to encode queries and another model to encode Documents
+- in the TableTextRetriever, we use 3 different models: one for queries, one for textual passages and one for tables
+- in Multimodal Retrieval, we can specify different models to encode queries and Documents
+
+Since the Embedder will not be included in the Retriever, it makes sense to have different Embedders, each one using a single model.
+
+```python
+dpr_query_embedder = SentenceTransformersTextEmbedder(model_name="facebook/dpr-question_encoder-single-nq-base")
+dpr_doc_embedder = SentenceTransformersDocumentEmbedder(model_name="facebook/dpr-ctx_encoder-single-nq-base")
+```
+
+## Implementation details
+
+*You can skip this section if you are primarily interested in user experience.*
+
+There have been much discussion on how to effectively implement this proposal.
+The most important aspects to consider:
+- we want different Embedders for queries and Documents as they require a different treatment
+- if the same model is internally used for different Embedders, we want to reuse the same instance in order to save memory
+
+On top of the embedder components we already discussed, we introduce one additional abstraction:
+an `EmbeddingService`, which is NOT a component, responsible for performing the actual embedding computation, implemented as a singleton class in order to reuse instances. It will live in a different package and will be hidden from the public API.
+```python
+@singleton  # implementation is out of scope
+class HFEmbeddingService:
+    """
+    NOT A COMPONENT!
+    """
+    def __init__(self, model_name: str, ... init params ...):
+        """
+        init takes the minimum parameters needed at init time, not
+        the params needed at inference, so they're easier to reuse.
+        """
+        self.model = ...
+
+    def embed(self, data: str, ... inference params ... ) -> np.ndarray:
+        # compute embedding
+        return embedding
+
+
+class OpenAIEmbeddingService:
+    ... same as above ...
+```
+
+Implemented as singletons, when instantiating an EmbeddingService class, if another identical one exists, the existing one will be returned without allocating additional resources for a new one. This makes model reusability transparent, saving lots of memory without any user intervention.
+
+This is how an EmbeddingService would be used by a text embedder component:
+**Part of the public API**.
+```python
+@component
+class HFTextEmbedder:
+
+    def __init__(self, model_name: str, ... init params ...):
+        self.model_name = model_name
+        self.model_params = ... params ...
+
+    def warm_up(self):
+        self.embedding_service = HFEmbeddingService(self.model_name, **self.model_params)
+
+    @component.output_types(result=List[np.ndarray])
+    def run(self, strings: List[str]):
+        return {"result": self.embedding_service.embed(data)}
+```
+
+Another example, using an embedder component expecting Documents:
+**Part of the public API**.
+
+```python
+@component
+class HFDocumentEmbedder:
+
+    def __init__(self, model_name: str, ... init params ...):
+        self.model_name = model_name
+        self.model_params = ... params ...
+
+    def warm_up(self):
+        self.embedding_service = HFEmbeddingService(self.model_name, **self.model_params)
+
+    @component.output_types(result=List[Document])
+    def run(self, documents: List[Document]):
+        text_strings = [document.content for document in data]
+        embeddings = self.embedding_service.embed(text_strings)
+        documents_with_embeddings = [Document.from_dict(**doc.to_dict, "embedding": emb) for doc, emb in zip(documents, embeddings)]
+        return {"result": documents_with_embeddings}
+```
+
+# Drawbacks
+
+## Migration
+The drawbacks of separating Retrievers and Embedders were already discussed in [this proposal](https://github.com/deepset-ai/haystack/blob/main/proposals/text/4370-documentstores-and-retrievers.md) and mainly consist of **migration effort**.
+
+For example, if a user has indexed documents in the store and wants to update the embeddings using a different model instead, with the current Haystack implementation the user would run `document_store.update_embeddings(retriever)`.
+
+With the new Embedder design, I can imagine something similar (based on the MemoryDocumentStore v2 implementation):
+```python
+# get all the documents
+docs = memory_document_store.filter_documents()
+
+# compute the embedding with the new model
+new_embedder = HFDocumentEmbedder(model_name="new-model")
+docs_with_embeddings = new_embedder.run(documents=docs)
+
+# overwrite the documents
+memory_document_store.write_documents(documents=docs_with_embeddings, policy=DuplicatePolicy.OVERWRITE)
+```
+## Other aspects
+Regarding the design proposed in this document, there are some potential drawbacks to consider:
+  - Proliferation of classes (though they will be small and easy to maintain).
+  - Users need to know which models are appropriate for which task (e.g. embedding queries rather than embedding documents, see [Different models in the same embedding/retrieval task](#different-models-in-the-same-embeddingretrieval-task)). On the other hand, this approach is more explicit and will help making users aware of problems and tradeoffs related to the topic.
+
+# Alternatives
+
+Several alternatives to this design were considered. The main challenge was handling the differences between queries and Documents.
+Some ideas:
+- Have a single Embedder component for text (HFTextEmbedder instead of HFEmbeddingService, HFTextEmbedder and HFDocumentEmbedder) and adapt Documents before and after that, using other Components. --> Many components.
+- Make Embedders only work on Documents and represent the query as a Document. --> Unintuitive and require changes in the Retriever.
+- Create another primitive like Data (content + embedding) and use it for both queries and Documents. --> More conversion components like DataToDocument.
+- Have the DocumentEmbedder take a TextEmbedder as an input parameter. --> Fewer classes but serialization issues.
+
+# Adoption strategy
+
+This change will constitute a part of Haystack v2.
+
+# How we teach this
+
+Documentation and tutorials will be of fundamental importance.
+
+# Unresolved questions
+
+- Migration and refactoring of existing Encoders hidden in Retrievers.
+I prepared a table. Should it be shared here?
+- The migration and refactoring of TableTextRetriever require input and ownership from people involved in TableQA.
+- How to approach MultiModal Embedding? How many classes? Take into consideration that a query could also be an Image or a Table.