feat: adding an hierarchical document builder and a auto-merge retriever

[davidsbatista]

Related Issues

• implements #7863

Proposed Changes:

Adds a HierarchicalDocumentBuilder: it's used to split a Document into multiple Document objects of different block sizes building a hierarchical tree structure where each smaller block is a child of a previous larger block.

Adds a AutoMergingRetriever leverages the hierarchical tree structure of documents, where the leaf nodes are indexed in a document store. During retrieval, if the number of matched leaf documents below the same parent is higher than a defined threshold, the retriever will return the parent document instead of the individual leaf documents.

Picture to help understand what's being implemented

How did you test it?

• unit tests
• integration tests
• manual verification

Notes for the reviewer

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 10560416383

Details

• 278 of 287 (96.86%) changed or added relevant lines in 6 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.3%) to 93.376%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
haystack_experimental/components/retrievers/auto_merging_retriever.py	40	49	81.63%

Totals
Change from base Build 10524780287:	0.3%
Covered Lines:	3031
Relevant Lines:	3246

---

💛 - Coveralls

[anakin87]

I took a first pass.

An end-to-end raw example script would still be helpful.

[anakin87]

I left a few comments.

Let's also add these components to the Experiments catalog in README.

[anakin87]

Let's also remember to add pydoc configs here.

[davidsbatista]

still work-in-progress, I'm testing this new Retriever against every possible DocumentStore in the integrations repo and collecting the issues.

[davidsbatista]

Here is the current status regarding all the doc stores, I'm still investigating Weaviate

	AutoMergeRetriever	NOTE
Astra	error	only if "id" is used in the filter instead of "_id"
Chroma	no support
ElasticSearch	supports
OpenSearch	supports
PGVector	supports
PineCone	error	cannot filter by ID; https://docs.pinecone.io/guides/data/query-data#querying-by-record-id; https://community.pinecone.io/t/does-pinecone-support-filtering-by-vector-id/3039/2
Qdrant	supports
Weaviate	error	DocumentStoreError: Failed to query documents in Weaviate. Error: no such prop with name 'id' found in class 'Default' in the schema. Check your schema files for which properties in this class are available

[shadeMe]

One small change then good to merge for me.