Auto-merging Retrieval

[davidsbatista]

Auto-merging retrieval aims to combine (or merge) information from multiple sources or segments of text to create a more comprehensive and contextually relevant response to a query.

This approach is particularly useful when no single document or segment fully answers the query but rather the answer lies in combining information from multiple sources.

It allows smaller chunks to be merged into bigger parent chunks. It does this via the following steps:

1. Define a hierarchy of smaller chunks linked to parent chunks.
2. If the set of smaller chunks linking to a parent chunk exceeds some threshold retrieve the parent chunk.

The method will finally retrieve the parent chunk for better context.

• splitting documents into nodes of different block sizes, including parent and child nodes
• implement a retriever based on hierarchical chunking merging

[julian-risch]

closed by deepset-ai/haystack-experimental#56