feat: metadata extractor based on a LLM

[davidsbatista]

Related Issues

• fixes #5700 and #5702

Proposed Changes:

• Adding a new component which relies on a LLM and a prompt to extract metadata from a Document. Meant to be used within a pipeline during indexing. This was a request from @sjrl - have a look, you can pull this branch and test it

How did you test it?

• added unit tests and integration tests
• manual verification

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 11104881204

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-1.7%) to 88.558%

---

Totals
Change from base Build 11070052105:	-1.7%
Covered Lines:	1486
Relevant Lines:	1678

---

💛 - Coveralls

[julian-risch]

@davidsbatista We should list the new component in the catalog as part of this PR: https://github.com/deepset-ai/haystack-experimental?tab=readme-ov-file#experiments-catalog

[sjrl]

Hey @davidsbatista thanks for you work on this!

One idea that came to mind that this component doesn't yet quite solve is how to handle extracting global information about a Document (at least not elegantly). For example, let's say I want to extract the title from a really long PDF (e.g. more than 200 pages) and store this title in all future chunks I create from this PDF file. With the way this component works I would need to do something like:

File -> PDFConverter -> LLMMetadataExtractor -> DocumentSplitter -> DocumentWriter

but since we have such a long PDF there is a good chance that it will surpass the context window of the LLM and in reality all I need to do is send the first few pages of the file to find the title so this version also feels inefficient latency and cost-wise.

So I was wondering if we could brainstorm a way to handle this type of use case as well? Maybe simply to allow a user to specify a page range as well at init time so we don't need to send the whole Document to the LLM?

[davidsbatista]

@shadeMe we would be happy to merge it, do you still want to have a review?

[sjrl]

Thanks for the work on this!

[shadeMe]

Looks like you've missed a comment above.