MetadataBuilder

[sjrl]

See the proposal: #5540 and see feature request for Haystack v1

---

LLMs clients output strings, but many components expect other object types, and LLMs may produce output in a parsable format that can be directly converted into objects. Output parsers transform these strings into objects of the user’s choosing.

MetadataBuilder. It takes the string replies and inserts them as metadata into the Documents that were originally passed to the LLM. I'm open to renaming this one, since the goal would be to output Documents with inserted metadata.

For example, a PromptNode could be used to summarize a longer doc and the user would like to have the result inserted as metadata for that Document. There it would allow us to easily add category tags, sentiment, summaries (...) to docs that can be utilized later at query time (e.g. to filter down the search space efficiently or utilize the metadata for online retrieval/generation steps)

[anakin87]

More information on the expected use cases and component I/O can be found here.

In general, it is probably best to focus on developing this component once looping and input lists are handleable by the Pipelines.
(Otherwise, we would be going to build a component that is effectively unusable in the Pipelines.)

[julian-risch]

@sjrl We are considering this issue for our next sprint. Is there any new info that will be relevant for the implementation of this component?

[anakin87]

This is probably relevant: https://www.notion.so/deepsetai/Advanced-Use-Case-Automatic-Metadata-Enrichment-8fdfc56e82434459963beaa7a9dc5069

[sjrl]

Hey @julian-risch thanks for reaching out! No new info on my end. I think the work @davidsbatista did that @anakin87 linked is exactly the type of use case we are thinking about. In general metadata enrichment of files to help with retrieval through filters, embed meta fields, etc. Also possibly for downstream applications (e.g. they want to show a summary along side a retrieved file). I'd be particularly interested in a set up that would allow me to automatically extract things like title, authors, publication date, etc. from PDF files and then save that as metadata with the file.

[davidsbatista]

see #5700 - it's related/duplicated

[julian-risch]

@davidsbatista Could you please check again if we can merge the two issues #5700 and #5702 or whether they should remain separate?

[davidsbatista]

After discussing it with Sebastian, these two issues should be merged.

[anakin87]

This should have been done in deepset-ai/haystack-experimental#92.
If not, feel free to reopen this issue.