feat: add JinaReaderConnector

[jlonge4]

Related Issues

• part of Add support for Reader API to convert HTMLs into Documents #663

Proposed Changes:

Implement the Jina AI Reader as a haystack component

How did you test it?

Manual Verification

Notes for the reviewer

N/A

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[CLAassistant]

All committers have signed the CLA.

[jlonge4]

Hi @anakin87, @Anitha6g and I were trying to think of the best way to handle returning the output of the tool.
A document definitely makes sense for the read and search mode, but what about ground as it returns: Statement, Factuality, Result, Reason.

The unfortunate thing is that it is all markdown and not structured. Should we parse the content and put those things in doc metadata? Thoughts?

[anakin87]

Let me have a look ⌛

[anakin87]

I'm finally back, after doing a deeper investigation 🕵️ and discussing this with @bilgeyucel.

• We think this component should be named JinaReaderConnector and placed in components/connectors

• While studying the issue with grounding, I found that you can always get a JSON response by all endpoints (reader, search, grounding), if you pass the header 'Accept': 'application/json'. You can play with this and see the results at https://jina.ai/reader/.

• After discovering this JSON response, I would say that we should use this by default because it is more granular and informative (we can add an init parameter called json_response wich defaults to True). This implies that for search we won't have a Document but one for each search result. So I would change the general return type to List[Document] and do something similar to what is discussed below.

  • If json_response==False: return a list of only one Document with only the content, which is a markdown string.
  • If json_response==True, this means we should do some parsing/packing of the results depending on the mode:
    • for reader, create a list of one Document with the returned content as content and all other fields (title, URL, ...) copied into meta
    • for search, create a list of Documents (one for each search result) with the returned content as content and all other fields (title, URL, ...) copied into meta
    • for grounding (not a great solution, but OK), create a list of one Document with the reason field as content and all fields (result, reference, ..., including reason) copied into meta

• By playing with the website, I discovered that there are many other options to run this API which can be set using headers. For this reason, I would add a parameter called headers to the run method, to make these option available to people.

Let me know what you think. I am quite convinced about the plan, even if it requires some work.

[jlonge4]

@anakin87 @bilgeyucel great points, will get right on it and rebase in a couple days. Thanks for the direction as always 😎

[vblagoje]

Very nice @jlonge4 , I wanted to use such a component.

[jlonge4]

@vblagoje thanks very much! Still have to whip up those tests but should be done(ish) today 🤞🏼

[jlonge4]

@vblagoje @anakin87 what stupid mistake am I making with these test failures and import structure?

[anakin87]

@jlonge4 the circular import error should be fixed now.
Integrations mount paths are a bit tricky.

[jlonge4]

@anakin87 how are we looking?

[anakin87]

@jlonge4 in the coming days, I will take a proper look, so as to push this to the finish line.

[anakin87]

I modified the code a bit, added more tests and an example.

This is ready to be merged now.
We may need help with other stuff related to this integration.

Let's sync in #663!