community: Add @mozilla/readability document transformer

[CNSeniorious000]

Description

langchain-js already has a useful document transformer that use @mozilla/readability to extract main content of a web page heuristically. [docs] [source]

This PR introduces a new ReadabilityTransformer class to the langchain_community/document_transformers, which class leverages the python-readability library to do the same thing.

Dependencies:

python-readability — a Standalone Python wrapper for @mozilla/readability

Mention that no nodejs environment is needed. In regular CPython distributions, python-readability requires PythonMonkey to interpret JavaScript, and in Pyodide, it uses the native JavaScript environment. So this package is available even if the user deploys langchain apps on Cloudflare Workers.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Dec 10, 2024 5:54pm

[ccurme]

Thanks for this @CNSeniorious000.

It's unclear to me that the demand for this is high enough to justify the additional maintenance burden.

Would you be interested in publishing an OSS integration package (e.g., langchain-readability or similar)? We've written a walkthrough on this process here:

https://python.langchain.com/docs/contributing/how_to/integrations/

We are encouraging contributors of LangChain integrations to go this route. This way we don't have to be in the loop for reviews, you're able to properly integration test the package, and you have control over versioning.

Docs would continue to be maintained in the langchain repo.

Let me know what you think!