bug: Crawler downloads pdfs to project root directory

[mohitlal31]

Related Issues

• fixes Crawler downloads pdfs to project root directory #6829

Proposed Changes:

• When a .pdf is passed to Selenium's webdriver.get() method, the .pdf is automatically downloaded to the current working directory. This issue happens here. Whereas for .html files, the crawler extracts the information and manually saves it in the provided output directory.
• By default, we use Selenium's chromium webdriver that provides additional preferences to set the download location for a file. These preferences are defined here.
• When a crawler object is initialized with an output directory, we use that directory as the download location. Else, the .pdf is downloaded to the PWD which is the current behaviour.

How did you test it?

• Added a unit test
• Tested it using the following script

from haystack.nodes import Crawler

crawler = Crawler(output_dir="output_files")
docs = crawler.crawl(urls=["https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"])

Notes for the reviewer

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[anakin87]

I tried your code and works great!

Can you please add a release note, as described in the contributors guidelines?

[anakin87]

Thanks for your contribution! 🙏