feat: GPT Crawler RAG pipeline

[zd1990]

Thank you for your work and hope to integrate GPT Crawler into RAG's functionality

[justinh-rahb]

Do you mean RAG from web URLs? If so that's already implemented, if you type a # followed by a URL and then click it, it'll be added as a RAG "document" to your chat.

[sahajamit]

But it only scraps the content of given page URL. Is it possible to scan all the links on the page which takes to other section of the website?

[justinh-rahb]

https://github.com/mendableai/firecrawl

I've tested this, it works.