up to date docs in context.

[scrungus]

the bot needs to have up to date documentation for the technologies we use. A good one to start with would be Ceph - https://docs.ceph.com/en/quincy/.

there should be some way of scraping all this documentation and inserting it into the data source. to start with, we can just do it for releases (e.g. quincy for ceph, yoga/xena/zed for openstack). it can be a manual job - e.g. run script to scrape + insert into data source. for Ceph, it looks like we'd want to scrape everything under the https://docs.ceph.com/en/latest/dev/* directory. there is probably some prebuilt web scraping we can adapt - i.e. don't write a web scraper.

we will need to experiment with chunk size etc to see how the model responds to this.