preprocessing_docs_cleaned.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1a8359b3-5822-4ae4-8f4c-2790033cb519",
   "metadata": {},
   "source": [
    "## Follow these steps to reload the vector index, currently running in DataStax\n",
    "\n",
    "* **Gather Data** \n",
    "    * Download The [docs](https://github.com/fiddler-labs/fiddler/tree/main/docs) folder from the [Fiddler Repo](https://github.com/fiddler-labs/fiddler)\n",
    "        * Clone or Make a pull request on the repo\n",
    "        * Make sure this is grabbed from the branch corresponding to the current release (branch name should be `release/2x.x`)\n",
    "        * Copy the Docs Folder, rename it to the correct release number (ex -v24.14) and put that folder in fiddler-chatbot/documentation_data\n",
    "    * Copy the latest [quickstart notebooks](https://github.com/fiddler-labs/fiddler-examples)\n",
    "        * Make sure to grab the notebook from the latest release\n",
    "        * Place all the .ipynb files from for the lastest release in fiddler-chatbot/documentation_data/notebooks-ipynb\n",
    "* **Run this notebook to generate the vector index feed for this verison**\n",
    "    * Update the `release_num` flag (in set state step) to the release version you are loading the docs for\n",
    "    * Generate the markdown version of .ipynb files and add it to quickstart pages on with the script below\n",
    "    * Crawl the our website for blog and other resources\n",
    "    * You will need to add the caveats from last version the current version\n",
    "    * Chunk the data to 750 token snippets\n",
    "    * Finally we generate the vector_index_feed_24.x.csv that we will upload to our vector database\n",
    "\n",
    "* **Last step:** Reloading the vector index table is done via the \"loader_cassandra_vector_index.ipynb\" notebook. Open that notebook and follow the instructions there"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e31ab6c-89d1-4191-b166-b930e40b95b0",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ca32fdd-957d-41d5-9a1c-8cff797ac19e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip -q install tiktoken\n",
    "!pip -q install openai\n",
    "!pip -q install scipy\n",
    "!pip -q install ast\n",
    "!pip -q install feedparser\n",
    "!pip -q install bs4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a221e7b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import tiktoken\n",
    "import openai \n",
    "import re\n",
    "from scipy import spatial \n",
    "import ast\n",
    "import feedparser\n",
    "from bs4 import BeautifulSoup\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7563f685-be29-4064-a416-78cea3659bcc",
   "metadata": {},
   "source": [
    "### Set State"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c13e1f17-8e72-43f7-8d99-de0e8e3afaf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "EMBEDDING_MODEL = \"text-embedding-ada-002\"  # OpenAI's best embeddings as of Apr 2023\n",
    "GPT_MODEL = \"gpt-3.5-turbo\"\n",
    "release_num = 'v24.19'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "034ee26e-03bc-49eb-8c74-4da5339d2aba",
   "metadata": {},
   "outputs": [],
   "source": [
    "def num_tokens(text: str, model: str = GPT_MODEL) -> int:\n",
    "    \"\"\"Return the number of tokens in a string.\"\"\"\n",
    "    encoding = tiktoken.encoding_for_model(model)\n",
    "    return len(encoding.encode(text))\n",
    "\n",
    "def chunked_string(\n",
    "    string: str,\n",
    "    model: str = EMBEDDING_MODEL,\n",
    "    max_tokens: int = 750,\n",
    ") -> str:\n",
    "    \"\"\"Truncate a string to a maximum number of tokens.\"\"\"\n",
    "    encoding = tiktoken.encoding_for_model(model)\n",
    "    encoded_string = encoding.encode(string)\n",
    "    chunked_string = [encoding.decode(encoded_string[i:i+max_tokens]) for i in range(0, len(encoded_string), max_tokens)]\n",
    "    return chunked_string"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a025d27-522a-4122-9676-9e0c9c313f7d",
   "metadata": {},
   "source": [
    "## Append Notebook markdowns to quick starts\n",
    "\n",
    "- Download [Quickstart guides](https://github.com/fiddler-labs/fiddler-examples) v24.x into *documentation_data/notebooks-ipynb* folder: \n",
    "- Running the NBConvert in the next step will update the md-notebooks folder with markdown version of all the quickstarts from the *notebooks-ipynb* directory "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb7352c6-7c88-4fa7-b8e3-e801e5b2a10e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!jupyter nbconvert --output-dir='documentation_data/md-notebooks' documentation_data/notebooks-ipynb/*.ipynb --to markdown"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4795caef-4377-4fde-9369-b0b35c12f4ff",
   "metadata": {},
   "source": [
    "The cell will take the markdown version of the notebooks and append it to the quickstart pages in the documentation directory "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b55511b-4148-44ba-a654-585aec1dee8b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for root, dirs, files in os.walk(f'/Users/anushrav/Projects/fiddler-chatbot/documentation_data/{release_num}/QuickStart_Notebooks'): \n",
    "    for name in files:\n",
    "        path = os.path.join(root, name)\n",
    "        if path[-3:] == '.md':\n",
    "            with open(path,'r') as f:\n",
    "                file_str = f.read()    \n",
    "            print(file_str)\n",
    "            ipynb_links = re.search(r'\\bFiddler_Quickstart_\\w+', file_str)\n",
    "            #print(ipynb_links)\n",
    "            if ipynb_links:\n",
    "                with open('/Users/anushrav/Projects/fiddler-chatbot/documentation_data/md-notebooks/'+ipynb_links.group()+'.md') as l: \n",
    "                    QS = l.read()\n",
    "                with open(path, 'a') as f:\n",
    "                    f.write(QS)\n",
    "                print(ipynb_links.group())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d43b1b58-85bc-492d-b7fe-4d1c45664471",
   "metadata": {},
   "source": [
    "## Creating list of chunked_docs from downloaded documentation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cadd988d-8d2b-46b9-8a3d-b7f18a920280",
   "metadata": {},
   "outputs": [],
   "source": [
    "#change the path to where your downloaded folder is and choose the version of the docs you want to process\n",
    "chunked_doc = []\n",
    "for root, dirs, files in os.walk(f'documentation_data/{release_num}'):\n",
    "    for name in files:\n",
    "        path = os.path.join(root, name)\n",
    "        if path[-3:] == '.md':\n",
    "            with open(path,'r') as f:\n",
    "                file_str = f.read()\n",
    "                chunked_doc.append(file_str)\n",
    "                \n",
    "len(chunked_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "128aa5b4-c398-4f26-b06b-c019f4456e7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#find and remove hidden pages\n",
    "pattern = r'hidden:\\s*(\\w+)'\n",
    "\n",
    "for doc in chunked_doc:\n",
    "    match = re.search(pattern, doc)\n",
    "    if match and match.group(1) == \"true\":\n",
    "        chunked_doc.remove(doc)\n",
    "        \n",
    "len(chunked_doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f0e8cc-ba4f-4be0-9e43-3c263b1f56bf",
   "metadata": {},
   "source": [
    "### Crawl the blog and resources content and append it to chunked_doc list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2edb2e96-9b61-452d-b40f-4426ff197d74",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def crawl_rss_feed(rss_url):\n",
    "    # Parse the RSS feed\n",
    "    feed = feedparser.parse(rss_url)\n",
    "\n",
    "    print(\"Number of Blogs:\", len(feed.entries))\n",
    "    \n",
    "    # Iterate through the entries in the feed\n",
    "    for entry in feed.entries:\n",
    "\n",
    "        # Get the URL of the blog article\n",
    "        article_url = entry.link\n",
    "\n",
    "        # Fetch the content of the article\n",
    "        response = requests.get(article_url)\n",
    "        html_content = response.content.decode('utf-8', 'ignore')\n",
    "\n",
    "        # Use BeautifulSoup to parse the HTML and extract the body\n",
    "        soup = BeautifulSoup(html_content, 'html.parser')\n",
    "        div_content = soup.find('div', class_='blog-post_content-wrapper')  # You may need to adjust this based on the HTML structure\n",
    "\n",
    "        # Print or manipulate the content of the div\n",
    "        if div_content:\n",
    "            print(\"Title:\", entry.title)\n",
    "            print(\"Link:\", entry.link)\n",
    "            itemtext=''\n",
    "            for item in div_content.select('p'):\n",
    "                itemtext+=item.text + ' '\n",
    "            chunked_doc.append(\"BlogLink:\" + entry.link + \" Content: \" + itemtext)\n",
    "        else:\n",
    "            print(\"Div not found.\")\n",
    "\n",
    "# Replace 'your_rss_feed_url' with the actual RSS feed URL\n",
    "rss_feed_url = 'https://www.fiddler.ai/blog/rss.xml'\n",
    "crawl_rss_feed(rss_feed_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f24c7bca-6280-4003-82d1-7d43dcdd6aa2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def crawl_rss_feed(rss_url):\n",
    "    # Parse the RSS feed\n",
    "    feed = feedparser.parse(rss_url)\n",
    "\n",
    "    print(\"Number of Resources:\", len(feed.entries))\n",
    "    \n",
    "    # Iterate through the entries in the feed\n",
    "    for entry in feed.entries:\n",
    "\n",
    "        # Get the URL of the blog article\n",
    "        article_url = entry.link\n",
    "\n",
    "        # Fetch the content of the article\n",
    "        response = requests.get(article_url)\n",
    "        html_content = response.content.decode('utf-8', 'ignore')\n",
    "\n",
    "        # Use BeautifulSoup to parse the HTML and extract the body\n",
    "        soup = BeautifulSoup(html_content, 'html.parser')\n",
    "\n",
    "        \n",
    "        div_content = soup.find('div', class_='resources-copy')  # You may need to adjust this based on the HTML structure\n",
    "\n",
    "        # Print or manipulate the content of the div\n",
    "        if div_content:\n",
    "            print(\"Title:\", entry.title)\n",
    "            print(\"Link:\", entry.link)\n",
    "            itemtext=''\n",
    "            for item in div_content.select('p'):\n",
    "                itemtext+=item.text + ' '\n",
    "            chunked_doc.append(\"ResourceLink:\" + entry.link + \" Content: \" + itemtext)\n",
    "        else:\n",
    "            print(\"Div not found.\")\n",
    "\n",
    "# Replace 'your_rss_feed_url' with the actual RSS feed URL\n",
    "rss_feed_url = 'http://www.fiddler.ai/resources/rss.xml'\n",
    "crawl_rss_feed(rss_feed_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94bc3813-bc83-40be-8605-de3839b50efb",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(chunked_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26ee60d6-0090-465c-b353-b29797915e67",
   "metadata": {},
   "outputs": [],
   "source": [
    "#we will append page slugs or blog links to every chunk\n",
    "slug_pattern = r'slug:\\s*\"(.*?)\"'\n",
    "blog_pattern = r'BlogLink:https://www\\.fiddler\\.ai/blog/([\\w-]+)'\n",
    "resource_pattern = r'ResourceLink:https://www\\.fiddler\\.ai/resources/([\\w-]+)'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69f7fabd-f713-432c-9ffe-9c348f87cd93",
   "metadata": {},
   "outputs": [],
   "source": [
    "#chunking docs to 750 tokens\n",
    "\n",
    "token_lim_doc = []\n",
    "for doc in chunked_doc:\n",
    "    if num_tokens(doc) > 750:\n",
    "        chunked_list = chunked_string(doc, max_tokens=200)\n",
    "        \n",
    "        # see if a doc slug or a blog link is detected\n",
    "        slug = re.search(slug_pattern, chunked_list[0])\n",
    "        blog = re.search(blog_pattern, chunked_list[0])\n",
    "\n",
    "        if slug:\n",
    "            chunked_doc_slug = slug.group(0)\n",
    "            for i in range(1, len(chunked_list)):\n",
    "                chunked_list[i] = chunked_doc_slug + ' ' + chunked_list[i]\n",
    "        \n",
    "        if blog:\n",
    "            chunked_doc_blog = blog.group(0)\n",
    "            for i in range(1, len(chunked_list)):\n",
    "                chunked_list[i] = chunked_doc_blog + ' ' + chunked_list[i]\n",
    "        \n",
    "        token_lim_doc += chunked_list\n",
    "    else:\n",
    "        token_lim_doc.append(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cf3db71-ccaf-44cc-b8a4-e03e043c55f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Read in old caveats from previous version and output them to a new file\n",
    "old_df = pd.read_csv('documentation_data/v24.16/caveats.csv')\n",
    "caveats_df = old_df[~old_df['text'].str.contains('slug', case=False)]\n",
    "#caveats_df = caveats_df.drop(columns=['embedding'])\n",
    "caveats_df.to_csv(f'documentation_data/{release_num}/caveats.csv', index=False)\n",
    "\n",
    "caveats_df.loc[len(caveats_df.index)] = ['LLM means large language model.  A large language model (LLM) is a type of artificial intelligence (AI) algorithm that uses deep learning techniques and massively large data sets to understand, summarize, generate and predict new content.']\n",
    "caveats_df.loc[len(caveats_df.index)] = ['The term generative AI, or GenAI, also is closely connected with LLMs, which are, in fact, a type of generative AI that has been specifically architected to help generate text-based content.']\n",
    "caveats_df.loc[len(caveats_df.index)] = ['FM, or FMs, means Foundation Models.  Foundation Models are the same as large language models.']\n",
    "\n",
    "caveats_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad02b4d7-ffe5-4610-95f5-9d6cec911baf",
   "metadata": {},
   "outputs": [],
   "source": [
    "#embeddings = generate_embeddings(token_lim_doc)\n",
    "#df = pd.DataFrame({\"text\": chunked_doc, \"embedding\": embeddings})\n",
    "df = pd.DataFrame({\"text\": token_lim_doc})\n",
    "df = pd.concat([df,caveats_df], ignore_index=True)\n",
    "df.to_csv(f'documentation_data/vector_index_feed_{release_num}.csv', index=False)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e59ae7c5-9aa6-4aae-a383-d05c058d5fd8",
   "metadata": {},
   "source": [
    "### You are done! Please check the **vector_index_feed_{release_num}.csv** file and navigate to loader_casandra_vector_index.ipynb to upload these snippets to our vector database."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}