ContextEnhancedQA-Local-GPT4ALL-FAISS-HuggingFaceEmbeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Context-enhanced question-answering, local version (all-mpnet-base-v2/FAISS/GPT4ALL)\n",
        "This notebook provides an end-to-end implementation of context-enhanced completions logic using open source tools which can be run locally:\n",
        "- For generating vector representations of text - HuggingFace model [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) is used;\n",
        "- Vectors are stored/similarity search is done using [FAISS](https://github.com/facebookresearch/faiss);\n",
        "- Text generation is done using [GPT4ALL](https://github.com/nomic-ai/gpt4all).\n",
        "\n",
        "For contexts the following book was used: JavaScript - Notes for Professionals. GoalKicker.com"
      ],
      "metadata": {
        "id": "2dVASxbN9yxQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##1. Prepare the language-generating model (GPT4ALL, 7B parameters version)"
      ],
      "metadata": {
        "id": "itdagO-j4FbS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Mount your drive to store or copy the needed models for language generation and embeddings\n",
        "# \"Gpt4allfiles\" is the name of a folder on my drive where all the assets used in this notebook were stored\n",
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Rw4vtxvg7EX7",
        "outputId": "d566d7d7-37fd-403a-9e74-8f4b01c6dff4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "gpt4all-lora-quantized.bin didn't work out of the box and I needed to convert it following https://gist.github.com/segestic/4822f3765147418fc084a250598c1fe6 . The resulting file (e.g. gpt4all-lora-q-converted.bin) can be stored on a mounted drive, so that the conversion step can be run only once"
      ],
      "metadata": {
        "id": "EdwrLW639cOJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "1A. If you don't have the converted model (this needs to be run only once)"
      ],
      "metadata": {
        "id": "zTQIqH4V9uKx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Download the original bin file to the colab's drive\n",
        "! wget https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-quantized.bin"
      ],
      "metadata": {
        "id": "Of6lqaPY0lR5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Download llama.cpp 7B model, install the packages needed for conversion\n",
        "%pip install pyllama\n",
        "%pip install transformers\n",
        "%pip install pyllamacpp # pygpt4all -- https://github.com/hwchase17/langchain/pull/3837\n",
        "!python3 -m llama.download --model_size 7B --folder llama/"
      ],
      "metadata": {
        "id": "sleWn8E_0vNQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert the source .bin file\n",
        "!pyllamacpp-convert-gpt4all ./gpt4all-lora-quantized.bin llama/tokenizer.model ./gpt4all-lora-q-converted.bin"
      ],
      "metadata": {
        "id": "ru6ECGjl078F"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Copy the converted model to your drive, so that it can be used later\n",
        "%cp ./gpt4all-lora-q-converted_pygpt4all.bin /content/gdrive/MyDrive/Gpt4allfiles/gpt4all-lora-q-converted.bin"
      ],
      "metadata": {
        "id": "tGx5T9Ia18EY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "1B. If you have a converted model stored in your gdrive"
      ],
      "metadata": {
        "id": "n-2qmIPz9_fB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Set the path to the converted model from your mounted drive (your path may differ)\n",
        "GPT4ALL_MODEL_PATH = \"/content/gdrive/MyDrive/Gpt4allfiles/gpt4all-lora-q-converted.bin\"\n",
        "print(f'''Will be using the model from {GPT4ALL_MODEL_PATH}''')"
      ],
      "metadata": {
        "id": "1c48Ifjl23Gn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3b151e2d-c599-4804-e2c4-c19876f2f370"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Will be using the model from /content/gdrive/MyDrive/Gpt4allfiles/gpt4all-lora-q-converted.bin\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Test the model\n",
        "!pip install langchain\n",
        "!pip install llama-cpp-python\n",
        "\n",
        "from langchain import PromptTemplate, LLMChain\n",
        "from langchain.llms import LlamaCpp\n",
        "\n",
        "llm = LlamaCpp(model_path=GPT4ALL_MODEL_PATH)"
      ],
      "metadata": {
        "id": "DMn4bE6TBAHp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"Question: {question}\n",
        "\n",
        "Answer: Let's think step by step.\"\"\"\n",
        "\n",
        "prompt = PromptTemplate(template=template, input_variables=[\"question\"])"
      ],
      "metadata": {
        "id": "qFBq0c1rBSvk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "llm_chain = LLMChain(prompt=prompt, llm=llm)"
      ],
      "metadata": {
        "id": "H_fVwIq8BcKd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "question = \"What is a banana?\"\n",
        "\n",
        "llm_chain.run(question)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        },
        "id": "lr_yknavBfKk",
        "outputId": "5c7436f4-120b-4b6e-e82a-4c06ade67e04"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Llama.generate: prefix-match hit\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 1min 35s, sys: 301 ms, total: 1min 36s\n",
            "Wall time: 1min 36s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "' A banana is a fruit. Fruits are made up of seeds that can reproduce if they are planted in soil. Bananas are also edible, so we can eat them and get nutrients from them. And finally, bananas are green when they first grow on the tree, but as they ripen their skin turns yellow and they become sweet.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##2. Import the \"contexts\".\n",
        "We'll be using a PDF document with a JavaScript course. In my test the the file js-for-profs.pdf was uploaded from the mounted google drive"
      ],
      "metadata": {
        "id": "xbcqAoLG4eFp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install langchain\n",
        "!pip install pypdf\n",
        "\n",
        "from langchain.document_loaders import PyPDFLoader\n",
        "\n",
        "contexts_pdf_path = '/content/gdrive/MyDrive/Gpt4allfiles/js-for-profs.pdf'\n",
        "loader = PyPDFLoader(contexts_pdf_path)\n",
        "pdf_data = loader.load() # an array of documents [Document(page_content='some string', metadata={'source': '/content/gdrive/MyDrive/Gpt4allfiles/js-for-profs.pdf', 'page': 0})]\n",
        "print(len(pdf_data)) # pages"
      ],
      "metadata": {
        "id": "mxtNYi4D40Y-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "text_splitter = RecursiveCharacterTextSplitter(\n",
        "   chunk_size = 250,\n",
        "   chunk_overlap  = 0,\n",
        "   length_function = len) # my gpt4all model has token limit of 512, need to fit it\n",
        "texts = text_splitter.split_documents(pdf_data)\n",
        "\n",
        "print (f'Now you have {len(texts)} documents')"
      ],
      "metadata": {
        "id": "sKj365C0ZZ-y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##3. Prepare the embeddings model\n",
        "Will be used by langchain.embeddings.HuggingFaceEmbeddings(). We will be using the model 'all-mpnet-base-v2' (768 dimensions, ~420Mb), which can be saved to our mounted drive to avoid downloading it each time."
      ],
      "metadata": {
        "id": "RTqE9AXU6Lkk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "3A. If you don't have the embeddings model saved in your gdrive yet"
      ],
      "metadata": {
        "id": "4hlyffLO-OJu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentence_transformers\n",
        "from sentence_transformers import SentenceTransformer\n",
        "\n",
        "embeddings_model = SentenceTransformer('all-mpnet-base-v2')\n",
        "embeddings_model_path = 'all-mpnet-base-v2'\n",
        "\n",
        "# Save the model to colab's drive\n",
        "embeddings_model.save(embeddings_model_path)\n",
        "print(f'''Downloaded {embeddings_model_path}''')\n",
        "\n",
        "# Copy the directory with the model to your mounted drive\n",
        "%cp -r all-mpnet-base-v2 /content/gdrive/MyDrive/Gpt4allfiles/all-mpnet-base-v2"
      ],
      "metadata": {
        "id": "p6deXpQG6wfx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "3B. If you have the embeddings model in your mounted gdrive already\n",
        "\n",
        "SequenceTransformer will search the weights from the directory SENTENCE_TRANSFORMERS_HOME, if the weights are not found, it will download the weights from huggingface hub (https://github.com/hwchase17/langchain/issues/3079)"
      ],
      "metadata": {
        "id": "bKlV88lyvt10"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentence_transformers\n",
        "\n",
        "import os\n",
        "\n",
        "EMBEDDINGS_MODEL_PATH = \"/content/gdrive/MyDrive/Gpt4allfiles/all-mpnet-base-v2\"\n",
        "os.environ['SENTENCE_TRANSFORMERS_HOME'] = EMBEDDINGS_MODEL_PATH\n",
        "\n",
        "from sentence_transformers import SentenceTransformer\n",
        "\n",
        "embeddings_model = SentenceTransformer()\n",
        "print(EMBEDDINGS_MODEL_PATH)"
      ],
      "metadata": {
        "id": "VAcTosLb72Yb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##4. [Vectors storage](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/faiss.html) preparation, generating and storing vectors"
      ],
      "metadata": {
        "id": "CI-KWvOv8Gn9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install faiss-cpu\n",
        "!pip install langchain\n",
        "from langchain.vectorstores import FAISS"
      ],
      "metadata": {
        "id": "CWfWmTOx8Ptf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "4A. If you don't have the index yet - run this once to populate it and save to the mounted gdrive"
      ],
      "metadata": {
        "id": "cz446W-Av5iv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentence_transformers\n",
        "\n",
        "import os\n",
        "from langchain.embeddings import HuggingFaceEmbeddings\n",
        "\n",
        "EMBEDDINGS_MODEL_PATH = \"/content/gdrive/MyDrive/Gpt4allfiles/all-mpnet-base-v2\"\n",
        "os.environ['SENTENCE_TRANSFORMERS_HOME'] = EMBEDDINGS_MODEL_PATH\n",
        "\n",
        "faiss_index_name = 'js-profs-faiss-index-250'\n",
        "\n",
        "faiss_index = FAISS.from_documents(pdf_data, HuggingFaceEmbeddings())\n",
        "\n",
        "# Save the index locally (not to generate it every time)\n",
        "faiss_index.save_local(faiss_index_name)"
      ],
      "metadata": {
        "id": "MxfYXDGT8sET"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Copy the index to your mounted gdrive\n",
        "%cp -r js-profs-faiss-index-250 /content/gdrive/MyDrive/Gpt4allfiles/js-profs-faiss-index-250"
      ],
      "metadata": {
        "id": "F8JHXq3DvfeQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "4B. If you have the index in your mounted drive already"
      ],
      "metadata": {
        "id": "nAT7gglov8rw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentence_transformers\n",
        "\n",
        "import os\n",
        "from langchain.embeddings import HuggingFaceEmbeddings\n",
        "\n",
        "EMBEDDINGS_MODEL_PATH = \"/content/gdrive/MyDrive/Gpt4allfiles/all-mpnet-base-v2\"\n",
        "os.environ['SENTENCE_TRANSFORMERS_HOME'] = EMBEDDINGS_MODEL_PATH\n",
        "\n",
        "FAISS_INDEX_PATH = \"/content/gdrive/MyDrive/Gpt4allfiles/js-profs-faiss-index-250\"\n",
        "EMBEDDINGS_MODEL_NAME = \"sentence-transformers/all-mpnet-base-v2\"\n",
        "db = FAISS.load_local(FAISS_INDEX_PATH, HuggingFaceEmbeddings(model_name=EMBEDDINGS_MODEL_NAME))"
      ],
      "metadata": {
        "id": "EF8nvLbi8pJU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Test the similarity search\n",
        "embeddings_test_query = \"What are constants?\"\n",
        "docs = db.similarity_search(embeddings_test_query, k=1)\n",
        "# for doc in docs:\n",
        "    # print(str(doc.metadata[\"page\"]) + \":\", doc.page_content)\n",
        "\n",
        "print(f'''>>> Page content:\\n\\n{docs[0].page_content}''')\n",
        "print(f'''>>> Metadata:{docs[0].metadata}''')\n",
        "print(f'''>>> Total docs: {len(docs)}''')\n",
        "\n",
        "docs_and_scores = db.similarity_search_with_score(embeddings_test_query)\n",
        "print(f'''>>> Similarity score of the 1st doc: {docs_and_scores[0][1]}''')"
      ],
      "metadata": {
        "id": "8BiuxVfh9adT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##5. Combining everything together"
      ],
      "metadata": {
        "id": "qyMAyPTEmD-C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Mount your drive to load the needed LLM, embeddings model and index\n",
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9WNdhaGf2t2o",
        "outputId": "dadca475-86dd-423f-b530-b353d80448ec"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "!pip install langchain\n",
        "!pip install faiss-cpu\n",
        "!pip install sentence_transformers\n",
        "!pip install llama-cpp-python\n",
        "\n",
        "import os\n",
        "from langchain import LLMChain, PromptTemplate\n",
        "from langchain.llms import LlamaCpp\n",
        "from langchain.embeddings import HuggingFaceEmbeddings\n",
        "from langchain.vectorstores import FAISS\n",
        "\n",
        "MOUNTED_GDRIVE_FOLDER_PATH = \"/content/gdrive/MyDrive/Gpt4allfiles/\"\n",
        "EMBEDDINGS_MODEL_NAME = \"all-mpnet-base-v2\"\n",
        "EMBEDDINGS_MODEL_PATH = f\"{MOUNTED_GDRIVE_FOLDER_PATH}{EMBEDDINGS_MODEL_NAME}\"\n",
        "os.environ['SENTENCE_TRANSFORMERS_HOME'] = EMBEDDINGS_MODEL_PATH\n",
        "print(f\">> Mounted the embeddings model {EMBEDDINGS_MODEL_PATH}\")\n",
        "GPT4ALL_MODEL_NAME = \"gpt4all-lora-q-converted.bin\"\n",
        "GPT4ALL_MODEL_PATH = f\"{MOUNTED_GDRIVE_FOLDER_PATH}{GPT4ALL_MODEL_NAME}\"\n",
        "print(f\">> Mounted the LLM {GPT4ALL_MODEL_PATH}\")\n",
        "FAISS_INDEX_NAME = \"js-profs-faiss-index-250\"\n",
        "FAISS_INDEX_PATH = f\"{MOUNTED_GDRIVE_FOLDER_PATH}{FAISS_INDEX_NAME}\"\n",
        "print(f\">> Mounted the FAISS index {FAISS_INDEX_PATH}\")\n",
        "\n",
        "llm = LlamaCpp(model_path=GPT4ALL_MODEL_PATH)\n",
        "db = FAISS.load_local(FAISS_INDEX_PATH, HuggingFaceEmbeddings())\n",
        "\n",
        "# Prepare the prompt template\n",
        "template = \"\"\"Respond to the question based on the context.\n",
        "\n",
        "Question:\n",
        "{question}\n",
        "\n",
        "Context:\n",
        "{context}\"\"\"\n",
        "prompt = PromptTemplate(template=template, input_variables=[\"question\", \"context\"])\n",
        "\n",
        "# Prepare the chain\n",
        "llm_chain = LLMChain(prompt=prompt, llm=llm)"
      ],
      "metadata": {
        "id": "KbO0Aqp3nvb0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Retrieve the texts to be used as a context for answering the question\n",
        "query = 'what is the difference between null and undefined?'\n",
        "print(f\">> Query: {query}\")\n",
        "contexts_list = db.similarity_search(query, k=1)\n",
        "print(f'>> Contexts list: {contexts_list}')\n",
        "\n",
        "context = contexts_list[0].page_content\n",
        "print(f'context: {context}, len={len(context)}')"
      ],
      "metadata": {
        "id": "p2AGkOR7fV7q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Run the chain to generate the answer\n",
        "%time\n",
        "llm_chain.run({'question': query, 'context': context[0:300]})"
      ],
      "metadata": {
        "id": "xChgRiT40ic2"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}