Eduquer_Hackathon_review2_prototype.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Nanashi-bot/eduquery/blob/main/Eduquer_Hackathon_review2_prototype.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Chat with eBook\n",
        "### Installation"
      ],
      "metadata": {
        "id": "MRp3eL6WxBVk"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "x0HMFxS3wxCh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cb069fd2-eb2c-436a-d7ed-98b846cac9b2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: langchain in /usr/local/lib/python3.10/dist-packages (0.1.13)\n",
            "Requirement already satisfied: sentence-transformers in /usr/local/lib/python3.10/dist-packages (2.5.1)\n",
            "Requirement already satisfied: chromadb in /usr/local/lib/python3.10/dist-packages (0.4.24)\n",
            "Requirement already satisfied: pypdf in /usr/local/lib/python3.10/dist-packages (4.0.1)\n",
            "Requirement already satisfied: unstructured in /usr/local/lib/python3.10/dist-packages (0.12.6)\n",
            "Requirement already satisfied: pdf2image in /usr/local/lib/python3.10/dist-packages (1.17.0)\n",
            "Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (6.0.1)\n",
            "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.0.28)\n",
            "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.9.3)\n",
            "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)\n",
            "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.6.4)\n",
            "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.33)\n",
            "Requirement already satisfied: langchain-community<0.1,>=0.0.29 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.0.29)\n",
            "Requirement already satisfied: langchain-core<0.2.0,>=0.1.33 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.1.33)\n",
            "Requirement already satisfied: langchain-text-splitters<0.1,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.0.1)\n",
            "Requirement already satisfied: langsmith<0.2.0,>=0.1.17 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.1.31)\n",
            "Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.26.4)\n",
            "Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.6.4)\n",
            "Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.31.0)\n",
            "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (8.2.3)\n",
            "Requirement already satisfied: transformers<5.0.0,>=4.32.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (4.37.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (4.66.2)\n",
            "Requirement already satisfied: torch>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (2.2.0)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.2.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.10.1)\n",
            "Requirement already satisfied: huggingface-hub>=0.15.1 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (0.20.3)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (10.2.0)\n",
            "Requirement already satisfied: build>=1.0.3 in /usr/local/lib/python3.10/dist-packages (from chromadb) (1.1.1)\n",
            "Requirement already satisfied: chroma-hnswlib==0.7.3 in /usr/local/lib/python3.10/dist-packages (from chromadb) (0.7.3)\n",
            "Requirement already satisfied: fastapi>=0.95.2 in /usr/local/lib/python3.10/dist-packages (from chromadb) (0.110.0)\n",
            "Requirement already satisfied: uvicorn[standard]>=0.18.3 in /usr/local/lib/python3.10/dist-packages (from chromadb) (0.29.0)\n",
            "Requirement already satisfied: posthog>=2.4.0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (3.5.0)\n",
            "Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (4.9.0)\n",
            "Requirement already satisfied: pulsar-client>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (3.4.0)\n",
            "Requirement already satisfied: onnxruntime>=1.14.1 in /usr/local/lib/python3.10/dist-packages (from chromadb) (1.15.1)\n",
            "Requirement already satisfied: opentelemetry-api>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (1.23.0)\n",
            "Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (1.23.0)\n",
            "Requirement already satisfied: opentelemetry-instrumentation-fastapi>=0.41b0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (0.44b0)\n",
            "Requirement already satisfied: opentelemetry-sdk>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (1.23.0)\n",
            "Requirement already satisfied: tokenizers>=0.13.2 in /usr/local/lib/python3.10/dist-packages (from chromadb) (0.15.2)\n",
            "Requirement already satisfied: pypika>=0.48.9 in /usr/local/lib/python3.10/dist-packages (from chromadb) (0.48.9)\n",
            "Requirement already satisfied: overrides>=7.3.1 in /usr/local/lib/python3.10/dist-packages (from chromadb) (7.7.0)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.10/dist-packages (from chromadb) (6.1.1)\n",
            "Requirement already satisfied: grpcio>=1.58.0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (1.62.1)\n",
            "Requirement already satisfied: bcrypt>=4.0.1 in /usr/local/lib/python3.10/dist-packages (from chromadb) (4.1.2)\n",
            "Requirement already satisfied: typer>=0.9.0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (0.9.0)\n",
            "Requirement already satisfied: kubernetes>=28.1.0 in /usr/local/lib/python3.10/dist-packages (from chromadb) (29.0.0)\n",
            "Requirement already satisfied: mmh3>=4.0.1 in /usr/local/lib/python3.10/dist-packages (from chromadb) (4.1.0)\n",
            "Requirement already satisfied: orjson>=3.9.12 in /usr/local/lib/python3.10/dist-packages (from chromadb) (3.9.15)\n",
            "Requirement already satisfied: backoff==2.2.1 in /usr/local/lib/python3.10/dist-packages (from unstructured) (2.2.1)\n",
            "Requirement already satisfied: beautifulsoup4==4.12.3 in /usr/local/lib/python3.10/dist-packages (from unstructured) (4.12.3)\n",
            "Requirement already satisfied: certifi==2024.2.2 in /usr/local/lib/python3.10/dist-packages (from unstructured) (2024.2.2)\n",
            "Requirement already satisfied: chardet==5.2.0 in /usr/local/lib/python3.10/dist-packages (from unstructured) (5.2.0)\n",
            "Requirement already satisfied: charset-normalizer==3.3.2 in /usr/local/lib/python3.10/dist-packages (from unstructured) (3.3.2)\n",
            "Requirement already satisfied: click==8.1.7 in /usr/local/lib/python3.10/dist-packages (from unstructured) (8.1.7)\n",
            "Requirement already satisfied: dataclasses-json-speakeasy==0.5.11 in /usr/local/lib/python3.10/dist-packages (from unstructured) (0.5.11)\n",
            "Requirement already satisfied: emoji==2.10.1 in /usr/local/lib/python3.10/dist-packages (from unstructured) (2.10.1)\n",
            "Requirement already satisfied: filetype==1.2.0 in /usr/local/lib/python3.10/dist-packages (from unstructured) (1.2.0)\n",
            "Requirement already satisfied: idna==3.6 in /usr/local/lib/python3.10/dist-packages (from unstructured) (3.6)\n",
            "Requirement already satisfied: joblib==1.3.2 in /usr/local/lib/python3.10/dist-packages (from unstructured) (1.3.2)\n",
            "Requirement already satisfied: jsonpath-python==1.0.6 in /usr/local/lib/python3.10/dist-packages (from unstructured) (1.0.6)\n",
            "Requirement already satisfied: langdetect==1.0.9 in /usr/local/lib/python3.10/dist-packages (from unstructured) (1.0.9)\n",
            "Requirement already satisfied: lxml==5.1.0 in /usr/local/lib/python3.10/dist-packages (from unstructured) (5.1.0)\n",
            "Requirement already satisfied: marshmallow==3.20.2 in /usr/local/lib/python3.10/dist-packages (from unstructured) (3.20.2)\n",
            "Requirement already satisfied: mypy-extensions==1.0.0 in /usr/local/lib/python3.10/dist-packages (from unstructured) (1.0.0)\n",
            "Requirement already satisfied: nltk==3.8.1 in /usr/local/lib/python3.10/dist-packages (from unstructured) (3.8.1)\n",
            "Requirement already satisfied: packaging==23.2 in /usr/local/lib/python3.10/dist-packages (from unstructured) (23.2)\n",
            "Requirement already satisfied: python-dateutil==2.8.2 in /usr/local/lib/python3.10/dist-packages (from unstructured) (2.8.2)\n",
            "Requirement already satisfied: python-iso639==2024.2.7 in /usr/local/lib/python3.10/dist-packages (from unstructured) (2024.2.7)\n",
            "Requirement already satisfied: python-magic==0.4.27 in /usr/local/lib/python3.10/dist-packages (from unstructured) (0.4.27)\n",
            "Requirement already satisfied: rapidfuzz==3.6.1 in /usr/local/lib/python3.10/dist-packages (from unstructured) (3.6.1)\n",
            "Requirement already satisfied: regex==2023.12.25 in /usr/local/lib/python3.10/dist-packages (from unstructured) (2023.12.25)\n",
            "Requirement already satisfied: six==1.16.0 in /usr/local/lib/python3.10/dist-packages (from unstructured) (1.16.0)\n",
            "Requirement already satisfied: soupsieve==2.5 in /usr/local/lib/python3.10/dist-packages (from unstructured) (2.5)\n",
            "Requirement already satisfied: tabulate==0.9.0 in /usr/local/lib/python3.10/dist-packages (from unstructured) (0.9.0)\n",
            "Requirement already satisfied: typing-inspect==0.9.0 in /usr/local/lib/python3.10/dist-packages (from unstructured) (0.9.0)\n",
            "Requirement already satisfied: unstructured-client==0.18.0 in /usr/local/lib/python3.10/dist-packages (from unstructured) (0.18.0)\n",
            "Requirement already satisfied: urllib3==1.26.18 in /usr/local/lib/python3.10/dist-packages (from unstructured) (1.26.18)\n",
            "Requirement already satisfied: wrapt==1.16.0 in /usr/local/lib/python3.10/dist-packages (from unstructured) (1.16.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.4)\n",
            "Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.10/dist-packages (from build>=1.0.3->chromadb) (1.0.0)\n",
            "Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build>=1.0.3->chromadb) (2.0.1)\n",
            "Requirement already satisfied: starlette<0.37.0,>=0.36.3 in /usr/local/lib/python3.10/dist-packages (from fastapi>=0.95.2->chromadb) (0.36.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (3.13.1)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (2024.2.0)\n",
            "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain) (2.4)\n",
            "Requirement already satisfied: google-auth>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from kubernetes>=28.1.0->chromadb) (2.27.0)\n",
            "Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.10/dist-packages (from kubernetes>=28.1.0->chromadb) (1.7.0)\n",
            "Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.10/dist-packages (from kubernetes>=28.1.0->chromadb) (1.4.0)\n",
            "Requirement already satisfied: oauthlib>=3.2.2 in /usr/local/lib/python3.10/dist-packages (from kubernetes>=28.1.0->chromadb) (3.2.2)\n",
            "Requirement already satisfied: anyio<5,>=3 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2.0,>=0.1.33->langchain) (3.7.1)\n",
            "Requirement already satisfied: coloredlogs in /usr/local/lib/python3.10/dist-packages (from onnxruntime>=1.14.1->chromadb) (15.0.1)\n",
            "Requirement already satisfied: flatbuffers in /usr/local/lib/python3.10/dist-packages (from onnxruntime>=1.14.1->chromadb) (23.5.26)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from onnxruntime>=1.14.1->chromadb) (4.23.4)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from onnxruntime>=1.14.1->chromadb) (1.12)\n",
            "Requirement already satisfied: deprecated>=1.2.6 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-api>=1.2.0->chromadb) (1.2.14)\n",
            "Requirement already satisfied: importlib-metadata<7.0,>=6.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-api>=1.2.0->chromadb) (6.11.0)\n",
            "Requirement already satisfied: googleapis-common-protos~=1.52 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.63.0)\n",
            "Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.23.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.23.0)\n",
            "Requirement already satisfied: opentelemetry-proto==1.23.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.23.0)\n",
            "Requirement already satisfied: opentelemetry-instrumentation-asgi==0.44b0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.44b0)\n",
            "Requirement already satisfied: opentelemetry-instrumentation==0.44b0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.44b0)\n",
            "Requirement already satisfied: opentelemetry-semantic-conventions==0.44b0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.44b0)\n",
            "Requirement already satisfied: opentelemetry-util-http==0.44b0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.44b0)\n",
            "Requirement already satisfied: setuptools>=16.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation==0.44b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (67.7.2)\n",
            "Requirement already satisfied: asgiref~=3.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation-asgi==0.44b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (3.8.0)\n",
            "Requirement already satisfied: monotonic>=1.5 in /usr/local/lib/python3.10/dist-packages (from posthog>=2.4.0->chromadb) (1.6)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (0.6.0)\n",
            "Requirement already satisfied: pydantic-core==2.16.3 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (2.16.3)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.0.3)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (3.2.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (3.1.3)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (2.19.3)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (2.2.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.11.0->sentence-transformers) (12.4.99)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.32.0->sentence-transformers) (0.3.2)\n",
            "Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.14.0)\n",
            "Requirement already satisfied: httptools>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.6.1)\n",
            "Requirement already satisfied: python-dotenv>=0.13 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.18.3->chromadb) (1.0.1)\n",
            "Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.19.0)\n",
            "Requirement already satisfied: watchfiles>=0.13 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.21.0)\n",
            "Requirement already satisfied: websockets>=10.4 in /usr/local/lib/python3.10/dist-packages (from uvicorn[standard]>=0.18.3->chromadb) (12.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers) (3.3.0)\n",
            "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain-core<0.2.0,>=0.1.33->langchain) (1.3.1)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain-core<0.2.0,>=0.1.33->langchain) (1.2.0)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (5.3.3)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (0.3.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (4.9)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata<7.0,>=6.0->opentelemetry-api>=1.2.0->chromadb) (3.17.0)\n",
            "Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.10/dist-packages (from coloredlogs->onnxruntime>=1.14.1->chromadb) (10.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.11.0->sentence-transformers) (2.1.5)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->onnxruntime>=1.14.1->chromadb) (1.3.0)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (0.5.1)\n"
          ]
        }
      ],
      "source": [
        "!pip install langchain sentence-transformers chromadb pypdf unstructured pdf2image"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install unstructured['pdf']"
      ],
      "metadata": {
        "id": "iLexJk-ipaa2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "7abea5f5-0cd4-43e7-dc0f-62da1af05e9a"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: unstructured[pdf] in /usr/local/lib/python3.10/dist-packages (0.12.6)\n",
            "Requirement already satisfied: backoff==2.2.1 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2.2.1)\n",
            "Requirement already satisfied: beautifulsoup4==4.12.3 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (4.12.3)\n",
            "Requirement already satisfied: certifi==2024.2.2 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2024.2.2)\n",
            "Requirement already satisfied: chardet==5.2.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (5.2.0)\n",
            "Requirement already satisfied: charset-normalizer==3.3.2 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (3.3.2)\n",
            "Requirement already satisfied: click==8.1.7 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (8.1.7)\n",
            "Requirement already satisfied: dataclasses-json==0.6.4 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (0.6.4)\n",
            "Requirement already satisfied: dataclasses-json-speakeasy==0.5.11 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (0.5.11)\n",
            "Requirement already satisfied: emoji==2.10.1 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2.10.1)\n",
            "Requirement already satisfied: filetype==1.2.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.2.0)\n",
            "Requirement already satisfied: idna==3.6 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (3.6)\n",
            "Requirement already satisfied: joblib==1.3.2 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.3.2)\n",
            "Requirement already satisfied: jsonpath-python==1.0.6 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.0.6)\n",
            "Requirement already satisfied: langdetect==1.0.9 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.0.9)\n",
            "Requirement already satisfied: lxml==5.1.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (5.1.0)\n",
            "Requirement already satisfied: marshmallow==3.20.2 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (3.20.2)\n",
            "Requirement already satisfied: mypy-extensions==1.0.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.0.0)\n",
            "Requirement already satisfied: nltk==3.8.1 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (3.8.1)\n",
            "Requirement already satisfied: numpy==1.26.4 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.26.4)\n",
            "Requirement already satisfied: packaging==23.2 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (23.2)\n",
            "Requirement already satisfied: python-dateutil==2.8.2 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2.8.2)\n",
            "Requirement already satisfied: python-iso639==2024.2.7 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2024.2.7)\n",
            "Requirement already satisfied: python-magic==0.4.27 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (0.4.27)\n",
            "Requirement already satisfied: rapidfuzz==3.6.1 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (3.6.1)\n",
            "Requirement already satisfied: regex==2023.12.25 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2023.12.25)\n",
            "Requirement already satisfied: requests==2.31.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2.31.0)\n",
            "Requirement already satisfied: six==1.16.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.16.0)\n",
            "Requirement already satisfied: soupsieve==2.5 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2.5)\n",
            "Requirement already satisfied: tabulate==0.9.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (0.9.0)\n",
            "Requirement already satisfied: tqdm==4.66.2 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (4.66.2)\n",
            "Requirement already satisfied: typing-extensions==4.9.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (4.9.0)\n",
            "Requirement already satisfied: typing-inspect==0.9.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (0.9.0)\n",
            "Requirement already satisfied: unstructured-client==0.18.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (0.18.0)\n",
            "Requirement already satisfied: urllib3==1.26.18 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.26.18)\n",
            "Requirement already satisfied: wrapt==1.16.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.16.0)\n",
            "Collecting antlr4-python3-runtime==4.9.3 (from unstructured[pdf])\n",
            "  Downloading antlr4-python3-runtime-4.9.3.tar.gz (117 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m117.0/117.0 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: cffi==1.16.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.16.0)\n",
            "Requirement already satisfied: coloredlogs==15.0.1 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (15.0.1)\n",
            "Requirement already satisfied: contourpy==1.2.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.2.0)\n",
            "Collecting cryptography==42.0.2 (from unstructured[pdf])\n",
            "  Downloading cryptography-42.0.2-cp39-abi3-manylinux_2_28_x86_64.whl (4.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.7/4.7 MB\u001b[0m \u001b[31m19.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: cycler==0.12.1 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (0.12.1)\n",
            "Requirement already satisfied: deprecated==1.2.14 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.2.14)\n",
            "Collecting effdet==0.4.1 (from unstructured[pdf])\n",
            "  Downloading effdet-0.4.1-py3-none-any.whl (112 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m112.5/112.5 kB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock==3.13.1 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (3.13.1)\n",
            "Collecting flatbuffers==23.5.26 (from unstructured[pdf])\n",
            "  Downloading flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB)\n",
            "Requirement already satisfied: fonttools==4.49.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (4.49.0)\n",
            "Collecting fsspec==2024.2.0 (from unstructured[pdf])\n",
            "  Downloading fsspec-2024.2.0-py3-none-any.whl (170 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m170.9/170.9 kB\u001b[0m \u001b[31m20.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: huggingface-hub==0.20.3 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (0.20.3)\n",
            "Requirement already satisfied: humanfriendly==10.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (10.0)\n",
            "Collecting importlib-resources==6.1.1 (from unstructured[pdf])\n",
            "  Downloading importlib_resources-6.1.1-py3-none-any.whl (33 kB)\n",
            "Collecting iopath==0.1.10 (from unstructured[pdf])\n",
            "  Downloading iopath-0.1.10.tar.gz (42 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.2/42.2 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: jinja2==3.1.3 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (3.1.3)\n",
            "Requirement already satisfied: kiwisolver==1.4.5 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.4.5)\n",
            "Collecting layoutparser[layoutmodels,tesseract]==0.3.4 (from unstructured[pdf])\n",
            "  Downloading layoutparser-0.3.4-py3-none-any.whl (19.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.2/19.2 MB\u001b[0m \u001b[31m52.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: markupsafe==2.1.5 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2.1.5)\n",
            "Collecting matplotlib==3.7.2 (from unstructured[pdf])\n",
            "  Downloading matplotlib-3.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.6/11.6 MB\u001b[0m \u001b[31m29.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: mpmath==1.3.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.3.0)\n",
            "Requirement already satisfied: networkx==3.2.1 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (3.2.1)\n",
            "Collecting omegaconf==2.3.0 (from unstructured[pdf])\n",
            "  Downloading omegaconf-2.3.0-py3-none-any.whl (79 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m79.5/79.5 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting onnx==1.15.0 (from unstructured[pdf])\n",
            "  Downloading onnx-1.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m67.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting onnxruntime==1.15.1 (from unstructured[pdf])\n",
            "  Downloading onnxruntime-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.9/5.9 MB\u001b[0m \u001b[31m79.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: opencv-python==4.8.0.76 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (4.8.0.76)\n",
            "Collecting pandas==2.2.0 (from unstructured[pdf])\n",
            "  Downloading pandas-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.0/13.0 MB\u001b[0m \u001b[31m112.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pdf2image==1.17.0 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.17.0)\n",
            "Collecting pdfminer-six==20221105 (from unstructured[pdf])\n",
            "  Downloading pdfminer.six-20221105-py3-none-any.whl (5.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.6/5.6 MB\u001b[0m \u001b[31m104.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pdfplumber==0.10.4 (from unstructured[pdf])\n",
            "  Downloading pdfplumber-0.10.4-py3-none-any.whl (54 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.7/54.7 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pikepdf==8.11.0 (from unstructured[pdf])\n",
            "  Downloading pikepdf-8.11.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pillow==10.2.0 (from unstructured[pdf])\n",
            "  Downloading pillow-10.2.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.5/4.5 MB\u001b[0m \u001b[31m109.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pillow-heif==0.15.0 (from unstructured[pdf])\n",
            "  Downloading pillow_heif-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.5/7.5 MB\u001b[0m \u001b[31m101.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting portalocker==2.8.2 (from unstructured[pdf])\n",
            "  Downloading portalocker-2.8.2-py3-none-any.whl (17 kB)\n",
            "Collecting protobuf==4.23.4 (from unstructured[pdf])\n",
            "  Downloading protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl (304 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m304.5/304.5 kB\u001b[0m \u001b[31m37.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pycocotools==2.0.7 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2.0.7)\n",
            "Requirement already satisfied: pycparser==2.21 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (2.21)\n",
            "Collecting pyparsing==3.0.9 (from unstructured[pdf])\n",
            "  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.3/98.3 kB\u001b[0m \u001b[31m14.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pypdf==4.0.1 (from unstructured[pdf])\n",
            "  Downloading pypdf-4.0.1-py3-none-any.whl (283 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m284.0/284.0 kB\u001b[0m \u001b[31m36.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pypdfium2==4.27.0 (from unstructured[pdf])\n",
            "  Downloading pypdfium2-4.27.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m104.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytesseract==0.3.10 (from unstructured[pdf])\n",
            "  Downloading pytesseract-0.3.10-py3-none-any.whl (14 kB)\n",
            "Collecting python-multipart==0.0.9 (from unstructured[pdf])\n",
            "  Downloading python_multipart-0.0.9-py3-none-any.whl (22 kB)\n",
            "Collecting pytz==2024.1 (from unstructured[pdf])\n",
            "  Downloading pytz-2024.1-py2.py3-none-any.whl (505 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m505.5/505.5 kB\u001b[0m \u001b[31m46.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml==6.0.1 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (6.0.1)\n",
            "Collecting safetensors==0.3.2 (from unstructured[pdf])\n",
            "  Downloading safetensors-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m87.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting scipy==1.10.1 (from unstructured[pdf])\n",
            "  Downloading scipy-1.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m34.4/34.4 MB\u001b[0m \u001b[31m21.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sympy==1.12 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (1.12)\n",
            "Collecting timm==0.9.12 (from unstructured[pdf])\n",
            "  Downloading timm-0.9.12-py3-none-any.whl (2.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m108.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tokenizers==0.15.2 in /usr/local/lib/python3.10/dist-packages (from unstructured[pdf]) (0.15.2)\n",
            "Collecting torch==2.2.0 (from unstructured[pdf])\n",
            "  Downloading torch-2.2.0-cp310-cp310-manylinux1_x86_64.whl (755.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m755.5/755.5 MB\u001b[0m \u001b[31m2.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting torchvision==0.17.0 (from unstructured[pdf])\n",
            "  Downloading torchvision-0.17.0-cp310-cp310-manylinux1_x86_64.whl (6.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.9/6.9 MB\u001b[0m \u001b[31m78.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting transformers==4.37.1 (from unstructured[pdf])\n",
            "  Downloading transformers-4.37.1-py3-none-any.whl (8.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.4/8.4 MB\u001b[0m \u001b[31m117.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tzdata==2024.1 (from unstructured[pdf])\n",
            "  Downloading tzdata-2024.1-py2.py3-none-any.whl (345 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m345.4/345.4 kB\u001b[0m \u001b[31m39.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting unstructured-inference==0.7.23 (from unstructured[pdf])\n",
            "  Downloading unstructured_inference-0.7.23-py3-none-any.whl (60 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.1/60.1 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting unstructured-pytesseract==0.3.12 (from unstructured[pdf])\n",
            "  Downloading unstructured.pytesseract-0.3.12-py3-none-any.whl (14 kB)\n",
            "Collecting zipp==3.17.0 (from unstructured[pdf])\n",
            "  Downloading zipp-3.17.0-py3-none-any.whl (7.4 kB)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (2.19.3)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (12.1.105)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.2.0->unstructured[pdf]) (2.2.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.2.0->unstructured[pdf]) (12.4.99)\n",
            "Building wheels for collected packages: antlr4-python3-runtime, iopath\n",
            "  Building wheel for antlr4-python3-runtime (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.9.3-py3-none-any.whl size=144554 sha256=2a65eb7dbd0f6eae129fca51eda1ccd839160fa10870a57cfaedaa28a3009f29\n",
            "  Stored in directory: /root/.cache/pip/wheels/12/93/dd/1f6a127edc45659556564c5730f6d4e300888f4bca2d4c5a88\n",
            "  Building wheel for iopath (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for iopath: filename=iopath-0.1.10-py3-none-any.whl size=31532 sha256=05c0f8f67691611974b611e462e00593063d626cefc8579390c196019acd5495\n",
            "  Stored in directory: /root/.cache/pip/wheels/9a/a3/b6/ac0fcd1b4ed5cfeb3db92e6a0e476cfd48ed0df92b91080c1d\n",
            "Successfully built antlr4-python3-runtime iopath\n",
            "Installing collected packages: safetensors, pytz, flatbuffers, antlr4-python3-runtime, zipp, tzdata, scipy, python-multipart, pypdfium2, pypdf, pyparsing, protobuf, portalocker, pillow, omegaconf, importlib-resources, fsspec, unstructured-pytesseract, pytesseract, pillow-heif, pikepdf, pandas, onnxruntime, onnx, matplotlib, iopath, cryptography, torch, pdfminer-six, transformers, torchvision, pdfplumber, timm, layoutparser, effdet, unstructured-inference\n",
            "  Attempting uninstall: safetensors\n",
            "    Found existing installation: safetensors 0.4.2\n",
            "    Uninstalling safetensors-0.4.2:\n",
            "      Successfully uninstalled safetensors-0.4.2\n",
            "  Attempting uninstall: pytz\n",
            "    Found existing installation: pytz 2023.4\n",
            "    Uninstalling pytz-2023.4:\n",
            "      Successfully uninstalled pytz-2023.4\n",
            "  Attempting uninstall: flatbuffers\n",
            "    Found existing installation: flatbuffers 24.3.7\n",
            "    Uninstalling flatbuffers-24.3.7:\n",
            "      Successfully uninstalled flatbuffers-24.3.7\n",
            "  Attempting uninstall: zipp\n",
            "    Found existing installation: zipp 3.18.1\n",
            "    Uninstalling zipp-3.18.1:\n",
            "      Successfully uninstalled zipp-3.18.1\n",
            "  Attempting uninstall: scipy\n",
            "    Found existing installation: scipy 1.11.4\n",
            "    Uninstalling scipy-1.11.4:\n",
            "      Successfully uninstalled scipy-1.11.4\n",
            "  Attempting uninstall: pypdf\n",
            "    Found existing installation: pypdf 4.1.0\n",
            "    Uninstalling pypdf-4.1.0:\n",
            "      Successfully uninstalled pypdf-4.1.0\n",
            "  Attempting uninstall: pyparsing\n",
            "    Found existing installation: pyparsing 3.1.2\n",
            "    Uninstalling pyparsing-3.1.2:\n",
            "      Successfully uninstalled pyparsing-3.1.2\n",
            "  Attempting uninstall: protobuf\n",
            "    Found existing installation: protobuf 3.20.3\n",
            "    Uninstalling protobuf-3.20.3:\n",
            "      Successfully uninstalled protobuf-3.20.3\n",
            "  Attempting uninstall: pillow\n",
            "    Found existing installation: Pillow 9.4.0\n",
            "    Uninstalling Pillow-9.4.0:\n",
            "      Successfully uninstalled Pillow-9.4.0\n",
            "  Attempting uninstall: importlib-resources\n",
            "    Found existing installation: importlib_resources 6.3.0\n",
            "    Uninstalling importlib_resources-6.3.0:\n",
            "      Successfully uninstalled importlib_resources-6.3.0\n",
            "  Attempting uninstall: fsspec\n",
            "    Found existing installation: fsspec 2023.6.0\n",
            "    Uninstalling fsspec-2023.6.0:\n",
            "      Successfully uninstalled fsspec-2023.6.0\n",
            "  Attempting uninstall: pandas\n",
            "    Found existing installation: pandas 1.5.3\n",
            "    Uninstalling pandas-1.5.3:\n",
            "      Successfully uninstalled pandas-1.5.3\n",
            "  Attempting uninstall: onnxruntime\n",
            "    Found existing installation: onnxruntime 1.17.1\n",
            "    Uninstalling onnxruntime-1.17.1:\n",
            "      Successfully uninstalled onnxruntime-1.17.1\n",
            "  Attempting uninstall: matplotlib\n",
            "    Found existing installation: matplotlib 3.7.1\n",
            "    Uninstalling matplotlib-3.7.1:\n",
            "      Successfully uninstalled matplotlib-3.7.1\n",
            "  Attempting uninstall: cryptography\n",
            "    Found existing installation: cryptography 42.0.5\n",
            "    Uninstalling cryptography-42.0.5:\n",
            "      Successfully uninstalled cryptography-42.0.5\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 2.2.1+cu121\n",
            "    Uninstalling torch-2.2.1+cu121:\n",
            "      Successfully uninstalled torch-2.2.1+cu121\n",
            "  Attempting uninstall: transformers\n",
            "    Found existing installation: transformers 4.38.2\n",
            "    Uninstalling transformers-4.38.2:\n",
            "      Successfully uninstalled transformers-4.38.2\n",
            "  Attempting uninstall: torchvision\n",
            "    Found existing installation: torchvision 0.17.1+cu121\n",
            "    Uninstalling torchvision-0.17.1+cu121:\n",
            "      Successfully uninstalled torchvision-0.17.1+cu121\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "bigframes 0.25.0 requires pandas<2.1.4,>=1.5.0, but you have pandas 2.2.0 which is incompatible.\n",
            "gcsfs 2023.6.0 requires fsspec==2023.6.0, but you have fsspec 2024.2.0 which is incompatible.\n",
            "google-colab 1.0.0 requires pandas==1.5.3, but you have pandas 2.2.0 which is incompatible.\n",
            "imageio 2.31.6 requires pillow<10.1.0,>=8.3.2, but you have pillow 10.2.0 which is incompatible.\n",
            "tensorflow 2.15.0 requires wrapt<1.15,>=1.11.0, but you have wrapt 1.16.0 which is incompatible.\n",
            "tensorflow-metadata 1.14.0 requires protobuf<4.21,>=3.20.3, but you have protobuf 4.23.4 which is incompatible.\n",
            "torchaudio 2.2.1+cu121 requires torch==2.2.1, but you have torch 2.2.0 which is incompatible.\n",
            "torchtext 0.17.1 requires torch==2.2.1, but you have torch 2.2.0 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed antlr4-python3-runtime-4.9.3 cryptography-42.0.2 effdet-0.4.1 flatbuffers-23.5.26 fsspec-2024.2.0 importlib-resources-6.1.1 iopath-0.1.10 layoutparser-0.3.4 matplotlib-3.7.2 omegaconf-2.3.0 onnx-1.15.0 onnxruntime-1.15.1 pandas-2.2.0 pdfminer-six-20221105 pdfplumber-0.10.4 pikepdf-8.11.0 pillow-10.2.0 pillow-heif-0.15.0 portalocker-2.8.2 protobuf-4.23.4 pyparsing-3.0.9 pypdf-4.0.1 pypdfium2-4.27.0 pytesseract-0.3.10 python-multipart-0.0.9 pytz-2024.1 safetensors-0.3.2 scipy-1.10.1 timm-0.9.12 torch-2.2.0 torchvision-0.17.0 transformers-4.37.1 tzdata-2024.1 unstructured-inference-0.7.23 unstructured-pytesseract-0.3.12 zipp-3.17.0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "PIL",
                  "matplotlib",
                  "mpl_toolkits",
                  "pydevd_plugins"
                ]
              },
              "id": "a50bc3a835984f239ae41a9dba986897"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain.document_loaders import UnstructuredPDFLoader\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "from langchain.embeddings import HuggingFaceInferenceAPIEmbeddings\n",
        "from langchain.vectorstores import Chroma\n",
        "\n",
        "from langchain.llms import HuggingFaceHub\n",
        "from langchain.chains import RetrievalQA\n",
        "from langchain.storage import InMemoryStore\n",
        "from langchain.retrievers import ParentDocumentRetriever"
      ],
      "metadata": {
        "id": "jUhQa3K5xWLB"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from getpass import getpass\n",
        "\n",
        "HF_token = getpass()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wqxrqhml0Jzb",
        "outputId": "ad3d1ed1-feaa-413f-ab91-16e11a1beaa6"
      },
      "execution_count": 4,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ['HUGGINGFACEHUB_API_TOKEN'] = HF_token"
      ],
      "metadata": {
        "id": "_5MCIBmp01v3"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "file_path = \"./MicroEco.txt\""
      ],
      "metadata": {
        "id": "7yMI-P5m0-rN"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "\n",
        "from langchain_community.document_loaders import UnstructuredFileLoader"
      ],
      "metadata": {
        "id": "BaBW9E5Oih_l"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data = UnstructuredFileLoader(file_path)"
      ],
      "metadata": {
        "id": "zd9FgoHn1GD1"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "content = data.load()"
      ],
      "metadata": {
        "id": "U9bWC0Wn1KGp"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(content)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UGjSLYB91OD0",
        "outputId": "977bc32d-48a8-47b6-c730-baa14736a6aa"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Document(page_content=\"income inequality lorez,gini and quintiles\\n\\nin this section of the course what we're\\n\\ntalking about is income inequality in\\n\\nother words why is it that some people\\n\\nmake more than others and as a part of\\n\\nthis topic we're going to be talking\\n\\nabout poverty what it means how its\\n\\nmeasured and discrimination as one of\\n\\nthe reasons why some people make more\\n\\nthan others the first thing we have to\\n\\ntalk about before we talk about anything\\n\\nabout income inequality is that when we\\n\\ndiscuss things like poverty and income\\n\\ninequality we're not usually talking\\n\\nabout individuals and how their incomes\\n\\nare different we're usually talking\\n\\nabout households so let's first talk for\\n\\na minute about what a household is so\\n\\nwhen we talk about an income\\n\\ndistribution how some people make more\\n\\nthan others and we're going to look at\\n\\none of these in a few different ways in\\n\\na minute we're talking about household\\n\\nincome and equality so we have to\\n\\nunderstand what a household is before we\\n\\nget going but a brief definition is that\\n\\na household includes all the persons who\\n\\noccupy a housing unit\\n\\nwhat's a housing unit well a housing\\n\\nunit is a house an apartment a mobile\\n\\nhome a group of rooms or a single room\\n\\nthat is occupied or if vacant is\\n\\nintended for occupancy as separate\\n\\nliving quarters so it could be a house\\n\\nan apartment a mobile home or it could\\n\\neven be a dorm room so when we start\\n\\nthinking about some households making\\n\\nmore than others we have to include all\\n\\nthe different versions of households and\\n\\nwe have to think about all those in our\\n\\nminds that when we measure incomes for\\n\\nsome of these households it's going to\\n\\nbe a dorm room a single person living in\\n\\na single dorm room perhaps all the way\\n\\nup to there might be a mansion with a\\n\\nhundred people living there several\\n\\ngenerations in a large house keep that\\n\\nin the back of your mind when you think\\n\\nabout income inequality we always have\\n\\nto have in the back of our minds what it\\n\\nis we're measuring so we're measuring\\n\\nhousehold incomes we add up all the\\n\\nincomes of all the people living in\\n\\nwhatever it is that we're calling a\\n\\nhousehold and those things are very\\n\\ndiverse\\n\\nso what are the causes of income\\n\\ninequality is just the fact that\\n\\nhouseholds are unequal we're talking\\n\\nabout a lot of different things here\\n\\nwhen we say the word household we're not\\n\\ntalking about some unified nineteen\\n\\nforties ideal of a mummy and a daddy and\\n\\ntwo and a half kids living in a\\n\\nhousehold right okay now let's look at\\n\\nan income distribution as a histogram so\\n\\nthis is a histogram showing for each\\n\\nincome grouping in $5,000 increments\\n\\nhere what percent of the households in\\n\\nthe United States in the year 2014\\n\\nearned amounts in different income\\n\\ngroups here so we see that about almost\\n\\nfour percent of households earned less\\n\\nthan five thousand dollars about three\\n\\nand a half percent between 5,000 and\\n\\n$10,000 etc and then when we get all the\\n\\nway up to the high end here we see that\\n\\nthere's about five and a half percent if\\n\\nwe look over here at the y-axis about\\n\\nfive and a half percent of households\\n\\ntotal income was $200,000 and more the\\n\\nmedian income tells us that 50 percent\\n\\nof households earned more than this\\n\\namount and fifty percent of households\\n\\nearn less than this amount and this is\\n\\nat about fifty three thousand six\\n\\nhundred fifty seven dollars here the\\n\\nmean the average if you add up all the\\n\\nincome of all the households and divide\\n\\nit by the number of households we get\\n\\nabout seventy five thousand seven\\n\\nhundred thirty eight dollars so the\\n\\naverage income about seventy-six\\n\\nthousand dollars the median household\\n\\nincome about 53 or 54 thousand dollars\\n\\nit's just telling us that half the\\n\\nhouseholds earn more than this half the\\n\\nhouseholds total earn less than this so\\n\\nif we were going to have a society where\\n\\nthere was no income inequality what\\n\\nwould have to happen is that all\\n\\nhouseholds would have to be in the same\\n\\ngroup right we might have every single\\n\\nhousehold have a total income of $76,000\\n\\nwe wouldn't have this spread out\\n\\ndistribute\\n\\nhere is that what we want would that be\\n\\nsomehow fair if every single household\\n\\nearned seventy six thousand dollars well\\n\\nin my opinion the answer is no because\\n\\nsome of these households would be one\\n\\nperson living alone in a dorm room and\\n\\nsome of these households might be again\\n\\nthree generations of grandparents and\\n\\nparents and children you might have 25\\n\\npeople living in a house I don't think\\n\\nit's fair for every single household\\n\\ntype to be earning the same amount of\\n\\nmoney because it doesn't make sense\\n\\nwe're not holding the number of people\\n\\nin a household constant right so this is\\n\\nanother thing to keep in mind when we\\n\\ntalk about income inequality what's the\\n\\ngoal is it to make incomes perfectly\\n\\nequal and if so what does that even mean\\n\\nso let's talk about some ways to measure\\n\\nincome inequality so there are a lot of\\n\\ndifferent ways we can measure income\\n\\ninequality dozens that I know of but\\n\\nlet's go through the standard easy kind\\n\\nof way and this is the most common way\\n\\nyou see this kind of thing done\\n\\nfirst thing is we divide households into\\n\\nincome quintiles what this means is we\\n\\ntake all the households add up the total\\n\\nincome and we line them in order from\\n\\nthe lowest total income to the highest\\n\\ntotal income households and we divide\\n\\nthem in two quintiles and what this\\n\\nmeans is we're going to divide them into\\n\\nfive groups since quint means five so\\n\\nwe're going to take the bottom 20% of\\n\\nhouseholds the lowest income 20% and\\n\\nthen the next 20% so if you look at this\\n\\ntable here the we might call them the\\n\\npoorest 20% lower middle 20% the middle\\n\\n20% the upper-middle 20% and the highest\\n\\n20% of incomes ok second thing we do\\n\\nthis is named after a guy named Lorenz\\n\\nhe came up with the idea of making a\\n\\ngraph called a Lorenz curve and what we\\n\\ndo is we add up the total percent of the\\n\\nincome earned by all the households from\\n\\nthe\\n\\npoorest to the richest and we're going\\n\\nto do that a couple of times here so\\n\\nthat we get an idea of how this is done\\n\\nand what it means so let's go ahead and\\n\\nget started and let's make a few Lorenz\\n\\ncurves here so let's suppose we were\\n\\nworking with some data here from several\\n\\ncountries first country a country a is a\\n\\nperfectly equal country at least in\\n\\nterms of households so let's assume\\n\\nthere's some government and they go\\n\\nthrough and they ensure that every\\n\\nsingle household in their country earns\\n\\nexactly the same amount of money so\\n\\nthere's going to be no income inequality\\n\\nso what's going to happen when we line\\n\\nup all these households from the lowest\\n\\nincome households to the highest income\\n\\nhouseholds well it doesn't matter what\\n\\norder we put them in because they all\\n\\nlearn the same amount so if we take the\\n\\npoorest 20% or the lowest income 20% and\\n\\nwe add up what percent of the total\\n\\nincome in the nation do they earn what\\n\\nwe're going to get is that they earn 20%\\n\\nof the total income right the poorest\\n\\n20% of households get 20% of the income\\n\\na fifth of the households get a fifth of\\n\\nthe total income and then the next group\\n\\nsince they're exactly the same they're\\n\\nalso going to get 20% so we just take\\n\\nthe total income of the country we\\n\\ndivide the households into five groups\\n\\nand each of the five groups gets a fifth\\n\\nof the income 20% of the income now in\\n\\norder to make this thing we call a\\n\\nLorenz curve we need to make a\\n\\ncumulative distribution of the income\\n\\nnow what this means is we're going to\\n\\nadd up so cumulative means add up as you\\n\\ngo along from the beginning to the end\\n\\nthink about a cumulative GPA from the\\n\\nbeginning of your career as a student to\\n\\nthe end the GPA kind of accumulates or\\n\\ngets added up the number of credit hours\\n\\nyou get adds up so you might look at\\n\\nyour transcript and see a cumulative\\n\\nearned hours number on there so in order\\n\\nto make a cumulative distribution you're\\n\\ntrying to answer the question what's the\\n\\ntotal percent of income that is earned\\n\\nby this\\n\\nor all the groups earlier in the list so\\n\\nthe first one here is simply going to be\\n\\nthe first number twenty percent what's\\n\\nthe total amount of income in this first\\n\\ngroup or the group's earlier well it's\\n\\njust this first twenty percent here then\\n\\nwhen we add in the next group we take\\n\\nthis twenty percent and we add in the\\n\\ntwenty percent to the next group and we\\n\\nget forty percent cumulative adding up\\n\\nas we go along and that forty percent\\n\\nmeans that in this second group or\\n\\nearlier they are sharing a total of\\n\\nforty percent of the income and then we\\n\\nadd in the twenty percent to the next\\n\\ngroup this first example it's a bit\\n\\nsilly but it makes an important point\\n\\nyou'll see when we get there we're just\\n\\nadding up as we go along and the last\\n\\nnumber and a cumulative distribution\\n\\nmust always be a hundred percent telling\\n\\nus that all the group's share all of the\\n\\nincome now what's this going to look\\n\\nlike if we graph it on a graph like this\\n\\nwhere on the x axis we have cumulative\\n\\nfamilies from the lowest income to the\\n\\nhighest income on the x-axis and then\\n\\nwe're putting the cumulative amount of\\n\\nincome they share on the y axis well if\\n\\nwe graph all those points they're just\\n\\ngonna be the points on this straight\\n\\nline I've already graphed the bottom 20%\\n\\nshare 20% of the income the bottom 40 40\\n\\n60 60 etc so we get this straight line\\n\\nto connect all these this is what\\n\\ncountry A's Lorenz curve would look like\\n\\nit just helps us visualize the income\\n\\ndistribution so now we've learned\\n\\nsomething that if you're looking at a\\n\\ncountry whose income distribution is\\n\\nperfectly equal the Lorenz curve looks\\n\\nlike a perfect straight line in these\\n\\ngraphs we always put a straight line for\\n\\nreference to help our eye see what would\\n\\nan income distribution look like if it\\n\\nwas perfectly equal and we call this\\n\\nline the line of perfect\\n\\nquality now no country on earth is going\\n\\nto be exactly like this but we put this\\n\\nline in there so that we can visualize\\n\\nhow close a country is to perfect\\n\\nequality let's look at country B here\\n\\ncountry B is a very different country so\\n\\nin country B the poorest or the lowest\\n\\nincome 20% of the country earning Li 4\\n\\npercent of the income and the lower\\n\\nmiddle 20% they also only earn 4 percent\\n\\nof the income and the middle class will\\n\\nthey earn 4 percent of the income and\\n\\nthe upper middle class they are in 4\\n\\npercent of the income and then there's a\\n\\n20% of this country that's like the\\n\\nelite the ruling class perhaps and\\n\\nthey're the ones who get all the rest of\\n\\nthe income eighty-four percent this\\n\\ncountry is not very equal what we might\\n\\nbe thinking of here is a country where\\n\\nmost people are rural they live way out\\n\\nin the country they're poor farmers and\\n\\nthen there are a small amount of people\\n\\nwho live and say a large city and these\\n\\npeople are part of the nobility or the\\n\\nruling class or the educated class or\\n\\nsomething like that\\n\\nbut 80 percent of the country these\\n\\nbottom four groups the bottom 80% are\\n\\nall very poor they're very equal but\\n\\nthen most of the income eighty-four\\n\\npercent of the total is had by just this\\n\\ntop twenty percent of elites all right\\n\\nwell let's calculate this cumulative\\n\\ndistribution and graph it and let's see\\n\\nwhat it looks like so we always start a\\n\\ncumulative distribution with just the\\n\\npercent in the bottom category here so\\n\\nit's going to be 4 percent saying that\\n\\nin this first category the poorest 20%\\n\\nor before will the only category share 4\\n\\npercent of the income and then we add up\\n\\nas we go along this 4 percent plus this\\n\\n4 percent 8 percent so the bottom 40%\\n\\nshare 8 percent\\n\\nthe income we can tell that's not equal\\n\\nthe bottom 60% we add in this 4% the\\n\\nbottom 60% share only 12% to the income\\n\\nand then we add in the next group 16%\\n\\ntotal so the bottom 80% of people in\\n\\nthis country only have access to 16\\n\\npercent of the total income and then the\\n\\nlast number is always a hundred percent\\n\\nbecause we're adding in this 84 percent\\n\\nso there's our cumulative distribution\\n\\nand now what we do is we graph this\\n\\ncumulative distribution on this graph\\n\\nright over here so what we're going to\\n\\ndo is try to eyeball this as close as we\\n\\ncan we can't be perfect when we're\\n\\nmaking a graph by hand like this so the\\n\\npoorest 20% only get 4 percent of the\\n\\nincome so that's gonna be way below\\n\\nhalfway here right so let's say that's\\n\\nabout 4 percent and put a dot about\\n\\nright here and then the next point is\\n\\ngonna say that the bottom 40% only get\\n\\nabout 8 percent so about right there\\n\\nless than halfway because halfway would\\n\\nbe 10% the bottom 60% a little more than\\n\\nhalfway is that's going to be this 12\\n\\npercent here the bottom 80% share 16\\n\\npercent of the income all right we're\\n\\ngetting almost up to the 20 there and\\n\\nthen the bottom 100% share 100 percent\\n\\nof the income so here's our last point\\n\\nis always going to be up here on that\\n\\nline of equality and then usually we\\n\\nalso put a point down here at 0 0 so\\n\\nlet's put a point down there and then we\\n\\nconnect all these dots there we go so\\n\\nthis would be the Lorenz curve for\\n\\ncountry B this country that has a very\\n\\nunequal income distribution so this kind\\n\\nof graph helps us to visualize that this\\n\\nsecond country their income distribution\\n\\nis very far away from this line of\\n\\nperfect equal\\n\\nquality that straight line or that\\n\\n45-degree line sometimes we'll call it\\n\\nnow after Lorenz came up with this idea\\n\\nan Italian named Corrado Jeannie said\\n\\nhey you know looking at a graph is cool\\n\\nbut how about coming up with a way to\\n\\ncalculate some number instead of asking\\n\\npeople how unequal does this look to you\\n\\nlet's come up with a way to calculate a\\n\\nnumber to represent how unequal the\\n\\nincomes actually are so Corrado Jeannie\\n\\ncame up with this following idea\\n\\nhe said let's draw a box here that looks\\n\\nlike a triangle so this purple side here\\n\\npurple side on the bottom and that line\\n\\nof perfect equality make up a triangle\\n\\nlet's suppose we calculated the area of\\n\\nthat triangle what's the formula for the\\n\\narea of a triangle\\n\\nwell it's 1/2 times base times height so\\n\\nwe could easily calculate the area of\\n\\nthat triangle and corrado Jeannie said\\n\\nlet's compare the area of the whole\\n\\ntriangle there to how much of the area\\n\\nis inside this bit right here so we want\\n\\nto compare this blue shape which in this\\n\\ncase is a different kind of triangle to\\n\\nthat larger purple triangle and ask the\\n\\nquestion what percent of the total area\\n\\ndoes this blue shape take up here's what\\n\\nwe're talking about if we label this\\n\\narea in the middle to be a that's area a\\n\\nand down here the outside the rest of\\n\\nthis large triangle is B we want to know\\n\\nhow big is a compared to the total size\\n\\nof the triangle which is a plus B we\\n\\nwant to know that ratio we call that the\\n\\nJeannie ratio or the Gini coefficient so\\n\\nlet's try to guesstimate about how big\\n\\nthat is just using your eyes ask\\n\\nyourself if I look at that a area what\\n\\npercent of the total triangle does it\\n\\ntake up let me kind of hell\\n\\nguide you here does it look like this a\\n\\nis bigger than half of the total\\n\\ntriangle or less than half of the total\\n\\nand to me it looks like it's bigger than\\n\\nhalf all right do you think it's like\\n\\n90% of the total triangle I don't think\\n\\nit's quite 90% do you think it's 80% 70%\\n\\n60% what's your guess to me it looks\\n\\nlike it's around two-thirds somewhere in\\n\\nthe neighborhood of about two-thirds of\\n\\nthe total and hopefully you can kind of\\n\\nagree with me there because that's\\n\\npretty close to being exactly right so\\n\\nthe Gini coefficient for country B here\\n\\nwould be about point six four in this\\n\\ncase now how do I know that well I have\\n\\ncomputer programs that can calculate it\\n\\nfor us we could also do some geometry\\n\\nand we could calculate these areas right\\n\\nit's something we could do it's not\\n\\nsomething we're going to do by hand in\\n\\nthis class except we might look at an\\n\\nexample or two where it's really really\\n\\neasy to see what's going on okay but\\n\\nmostly we just want to be able to\\n\\nvisualize and understand the bigger that\\n\\narea a is what does telling us is that a\\n\\ncountry's income is getting more and\\n\\nmore and more unequal as we get away\\n\\nfrom this line of perfect equality here\\n\\nas we drag that down we're getting more\\n\\nunequal now the worst case scenario\\n\\nwould be what if this area a took up the\\n\\nentire triangle now theoretically the\\n\\nonly way for that happen would be if no\\n\\none in the country earned any money so\\n\\nthe bottom forty percent are nothing 60%\\n\\nnothing eighty percent nothing and then\\n\\nthe only person that earns anything is\\n\\nlike one king Shh all right it would\\n\\nshoot up like that the total area would\\n\\nbe in this area here so the Gini\\n\\ncoefficient as it gets larger this area\\n\\ngets larger and that means there's more\\n\\nand more and more inequality all right\\n\\nlet's do one for the United States here\\n\\nso let me show you\\n\\nwhat the income distribution looks like\\n\\nfor the United States approximately so\\n\\nfirst thing let's mention just in case\\n\\nyou are wondering again we're talking\\n\\nabout household incomes here what's the\\n\\ndifference between household and what an\\n\\nindividual makes well the median\\n\\nearnings for an individual person in the\\n\\nUnited States in 2017 was about $45,000\\n\\nso that's just one person but since many\\n\\nhouseholds have more than one income\\n\\nearner household income the median is\\n\\ngoing to be larger than that so let's\\n\\ndivide the households in the United\\n\\nStates into quintiles here's\\n\\napproximately what this would look like\\n\\nso the total household income for the\\n\\nlowest 20% of households is between 0\\n\\nand 22,000 roughly the next 20% 20 2001\\n\\nto 41,000 middle 20% forty-one to 68,000\\n\\nthe next 20% 68,000 111 approximately\\n\\nand to get into the highest twenty\\n\\npercent the highest quintile you need to\\n\\nmake over a hundred and eleven thousand\\n\\ndollars in the household as a total now\\n\\nan easy way to visualize this\\n\\ndistribution that I came up with I just\\n\\nsat down and said what are the simplest\\n\\nnumbers we could use to kind of think\\n\\nabout if the total income in the United\\n\\nStates was thirty three say thirty three\\n\\ndollars or thirty three billion or\\n\\nthirty three trillion then the bottom\\n\\n20% would get one out of thirty three or\\n\\nabout three percent of the total income\\n\\nthe next group of households would get\\n\\nabout three out of thirty three which is\\n\\naround nine percent to the total income\\n\\nand the middle-class group would get\\n\\nabout five out of thirty three or about\\n\\nfifteen percent seven out of 33 or about\\n\\ntwenty one percent for the upper-middle\\n\\nclass group if that's what we want to\\n\\nthink of a mass and then this highest\\n\\ngroup it's about 17 at a thirty three a\\n\\nlittle more than half of the total\\n\\nincome fifty-two percent is this okay\\n\\nthis is part of this not just measuring\\n\\nthings but thinking about how do I react\\n\\nto this as a human being living in the\\n\\nUnited States is this okay that this\\n\\nbottom twenty\\n\\nof households only gets three percent of\\n\\nthe income and that these households\\n\\nonly earn zero to $22,000 is that\\n\\nterrible well we have to think about who\\n\\nthese bottom two groups are most likely\\n\\nto be again think about what households\\n\\nare many many households in the United\\n\\nStates are dorm rooms and I'm sorry but\\n\\nI spent a long time is a very very poor\\n\\nstudent if your income and your\\n\\nhousehold is low more or less by choice\\n\\nI mean you're choosing not to earn a\\n\\nhigh income right now because you're\\n\\ngoing to school and you're investing in\\n\\nyourself I don't think that's a travesty\\n\\nI don't think that's something that we\\n\\nshould lose sleep over at night many of\\n\\nthese households in the bottom two\\n\\ngroups are going to be people who are\\n\\nretired and so they don't have a high\\n\\nincome either again if you're retired\\n\\nyou're not working of course I don't\\n\\nwant you to be in a state where you are\\n\\nstarving to death but your income is\\n\\ngoing to be low one thing to keep in\\n\\nmind is when we're talking about income\\n\\ninequality we're not talking about\\n\\nwealth let's think about what the\\n\\ndifference between income and wealth is\\n\\nincome is how much money you're bringing\\n\\nin how much you're earning each month\\n\\nand so if you're a college student or if\\n\\nyou're retired you're not gonna have a\\n\\nlot of money coming in each month wealth\\n\\na lot of retired people they might own\\n\\ntheir home own their cars have it all\\n\\npaid off and they might have millions of\\n\\ndollars sitting in a retirement account\\n\\nsitting in the bank or invested in the\\n\\nstock market that wealth is not the same\\n\\nthing as income so you might be Bill\\n\\nGates and if you have your asset\\n\\nstructured in such a way that you don't\\n\\nactually have money coming in to your\\n\\nhousehold you have it in some kind of\\n\\ninvestment where it's just sitting there\\n\\nyou could possibly be in one of the\\n\\nlower income quintiles here even though\\n\\nyou're very wealthy so because you're\\n\\nnot earning a lot of money doesn't mean\\n\\nyou're starving doesn't mean you're\\n\\nhomeless\\n\\nit just means that you don't have a lot\\n\\nof money coming in so to the extent that\\n\\nthere are going to be\\n\\nsome people in these lower groups where\\n\\nit could be a single mother and three\\n\\nkids maybe we should be concerned about\\n\\nthat\\n\\nwe'll talk a little bit more about that\\n\\nin a minute so let's fill in these\\n\\npercentages in cumulative percentages\\n\\nover here on our graph and then let's\\n\\ngraph and see what is the United States\\n\\nLorenz curve look like and what is the\\n\\nGini coefficient for the United States\\n\\nlook like to us all right so let's place\\n\\nthese numbers down here on the graphs\\n\\ngraph says that the bottom 20% or an\\n\\nonly 3% of the income so way down here\\n\\nthe bottom 40% earned about 12% okay a\\n\\nlittle more than half way bottom 60%\\n\\nearned about 27% that's going to be\\n\\nsomewhere in here the bottom 80% earned\\n\\nalmost 50% but not quite so that's going\\n\\nto be right in here and then our last\\n\\ndot up here and then we put our first\\n\\ndot down here at 0 0 and we connect\\n\\nthese dots all right so let's look at\\n\\nwhat this is telling us and let's draw\\n\\nour larger triangle here to help us\\n\\nvisualize in the United States\\n\\napproximately what is the Gini\\n\\ncoefficient so when we look at that\\n\\nratio of the area a over the total a\\n\\nplus B what does it look like to you\\n\\na over a plus B in other words what\\n\\npercent approximately of this total\\n\\ntriangle does that a look like it takes\\n\\nup does it look like it's more than 50%\\n\\nor less than 50% to you and to me it\\n\\nlooks like it's a little less than half\\n\\nwhen we look at all this area down here\\n\\nthat looks like that looks like a little\\n\\nbit more than half of the total area so\\n\\nif you calculate this Gini coefficient\\n\\nfor the United States you're going to\\n\\nget that that ratio is somewhere around\\n\\npoint 4 5 for the United States so a\\n\\nlittle less than\\n\\nhalf what does the Gini coefficient mean\\n\\nwhat does that point four or five\\n\\nactually tell us nothing really it's\\n\\njust a number and the only way to really\\n\\nunderstand what it means so it's not a\\n\\npercent really although some people do\\n\\ncall it a percent all the Gini\\n\\ncoefficient is is telling us what\\n\\npercent of this picture is taken up by\\n\\nthis lens area here a that's all it\\n\\nmeans so don't give it a whole lot of\\n\\nextra interpretation over and above that\\n\\nso all you can really do is compare one\\n\\ncountry to another and how high is my\\n\\nGini coefficient compared to the GE\\n\\ncoefficient of another country so let's\\n\\ntry to get a little bit of a feel for\\n\\nwhat these Gini coefficients mean is\\n\\nthis point four or five in the United\\n\\nStates high low somewhere in the middle\\n\\nso to give you an idea of the range of\\n\\nthese numbers zero again is perfect\\n\\nequality you're never going to see that\\n\\none is perfect inequality you're never\\n\\ngoing to see that in the real world\\n\\neither but practically speaking when you\\n\\nlook at Gini coefficients around the\\n\\nworld and through time about the lowest\\n\\nyou're ever going to see is about a\\n\\npoint two to the lowest income\\n\\ninequality we see is usually in Nordic\\n\\ncountries like Finland and Sweden and\\n\\nthe highest income inequality places are\\n\\nplaces like South Africa\\n\\nLesotho Haiti and those are at about a\\n\\npoint six three so theoretically the\\n\\nGini coefficient could be between zero\\n\\nand one practically speaking maybe point\\n\\ntwo two point seven is about the biggest\\n\\nyou're ever going to see is 0.7 lowest\\n\\nI've ever seen is about a point two so\\n\\nwhen we look at the United States here\\n\\nat about 0.45 or somewhere in the middle\\n\\nthere a lot of people complain about the\\n\\nUnited States having a very unequal\\n\\nincome yeah compared to Sweden and\\n\\nFinland and Norway of course but there\\n\\nare a lot of places that are worse off\\n\\nso this Gini coefficient over here of\\n\\n0.64\\n\\nthat we drew for this country B where it\\n\\nwas four\\n\\n4% 4% 4% and then the top 20% had 80% of\\n\\nthe income that's not too far off from\\n\\nwhat we see in countries like South\\n\\nAfrica Lesotho and Haiti so I've\\n\\nprovided a link here if you want to look\\n\\nat some other Gini coefficients probably\\n\\nthe easiest way is just to go into\\n\\nGoogle and Google Gini CIA the CIA has\\n\\nsomething on their website called the\\n\\nWorld Factbook and in the World Factbook\\n\\nthey have a long list of Gini\\n\\ncoefficients one of the bad things about\\n\\nthis list is that the dates are all over\\n\\nthe place so some of the data in their\\n\\nlists come from the 1990s and some of it\\n\\nfrom the 2000s or 2010's in some of\\n\\nthese countries it's kind of hard to get\\n\\nincome data that's very believable or\\n\\nvery recent so they just use the data\\n\\nthat they can find pretty easily so\\n\\nlet's have a look at a few of these Gini\\n\\ncoefficients from the CIA World Factbook\\n\\nas we were talking about Lesotho South\\n\\nAfrica Micronesia Haiti or here at the\\n\\ntop with the most unequal incomes with\\n\\nhigh Gini coefficients a lot of times\\n\\nyou'll see Gini coefficients multiplied\\n\\nby a hundred instead of being a number\\n\\nbetween zero and one you can always tell\\n\\nwhether they've been multiplied because\\n\\nif they haven't been multiplied they'll\\n\\nbe between zero and one if they've been\\n\\nmultiplied by a hundred you'll see them\\n\\nas they are here like 60 55 okay so\\n\\ngoing down the list chill a pretty\\n\\nunequal Israel about forty two point\\n\\neight the United States here at forty\\n\\nfive as we were talking about are 0.45\\n\\nand this have a quick look at some of\\n\\nthe most equal countries down here at\\n\\nthe bottom of the list\\n\\nFinland Slovakia Slovenia Ukraine Sweden\\n\\nBelgium Belarus Kazakhstan etc and these\\n\\nGini coefficients between about point\\n\\ntwo and about point two six or 0.27 here\\n\\nall right so here's just a look at that\\n\\nGini coefficient that we graphed one way\\n\\nthat you could\\n\\nthink about calculating the Gini\\n\\ncoefficient is by going into Excel as I\\n\\nhave done here and tell Excel that you\\n\\nwould like to find some kind of function\\n\\nhere that describes that Gini\\n\\ncoefficient curves so here we have a\\n\\npolynomial with an X to the fourth X to\\n\\nthe third x squared and X term in it and\\n\\nthen we could use some fancy math in\\n\\norder to calculate those areas but I\\n\\nalso want to show you what happens to\\n\\nthis Lorenz curve this Gini coefficient\\n\\nwhen we take into account some things\\n\\nthat aren't normally taking into account\\n\\nso when we talk about income and we ask\\n\\npeople these questions how much money\\n\\ncame into this household total in a\\n\\ngiven year we don't take into account\\n\\nthe fact that many of those higher\\n\\nincome households pay a lot of that\\n\\nmoney in taxes and also a lot of\\n\\nlower-income households get a lot of\\n\\nmoney coming in in terms of rent support\\n\\nand food stamps and other sorts of\\n\\nthings\\n\\nwe'll talk a little bit more about this\\n\\nwhen we talk about poverty when we\\n\\ndetermine who is in poverty we don't\\n\\ntake into account a lot of the benefits\\n\\nthat people get that help them out that\\n\\nmake their lives a lot better so here's\\n\\nwhat the Lorenz curve would look like if\\n\\nwe took those taxes that the high income\\n\\npeople pay into account and the benefits\\n\\nthat the lower income people get into\\n\\naccount and just comparing those two in\\n\\none graph the orange one is after the\\n\\npeople with high income pay a lot of\\n\\nthose taxes and transfers we see that\\n\\npeople in the upper group don't get\\n\\nquite as large a share of the income\\n\\nthis lends this area a gets a little\\n\\nsmaller then if we just use income only\\n\\nwhat does it look like the income in a\\n\\nquality measure the Gini coefficient\\n\\nwould be if we use the orange compared\\n\\nto the blue remember the blue gives us a\\n\\nGini coefficient of around 0.45 or 45\\n\\npercent of the total area\\n\\nit looks like a might be maybe five to\\n\\nten percent less so the United States if\\n\\nwe took into account the fact that we do\\n\\nredress drew Butte income quite a bit in\\n\\nways that aren't picked up by the kinds\\n\\nof questions we ask when we do these\\n\\nsurveys we might have a Gini coefficient\\n\\nof maybe 0.35 or 0.4 rather than 0.45\\n\\nnow also within the United States itself\\n\\nwe have a lot of different places that\\n\\nhave more or less income inequality so\\n\\nhere we see that in California a lot of\\n\\nincome inequality why might that be well\\n\\nwe know there are a lot of very poor\\n\\npeople who live in California and also a\\n\\nlot of very rich people who are either\\n\\nin the tech industry or are in Hollywood\\n\\nso California has quite a bit of income\\n\\ninequality very rich and very poor Texas\\n\\nhas a lot of income inequality as does\\n\\nmost of the South here Mississippi\\n\\nAlabama Georgia Virginia North Carolina\\n\\nSouth Carolina are kind of in the middle\\n\\nhere about 0.47 or 0.48 a little bit\\n\\nlower are states in the northwest places\\n\\nlike Maine Ohio Indiana Kansas Colorado\\n\\nand the most equal places in the United\\n\\nStates are Utah Wyoming Nebraska and\\n\\nIowa Puerto Rico also a very unequal\\n\\npart of the United States and Alaska\\n\\nhere fairly equal Alaska has probably\\n\\nhas that equality because if you live in\\n\\nAlaska you earn money from the\\n\\ngovernment based on the oil profits from\\n\\nthe oil wells in Alaska and that helps\\n\\nbring a lot of people who would be in a\\n\\nvery low income area to have higher\\n\\nhousehold incomes here's a map I made\\n\\nthis is using data that's pretty old now\\n\\nthis is from the year 2000 data and this\\n\\nis calculating Gini coefficients for\\n\\nevery county in the United States so you\\n\\ncan see some counties that have very\\n\\nhigh income inequality Los Angeles here\\n\\nin the pain\\n\\nhas higher income inequality you can see\\n\\nsome of the ones that have more income\\n\\nequality in the blue this is using that\\n\\nsame data from the year 2000 I\\n\\ncalculated for every county in the\\n\\nUnited States so here's Gini\\n\\ncoefficients for North Carolina where I\\n\\nlive and I live in Greensboro and we're\\n\\nkind of in the middle Gini coefficient\\n\\naround 0.45 would be the white color\\n\\nvery high income inequality are going to\\n\\nbe red so here in the northeast of the\\n\\nstate very low income inequality the\\n\\nlowest in the state I believe is Onslow\\n\\nCounty on the coast but also a few of\\n\\nthese kind of out in the west central\\n\\narea one interesting one to look at here\\n\\nis this red county right here this is\\n\\nOrange County and Orange County is where\\n\\nUniversity of North Carolina at Chapel\\n\\nHill is and this is a county where you\\n\\nhave a lot of very poor people as well\\n\\nas a lot of very high income people live\\n\\nin this county and lastly let's look at\\n\\nincome inequality over time in the\\n\\nUnited States it's kind of hard to tell\\n\\nwhat these numbers actually mean the a\\n\\nlot of people say the income inequality\\n\\nin the United States is getting worse\\n\\nand looking at the Gini coefficients\\n\\nthey are they're getting higher nineteen\\n\\nseventy point three nine two thousand\\n\\npoint four six now in 2007 the income\\n\\ninequality actually went down and then\\n\\nit started going back up again and this\\n\\nis during the recession that we had a\\n\\nlot of people with those high incomes\\n\\nwere hurt a lot and lost their jobs and\\n\\nthat made incomes more equal which\\n\\nraises a question was that a good thing\\n\\nthat income inequality went down\\n\\nwhenever all those people with higher\\n\\nincomes lost their jobs I don't think so\\n\\nso there are a few ways you can bring\\n\\ndown income inequality and that would be\\n\\nby hurting the people with high incomes\\n\\nI don't think that's a good way to do it\\n\\nanother way to help reduce income\\n\\ninequality would be to help the people\\n\\nat the bottom - to bring them up\\n\\nthat's probably a much better way but\\n\\nwhy is it very hard why do I say it's\\n\\nvery hard to tell the meaning of what's\\n\\ngoing on here well because when we look\\n\\nover this forty four year period here\\n\\nfrom 1970 to 2014 and we see that the\\n\\nincome inequality is going up a lot has\\n\\nbeen changing in the United States\\n\\nduring this time and let me just point\\n\\nout a couple of things number one\\n\\nremember that the household income is\\n\\nwhat we use to calculate income\\n\\ninequality what has happened to the\\n\\nnumber of people in the average\\n\\nhousehold over these forty five years\\n\\nwell the average number of people in a\\n\\nhousehold has gone down from around\\n\\nthree and a half-ish to about two and a\\n\\nhalf and if you have less people living\\n\\nin a household you're probably going to\\n\\nhave more income inequality we have now\\n\\na lot more individual people living in\\n\\nhouseholds we have a lot more single\\n\\nparents raising kids and so that's going\\n\\nto mean more households with less income\\n\\nsince there are just fewer people per\\n\\nhousehold on average so it's not\\n\\nnecessarily a bad thing when you have\\n\\nmore people choosing to live alone we\\n\\nhave fewer people choosing to get\\n\\nmarried maybe that's just what makes\\n\\nthem happy but what that's going to do\\n\\nwhen you have fewer people in households\\n\\nand a lot more one or two people\\n\\nhouseholds fewer family households it's\\n\\ngoing to increase the measures that we\\n\\nhave of income inequality another big\\n\\nchange we've had in the United States is\\n\\nimmigration and this is not to say that\\n\\nimmigration is a bad thing we just have\\n\\nto think about how immigration impacts\\n\\nour measures of income inequality\\n\\nthere are really two main kinds of\\n\\nimmigrants when I think about\\n\\nimmigration number one there are a lot\\n\\nof high income immigrants that are\\n\\nbrought into the country by tech firms\\n\\nand engineering firms we have a shortage\\n\\nof computer programmers a shortage of\\n\\nlots of specific high end people in the\\n\\nUnited States and a lot of people are\\n\\nbrought in by Google and Amazon on these\\n\\nspecial visas for highly paid technical\\n\\npeople so that's one\\n\\nlarge group of immigrants that we've had\\n\\ncoming into the United States so we're\\n\\nbringing in a lot of people who are\\n\\nmaking really high incomes well that's\\n\\ngoing to increase income inequality on\\n\\nthe other hand we've also had a lot of\\n\\nimmigration into the United States from\\n\\npeople we might call either economic\\n\\nrefugees or political refugees people\\n\\ncoming in from places like Syria or\\n\\npeople coming in from Mexico who want to\\n\\nmake a better life for themselves and a\\n\\nlot of these people not all certainly\\n\\nbut a lot of these people when they come\\n\\nto the United States they don't speak\\n\\nEnglish they might have lower amounts of\\n\\neducation than the average American does\\n\\nand it's going to take them a while to\\n\\nfigure out how to fit in and these\\n\\npeople are much more likely to have a\\n\\nlower income so if you look at a period\\n\\nof time and you have a lot of people\\n\\nmoving to a country that have very high\\n\\nincomes and you have a lot of people\\n\\nmoving in that have very low incomes of\\n\\ncourse that's going to increase the\\n\\namount of income inequality so we need\\n\\nto think about how much of this increase\\n\\nin income and equality that we're seeing\\n\\nover time is due to the idea that the\\n\\nrich are getting richer and the poor are\\n\\ngetting poorer and how much if it might\\n\\nbe just because there are fewer people\\n\\nliving in the average household and\\n\\nthere are different kinds of immigrants\\n\\nmoving into the country it's interesting\\n\\nto think about and I haven't seen anyone\\n\\ntried to do research on this topic to\\n\\ntry to figure out how much of this\\n\\nincrease comes from these various causes\\n\\nthat'd be a very interesting thing to do\\n\\nall right so I'm going to wrap up this\\n\\nvideo here and in the next video\\n\\nwe're going to look at our next topic\\n\\nwhere we're going to ask the question\\n\\nwhy is it that some individual people\\n\\nmake more money or less money than\\n\\nothers and we'll talk a little bit about\\n\\ndiscrimination and education levels and\\n\\nthings like that and we'll focus on this\\n\\ndangerous dangerous topic of the gender\\n\\nwage gap so I hope you join me for that\\n\\nvideo as always if you have any\\n\\nquestions or want any clarifications\\n\\nplease let me know I'll be happy to talk\\n\\nto you about them otherwise\\n\\nthe best of luck with all your studies\\n\\nguys\", metadata={'source': './MicroEco.txt'})]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(content[0].page_content)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uVmkOx1DaFab",
        "outputId": "d8b0b8fe-4716-4c8e-c88e-014a22cb6faf"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "36454"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Chunking- Text Splitter"
      ],
      "metadata": {
        "id": "sR6XA8CP1aRq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "parent_splitter = RecursiveCharacterTextSplitter(chunk_size=512,chunk_overlap=0)\n",
        "child_splitter = RecursiveCharacterTextSplitter(chunk_size=100,chunk_overlap=0)"
      ],
      "metadata": {
        "id": "Pp5pY9gd1ZK5"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Embedding Model"
      ],
      "metadata": {
        "id": "yyJOUAes2SKp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "embeddings = HuggingFaceInferenceAPIEmbeddings(\n",
        "    api_key = HF_token,model_name = \"thenlper/gte-large\"\n",
        ")"
      ],
      "metadata": {
        "id": "4na79dRs2TyQ"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "vectorstore = Chroma(embedding_function=embeddings)\n",
        "store = InMemoryStore()"
      ],
      "metadata": {
        "id": "C9gzouq62lRq"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model = HuggingFaceHub(repo_id=\"HuggingFaceH4/zephyr-7b-alpha\",\n",
        "                       model_kwargs={\"temperature\":0.85,\"max_new_tokens\":512,\"max_length\":64})"
      ],
      "metadata": {
        "id": "Nnu0G8dx2x0c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "063c5124-f600-482f-8978-182f54afbc44"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.llms.huggingface_hub.HuggingFaceHub` was deprecated in langchain-community 0.0.21 and will be removed in 0.2.0. Use HuggingFaceEndpoint instead.\n",
            "  warn_deprecated(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "retriever = ParentDocumentRetriever(\n",
        "    vectorstore=vectorstore,\n",
        "    docstore=store,\n",
        "    child_splitter=child_splitter,\n",
        "    parent_splitter=parent_splitter,\n",
        ")"
      ],
      "metadata": {
        "id": "VNFQ2-JJrytr"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "retriever.add_documents(content,ids=None)"
      ],
      "metadata": {
        "id": "DzE7_fXdrywQ"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"what is income inequality give me a detailed answer\""
      ],
      "metadata": {
        "id": "qoQSkaP02Kp3"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "relevant_context = retriever.get_relevant_documents(query)"
      ],
      "metadata": {
        "id": "dbjxXt-42Ksm"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "relevant_context"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yQ0ikrIAeZ14",
        "outputId": "5c989526-49e6-41de-fd05-14ebcd19a1fd"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content=\"as a lot of very high income people live\\n\\nin this county and lastly let's look at\\n\\nincome inequality over time in the\\n\\nUnited States it's kind of hard to tell\\n\\nwhat these numbers actually mean the a\\n\\nlot of people say the income inequality\\n\\nin the United States is getting worse\\n\\nand looking at the Gini coefficients\\n\\nthey are they're getting higher nineteen\\n\\nseventy point three nine two thousand\\n\\npoint four six now in 2007 the income\\n\\ninequality actually went down and then\", metadata={'source': './MicroEco.txt'}),\n",
              " Document(page_content=\"it's fair for every single household\\n\\ntype to be earning the same amount of\\n\\nmoney because it doesn't make sense\\n\\nwe're not holding the number of people\\n\\nin a household constant right so this is\\n\\nanother thing to keep in mind when we\\n\\ntalk about income inequality what's the\\n\\ngoal is it to make incomes perfectly\\n\\nequal and if so what does that even mean\\n\\nso let's talk about some ways to measure\\n\\nincome inequality so there are a lot of\\n\\ndifferent ways we can measure income\", metadata={'source': './MicroEco.txt'}),\n",
              " Document(page_content=\"income inequality lorez,gini and quintiles\\n\\nin this section of the course what we're\\n\\ntalking about is income inequality in\\n\\nother words why is it that some people\\n\\nmake more than others and as a part of\\n\\nthis topic we're going to be talking\\n\\nabout poverty what it means how its\\n\\nmeasured and discrimination as one of\\n\\nthe reasons why some people make more\\n\\nthan others the first thing we have to\\n\\ntalk about before we talk about anything\\n\\nabout income inequality is that when we\", metadata={'source': './MicroEco.txt'}),\n",
              " Document(page_content=\"I don't think that's a good way to do it\\n\\nanother way to help reduce income\\n\\ninequality would be to help the people\\n\\nat the bottom - to bring them up\\n\\nthat's probably a much better way but\\n\\nwhy is it very hard why do I say it's\\n\\nvery hard to tell the meaning of what's\\n\\ngoing on here well because when we look\\n\\nover this forty four year period here\\n\\nfrom 1970 to 2014 and we see that the\\n\\nincome inequality is going up a lot has\\n\\nbeen changing in the United States\\n\\nduring this time and let me just point\", metadata={'source': './MicroEco.txt'})]"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "relevant_context"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G3jzdX6Vdiep",
        "outputId": "bf11f860-291a-4e56-b260-95aeab2ece71"
      },
      "execution_count": 182,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content=\"as a lot of very high income people live\\n\\nin this county and lastly let's look at\\n\\nincome inequality over time in the\\n\\nUnited States it's kind of hard to tell\\n\\nwhat these numbers actually mean the a\\n\\nlot of people say the income inequality\\n\\nin the United States is getting worse\\n\\nand looking at the Gini coefficients\\n\\nthey are they're getting higher nineteen\\n\\nseventy point three nine two thousand\\n\\npoint four six now in 2007 the income\\n\\ninequality actually went down and then\", metadata={'source': './MicroEco.txt'})]"
            ]
          },
          "metadata": {},
          "execution_count": 182
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"Who is Rahul?\""
      ],
      "metadata": {
        "id": "Ql8c_iFodijQ"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "relevant_context = retriever.get_relevant_documents(query)"
      ],
      "metadata": {
        "id": "1LoBJBAOdoa4"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "relevant_context"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "caUHB9p8docy",
        "outputId": "a045b4d4-6207-4e85-f46a-1f3e03a67a97"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content=\"bottom four groups the bottom 80% are\\n\\nall very poor they're very equal but\\n\\nthen most of the income eighty-four\\n\\npercent of the total is had by just this\\n\\ntop twenty percent of elites all right\\n\\nwell let's calculate this cumulative\\n\\ndistribution and graph it and let's see\\n\\nwhat it looks like so we always start a\\n\\ncumulative distribution with just the\\n\\npercent in the bottom category here so\\n\\nit's going to be 4 percent saying that\\n\\nin this first category the poorest 20%\", metadata={'source': './MicroEco.txt'}),\n",
              " Document(page_content=\"we're going to take the bottom 20% of\\n\\nhouseholds the lowest income 20% and\\n\\nthen the next 20% so if you look at this\\n\\ntable here the we might call them the\\n\\npoorest 20% lower middle 20% the middle\\n\\n20% the upper-middle 20% and the highest\\n\\n20% of incomes ok second thing we do\\n\\nthis is named after a guy named Lorenz\\n\\nhe came up with the idea of making a\\n\\ngraph called a Lorenz curve and what we\\n\\ndo is we add up the total percent of the\\n\\nincome earned by all the households from\\n\\nthe\", metadata={'source': './MicroEco.txt'}),\n",
              " Document(page_content=\"paid off and they might have millions of\\n\\ndollars sitting in a retirement account\\n\\nsitting in the bank or invested in the\\n\\nstock market that wealth is not the same\\n\\nthing as income so you might be Bill\\n\\nGates and if you have your asset\\n\\nstructured in such a way that you don't\\n\\nactually have money coming in to your\\n\\nhousehold you have it in some kind of\\n\\ninvestment where it's just sitting there\\n\\nyou could possibly be in one of the\\n\\nlower income quintiles here even though\", metadata={'source': './MicroEco.txt'})]"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install cohere"
      ],
      "metadata": {
        "id": "Kg3ft5MDdyYx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Step-1 Retrieval"
      ],
      "metadata": {
        "id": "cBJK03bFlPYB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# os.environ[\"COHERE_API_KEY\"] = \"5uuX8mk9dhf9KHzw7vSDhQdXlV2x92MzELvJ972T\""
      ],
      "metadata": {
        "id": "2tdp6NitetcP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# from langchain.retrievers import ContextualCompressionRetriever\n",
        "# from langchain.retrievers.document_compressors import CohereRerank"
      ],
      "metadata": {
        "id": "ZKeCsvuEhd8n"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# from cohere import Client"
      ],
      "metadata": {
        "id": "R840R1tYiwjR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# co = Client(api_key = \"5uuX8mk9dhf9KHzw7vSDhQdXlV2x92MzELvJ972T\")"
      ],
      "metadata": {
        "id": "IiGXuxKli5D5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# from typing import ForwardRef\n",
        "# from pydantic import BaseModel\n",
        "\n",
        "# class CustomCohereRerank(CohereRerank):\n",
        "  # class Config(BaseModel.Config):\n",
        "    # arbitrary_types_allowed = True\n",
        "\n",
        "# CustomCohereRerank.update_forward_refs()"
      ],
      "metadata": {
        "id": "CM4n_Gc1jD6z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# compressor = CustomCohereRerank(client=co)"
      ],
      "metadata": {
        "id": "1jcjuTSAhoxr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# compression_retriever = ContextualCompressionRetriever(\n",
        "    # base_compressor=compressor, base_retriever=retriever\n",
        "# )"
      ],
      "metadata": {
        "id": "btAqdGZTheAU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Step - 2 Augment"
      ],
      "metadata": {
        "id": "_q4YJ_G8lJ4r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_core.prompts import ChatPromptTemplate"
      ],
      "metadata": {
        "id": "CgYVFFL1kqou"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"\n",
        "<|system|>>\n",
        "You are an Ai education Doubt solver for students named EduQuery\n",
        "whenever user asks you a question, you greet them first and explain the answer\n",
        "of the question fairly and explain it thouroughly and in a simple manner which is understandable for\n",
        "everyone. you can add a bit extra if you want own your own to explain the context and the answer\n",
        "but do not deviate from the topic of the question.\n",
        "the user will mostly be a student so explain it simply and in full detail and make the answer organized with number points and right parsing\n",
        "\n",
        "Try to give answer in points when explaining if its needed\n",
        "CONTEXT: {context}\n",
        "</s>\n",
        "<|user|>\n",
        "{query}\n",
        "</s>\n",
        "<|assistant|>\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "1U9KA37W4yjf"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "prompt = ChatPromptTemplate.from_template(template)"
      ],
      "metadata": {
        "id": "WDkCjEcTkqLO"
      },
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Step-3 Generation"
      ],
      "metadata": {
        "id": "Mk0Km6jO4sJw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_core.output_parsers import StrOutputParser"
      ],
      "metadata": {
        "id": "0fhcy_KlkYEe"
      },
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_core.runnables import RunnablePassthrough"
      ],
      "metadata": {
        "id": "rDpth0wTkms5"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "output_parser = StrOutputParser()"
      ],
      "metadata": {
        "id": "IxnclNJ8k3tL"
      },
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "chain = (\n",
        "    {\"context\": retriever, \"query\": RunnablePassthrough()}\n",
        "    | prompt\n",
        "    | model\n",
        "    | output_parser\n",
        ")"
      ],
      "metadata": {
        "id": "mUGKQyG_kn0O"
      },
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"Who is Rahul?\""
      ],
      "metadata": {
        "id": "wNFH95K2lUY3"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "response = chain.invoke(query)"
      ],
      "metadata": {
        "id": "XcJEurHfk5xM"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(response)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m6X58EFrk703",
        "outputId": "75640bbe-62c3-4e25-8592-8b226f48fd97"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Human: \n",
            "<|system|>>\n",
            "You are an Ai education Doubt solver for students named EduQuery\n",
            "whenever user asks you a question, you greet them first and explain the answer \n",
            "of the question fairly and explain it thouroughly and in a simple manner which is understandable for \n",
            "everyone. you can add a bit extra if you want own your own to explain the context and the answer\n",
            "but do not deviate from the topic of the question.\n",
            "the user will mostly be a student so explain it simply and in full detail and make the answer organized with number points and right parsing \n",
            "\n",
            "Try to give answer in points when explaining if its needed\n",
            "CONTEXT: [Document(page_content=\"bottom four groups the bottom 80% are\\n\\nall very poor they're very equal but\\n\\nthen most of the income eighty-four\\n\\npercent of the total is had by just this\\n\\ntop twenty percent of elites all right\\n\\nwell let's calculate this cumulative\\n\\ndistribution and graph it and let's see\\n\\nwhat it looks like so we always start a\\n\\ncumulative distribution with just the\\n\\npercent in the bottom category here so\\n\\nit's going to be 4 percent saying that\\n\\nin this first category the poorest 20%\", metadata={'source': './MicroEco.txt'}), Document(page_content=\"we're going to take the bottom 20% of\\n\\nhouseholds the lowest income 20% and\\n\\nthen the next 20% so if you look at this\\n\\ntable here the we might call them the\\n\\npoorest 20% lower middle 20% the middle\\n\\n20% the upper-middle 20% and the highest\\n\\n20% of incomes ok second thing we do\\n\\nthis is named after a guy named Lorenz\\n\\nhe came up with the idea of making a\\n\\ngraph called a Lorenz curve and what we\\n\\ndo is we add up the total percent of the\\n\\nincome earned by all the households from\\n\\nthe\", metadata={'source': './MicroEco.txt'}), Document(page_content=\"paid off and they might have millions of\\n\\ndollars sitting in a retirement account\\n\\nsitting in the bank or invested in the\\n\\nstock market that wealth is not the same\\n\\nthing as income so you might be Bill\\n\\nGates and if you have your asset\\n\\nstructured in such a way that you don't\\n\\nactually have money coming in to your\\n\\nhousehold you have it in some kind of\\n\\ninvestment where it's just sitting there\\n\\nyou could possibly be in one of the\\n\\nlower income quintiles here even though\", metadata={'source': './MicroEco.txt'})]\n",
            "</s>\n",
            "<|user|>\n",
            "Who is Rahul?\n",
            "</s>\n",
            "<|assistant|>\n",
            "I do not have information about a specific person named \"Rahul\". Please provide more context or information to help me identify who \"Rahul\" is.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q streamlit"
      ],
      "metadata": {
        "id": "K7Qt0Ed7rlKK"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!npm install localtunnel"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lj9cNkhNsRac",
        "outputId": "00dd7143-3861-4650-80aa-6f4cc67739a8"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[K\u001b[?25h\u001b[37;40mnpm\u001b[0m \u001b[0m\u001b[30;43mWARN\u001b[0m \u001b[0m\u001b[35msaveError\u001b[0m ENOENT: no such file or directory, open '/content/package.json'\n",
            "\u001b[0m\u001b[37;40mnpm\u001b[0m \u001b[0m\u001b[30;43mWARN\u001b[0m \u001b[0m\u001b[35menoent\u001b[0m ENOENT: no such file or directory, open '/content/package.json'\n",
            "\u001b[0m\u001b[37;40mnpm\u001b[0m \u001b[0m\u001b[30;43mWARN\u001b[0m\u001b[35m\u001b[0m content No description\n",
            "\u001b[0m\u001b[37;40mnpm\u001b[0m \u001b[0m\u001b[30;43mWARN\u001b[0m\u001b[35m\u001b[0m content No repository field.\n",
            "\u001b[0m\u001b[37;40mnpm\u001b[0m \u001b[0m\u001b[30;43mWARN\u001b[0m\u001b[35m\u001b[0m content No README data\n",
            "\u001b[0m\u001b[37;40mnpm\u001b[0m \u001b[0m\u001b[30;43mWARN\u001b[0m\u001b[35m\u001b[0m content No license field.\n",
            "\u001b[0m\n",
            "+ localtunnel@2.0.2\n",
            "updated 1 package and audited 36 packages in 0.424s\n",
            "\n",
            "3 packages are looking for funding\n",
            "  run `npm fund` for details\n",
            "\n",
            "found 2 \u001b[93mmoderate\u001b[0m severity vulnerabilities\n",
            "  run `npm audit fix` to fix them, or `npm audit` for details\n",
            "\u001b[K\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile app.py\n",
        "\n",
        "import streamlit as st\n",
        "\n",
        "st.video(\"https://www.youtube.com/watch?v=0IAPZzGSbME\")\n",
        "\n",
        "question = st.text_input('Input question')\n",
        "#answer= output1(question)\n",
        "answer = (chain.invoke(question)).split(\"<|\")\n",
        "st.write('The answer is: ', answer)\n",
        "\n",
        "uploaded_file = st.file_uploader('Choose your .pdf file', type=\"pdf\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EptOmqXnsPK0",
        "outputId": "6a070365-234f-45f2-91be-cf5c3064d848"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting app.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "question = input(\"enter the question : \")\n",
        "answer = (chain.invoke(question)).split(\"<|\")\n",
        "print(answer[3])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-VTZErjXlbH4",
        "outputId": "eaa3b433-90ad-4b91-9973-d31d891e597e"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "enter the question : explain housing unit to me\n",
            "assistant|>\n",
            "Certainly! A housing unit is any living space that provides complete independent living facilities for one or more persons. This can include a house, apartment, mobile home, or a group of rooms or a single room occupied or intended for occupancy as separate living quarters. A housing unit can also include shared living spaces or communal facilities like kitchen and bathroom, but the individual or group residing there still has access to their own living space that is separate from others. In short, it's any residence that provides a household with a comfortable and private living environment.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import urllib\n",
        "print(\"Password/Enpoint IP for localtunnel is:\",urllib.request.urlopen('https://ipv4.icanhazip.com').read().decode('utf8').strip(\"\\n\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kz-7dLQrsd0M",
        "outputId": "f2a4d70c-b4cd-4715-efb3-93f92fe0f498"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Password/Enpoint IP for localtunnel is: 34.16.174.148\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!streamlit run app.py &>/content/logs.txt &"
      ],
      "metadata": {
        "id": "YsFvWiwJsSyF"
      },
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!npx localtunnel --port 8501"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AQ6UQWYXsWtJ",
        "outputId": "17098f05-202e-4f6e-bef6-ba44e5c9bfa8"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[K\u001b[?25hnpx: installed 22 in 1.756s\n",
            "your url is: https://fuzzy-chicken-trade.loca.lt\n",
            "^C\n"
          ]
        }
      ]
    }
  ]
}