Demo2-Semantic-search-with-pretain-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5889142c",
   "metadata": {},
   "source": [
    "# Semantic Search with Amazon OpenSearch Service \n",
    "To create semantic search, we will add a vector representation of the metadata to our data set in OpenSearch, then do the same with our sample query \"Wildfires in Canada\". In OpenSearch, we'll use a KNN search to find matches based on a cosine similarity rating on the vector.\n",
    "We will:\n",
    "1. Use a HuggingFace sentence-transformer BERT model to generate sentence embedding for the geo.ca metadata dataset\n",
    "2. Upload the dataset to OpenSearch, with the original metadata schema text combined with the vector representation of the questions.\n",
    "3. Translate the query question to a vector.\n",
    "4. Perform a KNN search in OpenSearch to perform semantic search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9384cbf1",
   "metadata": {},
   "source": [
    "### 1. Check PyTorch Version\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c3494a2",
   "metadata": {},
   "source": [
    "As in the previous modules, let's import PyTorch and confirm that have have the latest version of PyTorch. The version should already be 2.0.1 or higher. If not, please run the lab in order to get everything set up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9c12fd5b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.1.0\n"
     ]
    }
   ],
   "source": [
    "import torch \n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef78e2e6",
   "metadata": {},
   "source": [
    "### 2. Retrieve notebook variables\n",
    "\n",
    "The line below will retrieve your shared variables from the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5061e933",
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9989253",
   "metadata": {},
   "source": [
    "### 3. import library "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "91183382",
   "metadata": {},
   "outputs": [],
   "source": [
    "# #installed in the previous notebook\n",
    "# !pip install -q boto3\n",
    "# !pip install -q requests\n",
    "# !pip install -q requests-aws4auth\n",
    "# !pip install -q opensearch-py\n",
    "# !pip install -q tqdm\n",
    "# !pip install -q boto3\n",
    "# !pip install -q install transformers[torch]\n",
    "# !pip install -q transformers\n",
    "# !pip install -q sentence-transformers rank_bm25\n",
    "# !pip install -q nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a9198016",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "import re\n",
    "import time\n",
    "import sagemaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a0fc037",
   "metadata": {},
   "source": [
    "### 4. Prepare BERT Model \n",
    "#### Option 1: DistilBert model\n",
    "For this module, we will be using the HuggingFace BERT model to generate vectorization data, where every sentence is 768 dimension data. Let's create some helper functions we'll use later on.\n",
    "![BERT](image/nlp_bert.png)\n",
    "\n",
    "We are creating 2 functions:\n",
    "1. mean_pooling\n",
    "2. sentence_to_vector - this is the key function we'll use to generate our vector embedding for the metadata dataset.\n",
    "\n",
    "A reason for not using DistilBert:\n",
    " Transformer models like DistilBert have a fixed maximum input length (512), and any input longer than this limit can cause errors during processing.Our input sequence length (1086 tokens) exceeds the model's maximum sequence length (512 tokens)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "10c3050f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/cuda/__init__.py:551: UserWarning: Can't initialize NVML\n",
      "  warnings.warn(\"Can't initialize NVML\")\n"
     ]
    }
   ],
   "source": [
    "# import torch\n",
    "# from transformers import AutoTokenizer, AutoModel\n",
    "# from transformers import DistilBertTokenizer, DistilBertModel\n",
    "\n",
    "# #model_name = \"distilbert-base-uncased\"\n",
    "# #model_name = \"sentence-transformers/msmarco-distilbert-base-dot-prod-v3\"\n",
    "# model_name = \"sentence-transformers/distilbert-base-nli-stsb-mean-tokens\" #https://huggingface.co/sentence-transformers/distilbert-base-nli-stsb-mean-tokens\n",
    "\n",
    "\n",
    "# #Mean Pooling - Take attention mask into account for correct averaging\n",
    "# def mean_pooling(model_output, attention_mask):\n",
    "#     token_embeddings = model_output[0] #First element of model_output contains all token embeddings\n",
    "#     input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()\n",
    "#     sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)\n",
    "#     sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)\n",
    "#     return sum_embeddings / sum_mask\n",
    "\n",
    "\n",
    "# def sentence_to_vector(raw_inputs):\n",
    "#     tokenizer = DistilBertTokenizer.from_pretrained(model_name)\n",
    "#     model = DistilBertModel.from_pretrained(model_name)\n",
    "#     inputs_tokens = tokenizer(raw_inputs, padding=True, return_tensors=\"pt\")\n",
    "    \n",
    "#     with torch.no_grad():\n",
    "#         outputs = model(**inputs_tokens)\n",
    "\n",
    "#     sentence_embeddings = mean_pooling(outputs, inputs_tokens['attention_mask'])\n",
    "#     return sentence_embeddings\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bd51625",
   "metadata": {},
   "source": [
    "#### Option 2: all-MiniLM-L6-v2\n",
    "We can also use sentence-transformer models ['all-MiniLM-L6-v2'](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) which simplifies the process of obtaining sentence embeddings. It has 384 dimensions.It designed for generating sentence embeddings directly, which means we can use the Sentence Transformers library's functionality to handle both tokenization and embedding in a more streamlined manner compared to manually handling with DistilBertModel and DistilBertTokenizer.\n",
    "\n",
    "The SentenceTransformer class's encode method directly handles the text input, tokenization, and conversion to sentence embeddings, eliminating the need for manual mean pooling. The encode method returns a tensor of sentence embeddings, where each embedding corresponds to the input sentences provided to the function.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "92559020",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py:11: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  from tqdm.autonotebook import tqdm, trange\n",
      "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "#!pip install -U sentence-transformers\n",
    "from sentence_transformers import SentenceTransformer, util\n",
    "import numpy as np \n",
    "\n",
    "# Load the Sentence Transformer model\n",
    "model_name = \"all-MiniLM-L6-v2\"\n",
    "model = SentenceTransformer(model_name)\n",
    "\n",
    "def sentence_to_vector(raw_inputs):\n",
    "    # Encode sentences to get sentence embeddings\n",
    "    sentence_embeddings = model.encode(raw_inputs, convert_to_tensor=True)\n",
    "    \"\"\"\n",
    "    When you work with vectors (such as embeddings) in Elasticsearch or OpenSearch, you need to convert the PyTorch tensor to a list of floats before indexing the document. \n",
    "    \"\"\"\n",
    "    encod_np_array = np.array(sentence_embeddings)\n",
    "    encod_list = encod_np_array.tolist()\n",
    "        \n",
    "    return encod_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48f8ddf3",
   "metadata": {},
   "source": [
    "### 5.Preprocess and embed the text\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f11644ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import boto3\n",
    "import json\n",
    "from requests_aws4auth import AWS4Auth\n",
    "import io\n",
    "import os \n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b0531c05",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load metadata\n",
    "def read_parquet_from_s3_as_df(region, s3_bucket, s3_key):\n",
    "    \"\"\"\n",
    "    Load a Parquet file from an S3 bucket into a pandas DataFrame.\n",
    "\n",
    "    Parameters:\n",
    "    - region: AWS region where the S3 bucket is located.\n",
    "    - s3_bucket: Name of the S3 bucket.\n",
    "    - s3_key: Key (path) to the Parquet file within the S3 bucket.\n",
    "\n",
    "    Returns:\n",
    "    - df: pandas DataFrame containing the data from the Parquet file.\n",
    "    \"\"\"\n",
    "\n",
    "    # Setup AWS session and clients\n",
    "    session = boto3.Session(region_name=region)\n",
    "    s3 = session.resource('s3')\n",
    "\n",
    "    # Load the Parquet file as a pandas DataFrame\n",
    "    object = s3.Object(s3_bucket, s3_key)\n",
    "    body = object.get()['Body'].read()\n",
    "    df = pd.read_parquet(io.BytesIO(body))\n",
    "    return df\n",
    "\n",
    "# Upload the duplicate date to S3 as a parquet file \n",
    "def upload_df_to_s3_as_parquet(df, bucket_name, file_key):\n",
    "    # Save DataFrame as a Parquet file locally\n",
    "    parquet_file_path = 'temp.parquet'\n",
    "    df.to_parquet(parquet_file_path)\n",
    "\n",
    "    # Create an S3 client\n",
    "    s3_client = boto3.client('s3')\n",
    "\n",
    "    # Upload the Parquet file to S3 bucket\n",
    "    try:\n",
    "        response = s3_client.upload_file(parquet_file_path, bucket_name, file_key)\n",
    "        os.remove(parquet_file_path)\n",
    "        print(f'Uploading {file_key} to {bucket_name} as parquet file')\n",
    "        # Delete the local Parquet file\n",
    "        return True\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        return False\n",
    "\n",
    "# Create new column 'organization_en' required by the API JSON response \n",
    "def extract_organisation_en(contact_str):\n",
    "    try:\n",
    "        # Parse the stringified JSON into Python objects\n",
    "        contact_data = json.loads(contact_str)\n",
    "        # If the parsed data is a list, iterate through it\n",
    "        if isinstance(contact_data, list):\n",
    "            for item in contact_data:\n",
    "                # Check if 'organisation' and 'en' keys exist\n",
    "                if 'organisation' in item and 'en' in item['organisation']:\n",
    "                    return item['organisation']['en']\n",
    "        elif isinstance(contact_data, dict):\n",
    "            # If the data is a dictionary, extract 'organisation' in 'en' directly\n",
    "            return contact_data.get('organisation', {}).get('en', None)\n",
    "    except json.JSONDecodeError:\n",
    "        # Handle cases where the contact string is not valid JSON\n",
    "        return None\n",
    "    except Exception as e:\n",
    "        # Catch-all for any other unexpected errors\n",
    "        return f\"Error: {str(e)}\"\n",
    "\n",
    "# Text preprocess\n",
    "def preprocess_records_into_text(df):\n",
    "    selected_columns = ['features_properties_title_en','features_properties_description_en','features_properties_keywords_en']\n",
    "    df = df[selected_columns]\n",
    "    return df.apply(lambda x: f\"{x['features_properties_title_en']}\\n{x['features_properties_description_en']}\\nkeywords:{x['features_properties_keywords_en']}\",axis=1 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b34c78b4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>features_type</th>\n",
       "      <th>features_geometry_type</th>\n",
       "      <th>features_geometry_coordinates</th>\n",
       "      <th>features_properties_id</th>\n",
       "      <th>features_properties_title_en</th>\n",
       "      <th>features_properties_title_fr</th>\n",
       "      <th>features_properties_description_en</th>\n",
       "      <th>features_properties_description_fr</th>\n",
       "      <th>features_properties_keywords_en</th>\n",
       "      <th>features_properties_keywords_fr</th>\n",
       "      <th>...</th>\n",
       "      <th>features_properties_distributor</th>\n",
       "      <th>features_properties_options</th>\n",
       "      <th>features_properties_temporalExtent_end_@indeterminatePosition</th>\n",
       "      <th>features_properties_temporalExtent_end_#text</th>\n",
       "      <th>features_properties_eoCollection</th>\n",
       "      <th>features_properties_eoFilters</th>\n",
       "      <th>features_properties_sourceSystemName</th>\n",
       "      <th>features_properties_plugins</th>\n",
       "      <th>features_popularity</th>\n",
       "      <th>features_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Feature</td>\n",
       "      <td>Polygon</td>\n",
       "      <td>[[[-143, 39.05], [-47, 39.05], [-47, 85], [-14...</td>\n",
       "      <td>d3881c4c-650d-4070-bf9b-1e00aabf0a1d</td>\n",
       "      <td>Canadian Hydrographic Service Non-Navigational...</td>\n",
       "      <td>Données bathymétriques non navigationnelles (N...</td>\n",
       "      <td>**CHS NONNA data has been updated: April 21, 2...</td>\n",
       "      <td>**Les données NONNA du Service hydrographique ...</td>\n",
       "      <td>Bathymetry, Depth, Hydrography</td>\n",
       "      <td>Bathymétrie, les profondeurs des fonds marins,...</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"individual\": \"null\", \"position\": {\"en\": \"nu...</td>\n",
       "      <td>[{\"url\": \"https://data.chs-shc.ca/\", \"protocol...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>cgp</td>\n",
       "      <td>[]</td>\n",
       "      <td>3929</td>\n",
       "      <td>[{\"sim\": \"sim1\", \"features_properties_id\": \"ff...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Feature</td>\n",
       "      <td>Polygon</td>\n",
       "      <td>[[[-141.0027151, 41.7], [-52.6, 41.7], [-52.6,...</td>\n",
       "      <td>3d282116-e556-400c-9306-ca1a3cada77f</td>\n",
       "      <td>National Road Network - NRN - GeoBase Series</td>\n",
       "      <td>Réseau routier national - RRN - Série GéoBase</td>\n",
       "      <td>Notice - Format decommissioning\\n\\nGML (Geogra...</td>\n",
       "      <td>Avis - Changement aux formats offerts\\n\\nLes f...</td>\n",
       "      <td>Canada, Geographic Infrastructure, NRN, Nation...</td>\n",
       "      <td>Canada, Infrastructure géographique, RRN, Rése...</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"individual\": \"null\", \"position\": {\"en\": \"nu...</td>\n",
       "      <td>[{\"url\": \"https://geo.statcan.gc.ca/geo_wa/ser...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>cgp</td>\n",
       "      <td>[]</td>\n",
       "      <td>1991</td>\n",
       "      <td>[{\"sim\": \"sim1\", \"features_properties_id\": \"24...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Feature</td>\n",
       "      <td>Polygon</td>\n",
       "      <td>[[[-141.003, 41.6755], [-52.6174, 41.6755], [-...</td>\n",
       "      <td>b6567c5c-8339-4055-99fa-63f92114d9e4</td>\n",
       "      <td>First Nations Location</td>\n",
       "      <td>Localisation des Premières Nations</td>\n",
       "      <td>The First Nations geographic location dataset ...</td>\n",
       "      <td>Le jeu de données des Premières Nations contie...</td>\n",
       "      <td>First Nation, Band, Aboriginal, Indian and Nor...</td>\n",
       "      <td>Première Nation, bande, autochtone, Affaires i...</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"individual\": \"null\", \"position\": {\"en\": \"nu...</td>\n",
       "      <td>[{\"url\": \"https://data.aadnc-aandc.gc.ca/geoma...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>cgp</td>\n",
       "      <td>[]</td>\n",
       "      <td>1881</td>\n",
       "      <td>[{\"sim\": \"sim1\", \"features_properties_id\": \"f4...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Feature</td>\n",
       "      <td>Polygon</td>\n",
       "      <td>[[[-141.003, 41.6755], [-52.6174, 41.6755], [-...</td>\n",
       "      <td>522b07b9-78e2-4819-b736-ad9208eb1067</td>\n",
       "      <td>Aboriginal Lands of Canada Legislative Boundaries</td>\n",
       "      <td>Limites législatives des terres autochtones du...</td>\n",
       "      <td>The Aboriginal Lands of Canada Legislative Bou...</td>\n",
       "      <td>Le service web des limites législatives des te...</td>\n",
       "      <td>Canada Lands, Indian reserves, Land management...</td>\n",
       "      <td>Terres du Canada, Réserves indiennes, Gestion ...</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"individual\": \"null\", \"position\": {\"en\": \"nu...</td>\n",
       "      <td>[{\"url\": \"https://proxyinternet.nrcan-rncan.gc...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>cgp</td>\n",
       "      <td>[]</td>\n",
       "      <td>1832</td>\n",
       "      <td>[{\"sim\": \"sim1\", \"features_properties_id\": \"65...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feature</td>\n",
       "      <td>Polygon</td>\n",
       "      <td>[[[-139.88850063, 42.16046612], [-50.94318813,...</td>\n",
       "      <td>db177a8c-5d7d-49eb-8290-31e6a45d786c</td>\n",
       "      <td>Critical Habitat of Species at Risk</td>\n",
       "      <td>L'habitat essentiel désigné en vertu de la Loi...</td>\n",
       "      <td>The Species at Risk (SAR) Program is responsib...</td>\n",
       "      <td>Le Programme des espèces en péril consiste à r...</td>\n",
       "      <td>Species at risk, Canada, Diadromous fish, Sea ...</td>\n",
       "      <td>Espèces en péril, Canada, Les poissons diadrom...</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"individual\": \"null\", \"position\": {\"en\": \"nu...</td>\n",
       "      <td>[{\"url\": \"https://pacgis01.dfo-mpo.gc.ca/FGPPu...</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>cgp</td>\n",
       "      <td>[]</td>\n",
       "      <td>1711</td>\n",
       "      <td>[{\"sim\": \"sim1\", \"features_properties_id\": \"e0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 71 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  features_type features_geometry_type  \\\n",
       "0       Feature                Polygon   \n",
       "1       Feature                Polygon   \n",
       "2       Feature                Polygon   \n",
       "3       Feature                Polygon   \n",
       "4       Feature                Polygon   \n",
       "\n",
       "                       features_geometry_coordinates  \\\n",
       "0  [[[-143, 39.05], [-47, 39.05], [-47, 85], [-14...   \n",
       "1  [[[-141.0027151, 41.7], [-52.6, 41.7], [-52.6,...   \n",
       "2  [[[-141.003, 41.6755], [-52.6174, 41.6755], [-...   \n",
       "3  [[[-141.003, 41.6755], [-52.6174, 41.6755], [-...   \n",
       "4  [[[-139.88850063, 42.16046612], [-50.94318813,...   \n",
       "\n",
       "                 features_properties_id  \\\n",
       "0  d3881c4c-650d-4070-bf9b-1e00aabf0a1d   \n",
       "1  3d282116-e556-400c-9306-ca1a3cada77f   \n",
       "2  b6567c5c-8339-4055-99fa-63f92114d9e4   \n",
       "3  522b07b9-78e2-4819-b736-ad9208eb1067   \n",
       "4  db177a8c-5d7d-49eb-8290-31e6a45d786c   \n",
       "\n",
       "                        features_properties_title_en  \\\n",
       "0  Canadian Hydrographic Service Non-Navigational...   \n",
       "1       National Road Network - NRN - GeoBase Series   \n",
       "2                             First Nations Location   \n",
       "3  Aboriginal Lands of Canada Legislative Boundaries   \n",
       "4                Critical Habitat of Species at Risk   \n",
       "\n",
       "                        features_properties_title_fr  \\\n",
       "0  Données bathymétriques non navigationnelles (N...   \n",
       "1      Réseau routier national - RRN - Série GéoBase   \n",
       "2                 Localisation des Premières Nations   \n",
       "3  Limites législatives des terres autochtones du...   \n",
       "4  L'habitat essentiel désigné en vertu de la Loi...   \n",
       "\n",
       "                  features_properties_description_en  \\\n",
       "0  **CHS NONNA data has been updated: April 21, 2...   \n",
       "1  Notice - Format decommissioning\\n\\nGML (Geogra...   \n",
       "2  The First Nations geographic location dataset ...   \n",
       "3  The Aboriginal Lands of Canada Legislative Bou...   \n",
       "4  The Species at Risk (SAR) Program is responsib...   \n",
       "\n",
       "                  features_properties_description_fr  \\\n",
       "0  **Les données NONNA du Service hydrographique ...   \n",
       "1  Avis - Changement aux formats offerts\\n\\nLes f...   \n",
       "2  Le jeu de données des Premières Nations contie...   \n",
       "3  Le service web des limites législatives des te...   \n",
       "4  Le Programme des espèces en péril consiste à r...   \n",
       "\n",
       "                     features_properties_keywords_en  \\\n",
       "0                     Bathymetry, Depth, Hydrography   \n",
       "1  Canada, Geographic Infrastructure, NRN, Nation...   \n",
       "2  First Nation, Band, Aboriginal, Indian and Nor...   \n",
       "3  Canada Lands, Indian reserves, Land management...   \n",
       "4  Species at risk, Canada, Diadromous fish, Sea ...   \n",
       "\n",
       "                     features_properties_keywords_fr  ...  \\\n",
       "0  Bathymétrie, les profondeurs des fonds marins,...  ...   \n",
       "1  Canada, Infrastructure géographique, RRN, Rése...  ...   \n",
       "2  Première Nation, bande, autochtone, Affaires i...  ...   \n",
       "3  Terres du Canada, Réserves indiennes, Gestion ...  ...   \n",
       "4  Espèces en péril, Canada, Les poissons diadrom...  ...   \n",
       "\n",
       "                     features_properties_distributor  \\\n",
       "0  [{\"individual\": \"null\", \"position\": {\"en\": \"nu...   \n",
       "1  [{\"individual\": \"null\", \"position\": {\"en\": \"nu...   \n",
       "2  [{\"individual\": \"null\", \"position\": {\"en\": \"nu...   \n",
       "3  [{\"individual\": \"null\", \"position\": {\"en\": \"nu...   \n",
       "4  [{\"individual\": \"null\", \"position\": {\"en\": \"nu...   \n",
       "\n",
       "                         features_properties_options  \\\n",
       "0  [{\"url\": \"https://data.chs-shc.ca/\", \"protocol...   \n",
       "1  [{\"url\": \"https://geo.statcan.gc.ca/geo_wa/ser...   \n",
       "2  [{\"url\": \"https://data.aadnc-aandc.gc.ca/geoma...   \n",
       "3  [{\"url\": \"https://proxyinternet.nrcan-rncan.gc...   \n",
       "4  [{\"url\": \"https://pacgis01.dfo-mpo.gc.ca/FGPPu...   \n",
       "\n",
       "  features_properties_temporalExtent_end_@indeterminatePosition  \\\n",
       "0                                               <NA>              \n",
       "1                                               <NA>              \n",
       "2                                               <NA>              \n",
       "3                                               <NA>              \n",
       "4                                               <NA>              \n",
       "\n",
       "  features_properties_temporalExtent_end_#text  \\\n",
       "0                                         <NA>   \n",
       "1                                         <NA>   \n",
       "2                                         <NA>   \n",
       "3                                         <NA>   \n",
       "4                                         <NA>   \n",
       "\n",
       "  features_properties_eoCollection features_properties_eoFilters  \\\n",
       "0                             <NA>                            []   \n",
       "1                             <NA>                            []   \n",
       "2                             <NA>                            []   \n",
       "3                             <NA>                            []   \n",
       "4                             <NA>                            []   \n",
       "\n",
       "  features_properties_sourceSystemName features_properties_plugins  \\\n",
       "0                                  cgp                          []   \n",
       "1                                  cgp                          []   \n",
       "2                                  cgp                          []   \n",
       "3                                  cgp                          []   \n",
       "4                                  cgp                          []   \n",
       "\n",
       "  features_popularity                                features_similarity  \n",
       "0                3929  [{\"sim\": \"sim1\", \"features_properties_id\": \"ff...  \n",
       "1                1991  [{\"sim\": \"sim1\", \"features_properties_id\": \"24...  \n",
       "2                1881  [{\"sim\": \"sim1\", \"features_properties_id\": \"f4...  \n",
       "3                1832  [{\"sim\": \"sim1\", \"features_properties_id\": \"65...  \n",
       "4                1711  [{\"sim\": \"sim1\", \"features_properties_id\": \"e0...  \n",
       "\n",
       "[5 rows x 71 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#1) Step1: Load the data \n",
    "df_parquet = read_parquet_from_s3_as_df('ca-central-1', 'webpresence-geocore-geojson-to-parquet-stage', 'records.parquet')\n",
    "df_sentinel1 = read_parquet_from_s3_as_df('ca-central-1', 'webpresence-geocore-geojson-to-parquet-stage', 'sentinel1.parquet')\n",
    "df = pd.concat([df_parquet, df_sentinel1], ignore_index=True)\n",
    "df.head()\n",
    "#df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd1b73b",
   "metadata": {},
   "source": [
    "Subset to columns that are required in the app.geo.ca [api response](https://geocore.api.geo.ca/geo?north=81.77364370720657&east=360&south=-8.407168163601076&west=-359.6484375&keyword=&lang=en&min=1&max=10&sort=popularity-desc). \n",
    "##### Note, we are focus on the english search at the moment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "04376a3a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_12036/538677574.py:14: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_en['organisation_en'] = df_en['features_properties_contact'].apply(extract_organisation_en)\n",
      "/tmp/ipykernel_12036/538677574.py:19: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_en[columns_to_replace] = df_en[columns_to_replace].replace(values_to_replace)\n",
      "/tmp/ipykernel_12036/538677574.py:21: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_en['temporalExtent'] = df_en.apply(lambda row: {'begin': row['features_properties_temporalExtent_begin'], 'end': row['features_properties_temporalExtent_end']}, axis=1)\n"
     ]
    }
   ],
   "source": [
    "#2) Step2: Clean the data  \n",
    "col_names_list = [\n",
    "    'features_properties_id','features_geometry_coordinates','features_properties_title_en',\n",
    "    'features_properties_description_en','features_properties_date_published_date',\n",
    "    'features_properties_keywords_en','features_properties_options','features_properties_contact',\n",
    "    'features_properties_topicCategory','features_properties_date_created_date',\n",
    "    'features_properties_spatialRepresentation','features_properties_type',\n",
    "    'features_properties_temporalExtent_begin','features_properties_temporalExtent_end',\n",
    "    'features_properties_graphicOverview','features_properties_language','features_popularity',\n",
    "    'features_properties_sourceSystemName','features_properties_eoCollection',\n",
    "    'features_properties_eoFilters'\n",
    "]\n",
    "df_en = df[col_names_list]\n",
    "df_en['organisation_en'] = df_en['features_properties_contact'].apply(extract_organisation_en)\n",
    "\n",
    "# Create a new column 'temporalExtent' as a dictionary of {'begin': ..., 'end': ...}\n",
    "values_to_replace = {'Present': None, 'Not Available; Indisponible': None}\n",
    "columns_to_replace = ['features_properties_temporalExtent_begin', 'features_properties_temporalExtent_end']\n",
    "df_en[columns_to_replace] = df_en[columns_to_replace].replace(values_to_replace)\n",
    "\n",
    "df_en['temporalExtent'] = df_en.apply(lambda row: {'begin': row['features_properties_temporalExtent_begin'], 'end': row['features_properties_temporalExtent_end']}, axis=1)\n",
    "df_en = df_en.drop(columns =['features_properties_temporalExtent_begin', 'features_properties_temporalExtent_end'])\n",
    "\n",
    "values_to_replace = {'Not Available; Indisponible': None} # modifies dates to acceptable values\n",
    "columns_to_replace = ['features_properties_date_published_date', 'features_properties_date_created_date']\n",
    "df_en[columns_to_replace] = df_en[columns_to_replace].replace(values_to_replace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "717b2fd9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'str'>\n",
      "National Road Network - NRN - GeoBase Series\n",
      "Notice - Format decommissioning\\n\\nGML (Geography Markup Language) and KML (Keyhole Markup Language) distribution formats will no longer be included in NRN releases produced after April 1, 2022. After this date, NRN versions will still be available in GeoPackage and ESRI Shapefile formats. This change does not impact the currently available NRN releases on this portal.\\n\\nThe NRN product is distributed in the form of thirteen provincial or territorial datasets and consists of two linear entities (Road Segment and Ferry Connection Segment) and three punctual entities (Junction, Blocked Passage, Toll Point) with which is associated a series of descriptive attributes such as, among others: First House Number, Last House Number, Street Name Body, Place Name, Functional Road Class, Pavement Status, Number Of Lanes, Structure Type, Route Number, Route Name, Exit Number. The development of the NRN was realized by means of individual meetings and national workshops with interested data providers from the federal, provincial, territorial and municipal governments. In 2005, the NRN edition 2.0 was alternately adopted by members from the Inter-Agency Committee on Geomatics (IACG) and the Canadian Council on Geomatics (CCOG). The NRN content largely conforms to the ISO 14825 from ISO/TC 204.\n",
      "keywords:Canada, Geographic Infrastructure, NRN, National Road Network, Transport, Road transport, Infrastructure, Road maps, Road networks\n"
     ]
    }
   ],
   "source": [
    "#3) Step 3: Preprocess text \n",
    "df_en['text'] = preprocess_records_into_text(df_en)\n",
    "\n",
    "df_en.describe()\n",
    "print(type(df_en['text'].head(6)[1]))\n",
    "print(df_en['text'].head(6)[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "078dd906",
   "metadata": {},
   "source": [
    "### (optional) Text Preprocess using NLTK  \n",
    "\n",
    "Create a new column that concadenate the selected columns: features_properties_title_en, features_properties_description_en,features_properties_keywords_en, and apply the following preprocesing before tokenization:\n",
    "- convert to lower case \n",
    "- remove stopwords and punctuation\n",
    "- remove apostrophe\n",
    "- stemming\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2258c36f-b4ba-4e54-aec7-a8b03143142c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import nltk\n",
    "# from nltk.corpus import stopwords          # module for stop words that come with NLTK\n",
    "# from nltk.stem import PorterStemmer        # module for stemming\n",
    "# from nltk.tokenize import word_tokenize   # module for tokenizing strings \n",
    "# import string\n",
    "# # Download necessary NLTK resources\n",
    "# nltk.download('punkt')\n",
    "# nltk.download('stopwords')\n",
    "\n",
    "\n",
    "# df_en['metadata_en'] = df_en['features_properties_title_en'] + ' ' + df_en['features_properties_description_en'] + ' ' + df_en['features_properties_keywords_en'] \n",
    "# if df_en['metadata_en'].isnull().any():\n",
    "#     df_en['metadata_en'] = df_en['metadata_en'].fillna('')\n",
    "\n",
    "# # Function to clean text\n",
    "# def clean_text(text):\n",
    "#     \"\"\"\n",
    "#     text: raw tex input, a string of text, or a list of string\n",
    "#     output: preprocess text in string format \n",
    "#     \"\"\"\n",
    "#     # Set of stopwords\n",
    "#     stop_words = set(stopwords.words('english'))\n",
    "#     # Initialize the Porter Stemmer\n",
    "#     stemmer = PorterStemmer()\n",
    "    \n",
    "#     # Convert text to lowercase\n",
    "#     text = text.lower()\n",
    "#     # Remove punctuation\n",
    "#     text = text.translate(str.maketrans('', '', string.punctuation.replace(\"'\", \"\")))  # Keep apostrophe\n",
    "#     # Remove apostrophes\n",
    "#     text = text.replace(\"'\", \"\")\n",
    "#     # Tokenize text\n",
    "#     word_tokens = word_tokenize(text)\n",
    "#     # Remove stopwords and stem\n",
    "#     filtered_text = [stemmer.stem(word) for word in word_tokens if word not in stop_words]\n",
    "#     return \" \".join(filtered_text)\n",
    "\n",
    "# df_en['processed_metadata_en'] = df_en['metadata_en'].apply(clean_text)\n",
    "# # Show the processed column\n",
    "# print(df_en.loc[1:3, ['metadata_en', 'processed_metadata_en']])\n",
    "# df_en.head(4)\n",
    "\n",
    "# tqdm.pandas()\n",
    "# df_en['vector'] = df_en[\"processed_metadata_en\"].progress_apply(sentence_to_vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b81fb697-37f2-4c9c-8e11-cb4396e38cf2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 61302/61302 [49:15<00:00, 20.74it/s]  \n"
     ]
    }
   ],
   "source": [
    "# Step 4: Embedding text \n",
    "tqdm.pandas()\n",
    "df_en['vector'] = df_en[\"text\"].progress_apply(sentence_to_vector)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d0a3d02f-7405-4502-9608-3ba66a3a0886",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dimension of the vector is 384\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>features_properties_id</th>\n",
       "      <th>features_geometry_coordinates</th>\n",
       "      <th>features_properties_title_en</th>\n",
       "      <th>features_properties_description_en</th>\n",
       "      <th>features_properties_date_published_date</th>\n",
       "      <th>features_properties_keywords_en</th>\n",
       "      <th>features_properties_options</th>\n",
       "      <th>features_properties_contact</th>\n",
       "      <th>features_properties_topicCategory</th>\n",
       "      <th>features_properties_date_created_date</th>\n",
       "      <th>...</th>\n",
       "      <th>features_properties_graphicOverview</th>\n",
       "      <th>features_properties_language</th>\n",
       "      <th>features_popularity</th>\n",
       "      <th>features_properties_sourceSystemName</th>\n",
       "      <th>features_properties_eoCollection</th>\n",
       "      <th>features_properties_eoFilters</th>\n",
       "      <th>organisation_en</th>\n",
       "      <th>temporalExtent</th>\n",
       "      <th>text</th>\n",
       "      <th>vector</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>d3881c4c-650d-4070-bf9b-1e00aabf0a1d</td>\n",
       "      <td>[[[-143, 39.05], [-47, 39.05], [-47, 85], [-14...</td>\n",
       "      <td>Canadian Hydrographic Service Non-Navigational...</td>\n",
       "      <td>**CHS NONNA data has been updated: April 21, 2...</td>\n",
       "      <td>2018-10-11</td>\n",
       "      <td>Bathymetry, Depth, Hydrography</td>\n",
       "      <td>[{\"url\": \"https://data.chs-shc.ca/\", \"protocol...</td>\n",
       "      <td>[{\"individual\": \"null\", \"position\": {\"en\": \"nu...</td>\n",
       "      <td>oceans, inlandWaters</td>\n",
       "      <td>2018-10-01</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"overviewFileName\": \"https://pacgis01.dfo-mp...</td>\n",
       "      <td>eng; CAN</td>\n",
       "      <td>3929</td>\n",
       "      <td>cgp</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>Government of Canada; Fisheries and Oceans Can...</td>\n",
       "      <td>{'begin': &lt;NA&gt;, 'end': &lt;NA&gt;}</td>\n",
       "      <td>Canadian Hydrographic Service Non-Navigational...</td>\n",
       "      <td>[-0.03204740211367607, 0.007585172541439533, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3d282116-e556-400c-9306-ca1a3cada77f</td>\n",
       "      <td>[[[-141.0027151, 41.7], [-52.6, 41.7], [-52.6,...</td>\n",
       "      <td>National Road Network - NRN - GeoBase Series</td>\n",
       "      <td>Notice - Format decommissioning\\n\\nGML (Geogra...</td>\n",
       "      <td>2015</td>\n",
       "      <td>Canada, Geographic Infrastructure, NRN, Nation...</td>\n",
       "      <td>[{\"url\": \"https://geo.statcan.gc.ca/geo_wa/ser...</td>\n",
       "      <td>[{\"individual\": \"null\", \"position\": {\"en\": \"nu...</td>\n",
       "      <td>transportation</td>\n",
       "      <td>2010</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"overviewFileName\": \"http://ftp.geogratis.gc...</td>\n",
       "      <td>eng; CAN</td>\n",
       "      <td>1991</td>\n",
       "      <td>cgp</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>Government of Canada; Statistics Canada</td>\n",
       "      <td>{'begin': '1979-07', 'end': '2020-05'}</td>\n",
       "      <td>National Road Network - NRN - GeoBase Series\\n...</td>\n",
       "      <td>[-0.03426723554730415, -0.06323028355836868, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b6567c5c-8339-4055-99fa-63f92114d9e4</td>\n",
       "      <td>[[[-141.003, 41.6755], [-52.6174, 41.6755], [-...</td>\n",
       "      <td>First Nations Location</td>\n",
       "      <td>The First Nations geographic location dataset ...</td>\n",
       "      <td>2015-05-01</td>\n",
       "      <td>First Nation, Band, Aboriginal, Indian and Nor...</td>\n",
       "      <td>[{\"url\": \"https://data.aadnc-aandc.gc.ca/geoma...</td>\n",
       "      <td>[{\"individual\": \"null\", \"position\": {\"en\": \"Me...</td>\n",
       "      <td>location, society</td>\n",
       "      <td>2007-06-01</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"overviewFileName\": \"https://data.aadnc-aand...</td>\n",
       "      <td>eng; CAN</td>\n",
       "      <td>1881</td>\n",
       "      <td>cgp</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>Government of Canada;Indigenous Services Canad...</td>\n",
       "      <td>{'begin': '2007-06-01', 'end': &lt;NA&gt;}</td>\n",
       "      <td>First Nations Location\\nThe First Nations geog...</td>\n",
       "      <td>[0.029996875673532486, -0.042383823543787, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>522b07b9-78e2-4819-b736-ad9208eb1067</td>\n",
       "      <td>[[[-141.003, 41.6755], [-52.6174, 41.6755], [-...</td>\n",
       "      <td>Aboriginal Lands of Canada Legislative Boundaries</td>\n",
       "      <td>The Aboriginal Lands of Canada Legislative Bou...</td>\n",
       "      <td>2017-07-28</td>\n",
       "      <td>Canada Lands, Indian reserves, Land management...</td>\n",
       "      <td>[{\"url\": \"https://proxyinternet.nrcan-rncan.gc...</td>\n",
       "      <td>[{\"individual\": \"null\", \"position\": {\"en\": \"nu...</td>\n",
       "      <td>boundaries</td>\n",
       "      <td>2016-02-03</td>\n",
       "      <td>...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eng; CAN</td>\n",
       "      <td>1832</td>\n",
       "      <td>cgp</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>[]</td>\n",
       "      <td>Government of Canada; Natural Resources Canada...</td>\n",
       "      <td>{'begin': '2004-04-02', 'end': &lt;NA&gt;}</td>\n",
       "      <td>Aboriginal Lands of Canada Legislative Boundar...</td>\n",
       "      <td>[0.037921637296676636, -0.005010603461414576, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 features_properties_id  \\\n",
       "0  d3881c4c-650d-4070-bf9b-1e00aabf0a1d   \n",
       "1  3d282116-e556-400c-9306-ca1a3cada77f   \n",
       "2  b6567c5c-8339-4055-99fa-63f92114d9e4   \n",
       "3  522b07b9-78e2-4819-b736-ad9208eb1067   \n",
       "\n",
       "                       features_geometry_coordinates  \\\n",
       "0  [[[-143, 39.05], [-47, 39.05], [-47, 85], [-14...   \n",
       "1  [[[-141.0027151, 41.7], [-52.6, 41.7], [-52.6,...   \n",
       "2  [[[-141.003, 41.6755], [-52.6174, 41.6755], [-...   \n",
       "3  [[[-141.003, 41.6755], [-52.6174, 41.6755], [-...   \n",
       "\n",
       "                        features_properties_title_en  \\\n",
       "0  Canadian Hydrographic Service Non-Navigational...   \n",
       "1       National Road Network - NRN - GeoBase Series   \n",
       "2                             First Nations Location   \n",
       "3  Aboriginal Lands of Canada Legislative Boundaries   \n",
       "\n",
       "                  features_properties_description_en  \\\n",
       "0  **CHS NONNA data has been updated: April 21, 2...   \n",
       "1  Notice - Format decommissioning\\n\\nGML (Geogra...   \n",
       "2  The First Nations geographic location dataset ...   \n",
       "3  The Aboriginal Lands of Canada Legislative Bou...   \n",
       "\n",
       "  features_properties_date_published_date  \\\n",
       "0                              2018-10-11   \n",
       "1                                    2015   \n",
       "2                              2015-05-01   \n",
       "3                              2017-07-28   \n",
       "\n",
       "                     features_properties_keywords_en  \\\n",
       "0                     Bathymetry, Depth, Hydrography   \n",
       "1  Canada, Geographic Infrastructure, NRN, Nation...   \n",
       "2  First Nation, Band, Aboriginal, Indian and Nor...   \n",
       "3  Canada Lands, Indian reserves, Land management...   \n",
       "\n",
       "                         features_properties_options  \\\n",
       "0  [{\"url\": \"https://data.chs-shc.ca/\", \"protocol...   \n",
       "1  [{\"url\": \"https://geo.statcan.gc.ca/geo_wa/ser...   \n",
       "2  [{\"url\": \"https://data.aadnc-aandc.gc.ca/geoma...   \n",
       "3  [{\"url\": \"https://proxyinternet.nrcan-rncan.gc...   \n",
       "\n",
       "                         features_properties_contact  \\\n",
       "0  [{\"individual\": \"null\", \"position\": {\"en\": \"nu...   \n",
       "1  [{\"individual\": \"null\", \"position\": {\"en\": \"nu...   \n",
       "2  [{\"individual\": \"null\", \"position\": {\"en\": \"Me...   \n",
       "3  [{\"individual\": \"null\", \"position\": {\"en\": \"nu...   \n",
       "\n",
       "  features_properties_topicCategory features_properties_date_created_date  \\\n",
       "0              oceans, inlandWaters                            2018-10-01   \n",
       "1                    transportation                                  2010   \n",
       "2                 location, society                            2007-06-01   \n",
       "3                        boundaries                            2016-02-03   \n",
       "\n",
       "   ...                features_properties_graphicOverview  \\\n",
       "0  ...  [{\"overviewFileName\": \"https://pacgis01.dfo-mp...   \n",
       "1  ...  [{\"overviewFileName\": \"http://ftp.geogratis.gc...   \n",
       "2  ...  [{\"overviewFileName\": \"https://data.aadnc-aand...   \n",
       "3  ...                                                 []   \n",
       "\n",
       "  features_properties_language features_popularity  \\\n",
       "0                     eng; CAN                3929   \n",
       "1                     eng; CAN                1991   \n",
       "2                     eng; CAN                1881   \n",
       "3                     eng; CAN                1832   \n",
       "\n",
       "  features_properties_sourceSystemName features_properties_eoCollection  \\\n",
       "0                                  cgp                             <NA>   \n",
       "1                                  cgp                             <NA>   \n",
       "2                                  cgp                             <NA>   \n",
       "3                                  cgp                             <NA>   \n",
       "\n",
       "  features_properties_eoFilters  \\\n",
       "0                            []   \n",
       "1                            []   \n",
       "2                            []   \n",
       "3                            []   \n",
       "\n",
       "                                     organisation_en  \\\n",
       "0  Government of Canada; Fisheries and Oceans Can...   \n",
       "1            Government of Canada; Statistics Canada   \n",
       "2  Government of Canada;Indigenous Services Canad...   \n",
       "3  Government of Canada; Natural Resources Canada...   \n",
       "\n",
       "                           temporalExtent  \\\n",
       "0            {'begin': <NA>, 'end': <NA>}   \n",
       "1  {'begin': '1979-07', 'end': '2020-05'}   \n",
       "2    {'begin': '2007-06-01', 'end': <NA>}   \n",
       "3    {'begin': '2004-04-02', 'end': <NA>}   \n",
       "\n",
       "                                                text  \\\n",
       "0  Canadian Hydrographic Service Non-Navigational...   \n",
       "1  National Road Network - NRN - GeoBase Series\\n...   \n",
       "2  First Nations Location\\nThe First Nations geog...   \n",
       "3  Aboriginal Lands of Canada Legislative Boundar...   \n",
       "\n",
       "                                              vector  \n",
       "0  [-0.03204740211367607, 0.007585172541439533, 0...  \n",
       "1  [-0.03426723554730415, -0.06323028355836868, 0...  \n",
       "2  [0.029996875673532486, -0.042383823543787, -0....  \n",
       "3  [0.037921637296676636, -0.005010603461414576, ...  \n",
       "\n",
       "[4 rows x 22 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(f'The dimension of the vector is {len(df_en.loc[1, \"vector\"])}')\n",
    "df_en.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3e0edff0-53ab-4869-a50d-a96649f6fa49",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading semantic_search_embeddings-pretrain.parquet to webpresence-nlp-data-preprocessing-stage as parquet file\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 5 Upload the embeddings as a parquet file to S3 bucket \n",
    "upload_df_to_s3_as_parquet(df=df_en, bucket_name='webpresence-nlp-data-preprocessing-stage', file_key='semantic_search_embeddings-pretrain.parquet') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7713302",
   "metadata": {},
   "source": [
    "### 6. Create an OpenSearch cluster connection.\n",
    "Next, we'll use Python API to set up connection with OpenSearch Cluster.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "145a4127-e439-4041-b7ff-430a7fb6feca",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from opensearch import get_awsauth_from_secret, create_opensearch_connection, delete_aos_index_if_exists, load_data_to_opensearch_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2c30507c-8c39-4221-834d-e8f905a861fa",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "def read_parquet_from_s3_as_df(region, s3_bucket, s3_key):\n",
    "    \"\"\"\n",
    "    Load a Parquet file from an S3 bucket into a pandas DataFrame.\n",
    "\n",
    "    Parameters:\n",
    "    - region: AWS region where the S3 bucket is located.\n",
    "    - s3_bucket: Name of the S3 bucket.\n",
    "    - s3_key: Key (path) to the Parquet file within the S3 bucket.\n",
    "\n",
    "    Returns:\n",
    "    - df: pandas DataFrame containing the data from the Parquet file.\n",
    "    \"\"\"\n",
    "\n",
    "    # Setup AWS session and clients\n",
    "    session = boto3.Session(region_name=region)\n",
    "    s3 = session.resource('s3')\n",
    "\n",
    "    # Load the Parquet file as a pandas DataFrame\n",
    "    object = s3.Object(s3_bucket, s3_key)\n",
    "    body = object.get()['Body'].read()\n",
    "    df = pd.read_parquet(io.BytesIO(body))\n",
    "    return df\n",
    "\n",
    "#Optional: read the embedding data from the S3 bucket \n",
    "# import pandas as pd \n",
    "# import io\n",
    "# df_en = read_parquet_from_s3_as_df('ca-central-1', 'webpresence-nlp-data-preprocessing-stage', 'semantic_search_embeddings-mpnet-mpf.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4d152847-1dbe-4d5b-8987-6a07229e00e0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_test = df_en.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "d8471fe1-c9d6-4053-a9f5-9e738e39ba16",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n",
      "Series has null values: False\n",
      "<class 'list'>\n",
      "<class 'list'>\n",
      "List of lists has null values: False\n"
     ]
    }
   ],
   "source": [
    "vector = df_en['vector']\n",
    "print(type(vector))\n",
    "# Check for null values\n",
    "has_null = vector.isnull().any()\n",
    "print(f\"Series has null values: {has_null}\")\n",
    "\n",
    "json_en = df_en.to_dict(\"records\")\n",
    "print(type(json_en))\n",
    "#print(json_en[0])\n",
    "\n",
    "# Extract the 'vector' values\n",
    "vectors = [item['vector'] for item in json_en]\n",
    "print(type(vectors))\n",
    "#print(vectors[0])\n",
    "\n",
    "#heck if there is null values \n",
    "import numpy as np \n",
    "array = np.array(vectors, dtype=object)\n",
    "has_null = np.any(array == None)\n",
    "print(f\"List of lists has null values: {has_null}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21091b5a-ea88-42f3-b544-fb703f1bd138",
   "metadata": {},
   "source": [
    "Under the cloudformation template 'geocore-semantic-search-with-opensearch-stage; Output tab, find the values for region, aos_host, and os_secret_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f644d506",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connection to OpenSearch established: <OpenSearch([{'host': 'search-semantic-search-arieibeskhrn6vn2qd7gf5br7q.ca-central-1.es.amazonaws.com', 'port': 443}])>\n"
     ]
    }
   ],
   "source": [
    "region = \"ca-central-1\"\n",
    "aos_host = \"search-semantic-search-arieibeskhrn6vn2qd7gf5br7q.ca-central-1.es.amazonaws.com\"\n",
    "os_secret_id = \"OpenSearchSecret-geocore-semantic-search-with-opensearch-stage\"\n",
    "\n",
    "awsauth = get_awsauth_from_secret(region, secret_id=os_secret_id)\n",
    "aos_client =create_opensearch_connection(aos_host, awsauth)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ba38706",
   "metadata": {},
   "source": [
    "### 7. Create a index in Amazon Opensearch Service \n",
    "The following index setting is configuring an OpenSearch index to support k-nearest neighbor (KNN) searches with specific characteristics. KNN is a feature that allows for similarity searches, finding the \"nearest\" documents in a high-dimensional space. This setting is crucial for enabling vector-based searches, where vectors represent document features in a multidimensional space.\n",
    "\n",
    "How k-NN search works:K-NN search works by calculating the distance between a query vector and the vectors in the dataset to find the closest matches. OpenSearch stores these vectors in an index and uses specialized algorithms (like HNSW, Hierarchical Navigable Small World graphs) to perform efficient similarity search at scale.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "29f67fac",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "index_name = \"minilm-knn\"\n",
    "knn_index = {\n",
    "    \"settings\": {\n",
    "        \"index.knn\": True, #This enables the k-nearest neighbor (KNN) search capability on the index.\n",
    "        \"index.knn.space_type\": \"cosinesimil\", #cosine similarity \n",
    "        \"analysis\": {\n",
    "          \"analyzer\": {\n",
    "            \"default\": {\n",
    "              \"type\": \"standard\",\n",
    "              \"stopwords\": \"_english_\"\n",
    "            }\n",
    "          }\n",
    "        }\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"vector\": {\n",
    "                \"type\": \"knn_vector\",\n",
    "                \"dimension\": 384,\n",
    "                \"store\": True\n",
    "            },\n",
    "            \"coordinates\":{\n",
    "              \"type\": \"geo_shape\", \n",
    "              \"store\": True \n",
    "            }  \n",
    "        }\n",
    "    }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2ced5690-ccbb-4b55-a4fc-5baef8fa98a2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current indexes: ['minilm-knn', 'minilm-pretrain-knn', '.opensearch-observability', '.plugins-ml-config', '.ql-datasources', '.kibana_1', '.opendistro_security', 'mpnet-mpf-knn']\n",
      "Deleted index: minilm-knn\n",
      "Response: {'acknowledged': True}\n",
      "Indexes after deletion attempt: ['minilm-pretrain-knn', '.opensearch-observability', '.plugins-ml-config', '.ql-datasources', '.kibana_1', '.opendistro_security', 'mpnet-mpf-knn']\n"
     ]
    }
   ],
   "source": [
    "#Delete index if it exists \n",
    "delete_aos_index_if_exists(aos_client, index_to_delete=index_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "39b1e8a0-8fee-43b3-a63c-efa709395459",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'acknowledged': True, 'shards_acknowledged': True, 'index': 'minilm-knn'}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Create a index \n",
    "aos_client.indices.create(index=index_name,body=knn_index,ignore=400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05fdbefe-1dd5-4a63-a96d-33ec2bd5cccd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load data to OpenSearch Index \n",
    "load_data_to_opensearch_index(df_en, aos_client, index_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4ec581fe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Records loaded into the index minilm-knn is 10000.\n"
     ]
    }
   ],
   "source": [
    "res = aos_client.search(index=index_name, body={\"query\": {\"match_all\": {}}})\n",
    "print(f\"Records loaded into the index {index_name} is {res['hits']['total']['value']}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01160d3f",
   "metadata": {},
   "source": [
    "### 8. Test \"Semantic Search\" \n",
    "\n",
    "Now that we have vector in OpenSearch and a vector for our query question, let's perform a KNN search in OpenSearch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "68bf1654",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "minilm-knn\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_id</th>\n",
       "      <th>relevancy_score</th>\n",
       "      <th>title</th>\n",
       "      <th>uuid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>sZ6rWpABNvGO3XAAW7u7</td>\n",
       "      <td>0.653354</td>\n",
       "      <td>Mapping major floods April-May 2017</td>\n",
       "      <td>34085f6d-106a-41af-a29b-53ed6947c249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Z56qWpABNvGO3XAA-rPd</td>\n",
       "      <td>0.643385</td>\n",
       "      <td>Ontario Hydro Network - Watercourses</td>\n",
       "      <td>17bfc1bb-4849-4615-83a4-f7bd1dd21fd2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>lp6qWpABNvGO3XAA_LO4</td>\n",
       "      <td>0.642681</td>\n",
       "      <td>Ontario Integrated Hydrology (OIH) data</td>\n",
       "      <td>ef0c4387-38ce-4adc-b761-f0506b82564e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>eZ6rWpABNvGO3XAAZLyp</td>\n",
       "      <td>0.639701</td>\n",
       "      <td>2022 Events</td>\n",
       "      <td>c5bb7cfa-1eb1-45fb-8d1a-1ca9d0eafec4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b56rWpABNvGO3XAAK7cL</td>\n",
       "      <td>0.638624</td>\n",
       "      <td>Road Point Events</td>\n",
       "      <td>01164c08-450c-4d66-8291-1ba018f2fc1c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3p6qWpABNvGO3XAA_7PK</td>\n",
       "      <td>0.636887</td>\n",
       "      <td>Major river systems in the Far North</td>\n",
       "      <td>e10f6d04-8eff-4ebc-bd47-37d640c443c5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Ep6qWpABNvGO3XAAva5R</td>\n",
       "      <td>0.635798</td>\n",
       "      <td>Lakes and Rivers Database (LCE)</td>\n",
       "      <td>a4a5575d-e8e8-4410-bfc6-18e9361ffd3f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9p6qWpABNvGO3XAAjanz</td>\n",
       "      <td>0.635618</td>\n",
       "      <td>Waterfront course</td>\n",
       "      <td>14be6566-5e91-4553-9049-857bcfc0f7ca</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bZ6rWpABNvGO3XAABrRN</td>\n",
       "      <td>0.635004</td>\n",
       "      <td>Rivers and ditches</td>\n",
       "      <td>c128aff5-325c-4599-ab66-1c9d0b3abc94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Gp6qWpABNvGO3XAA97OP</td>\n",
       "      <td>0.634775</td>\n",
       "      <td>River-Mountain Promenade</td>\n",
       "      <td>71433534-694b-4538-8cd7-86530790ab0c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>7Z6qWpABNvGO3XAAgaje</td>\n",
       "      <td>0.634121</td>\n",
       "      <td>Freshwater Atlas Rivers</td>\n",
       "      <td>f7dac054-efbf-402f-ab62-6fc4b32a619e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>M56qWpABNvGO3XAA1rCX</td>\n",
       "      <td>0.633583</td>\n",
       "      <td>NCC Picnic areas</td>\n",
       "      <td>8962377c-5751-4953-bb85-8362e14e1f64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Kp6rWpABNvGO3XAAGrZa</td>\n",
       "      <td>0.633204</td>\n",
       "      <td>Base Stream and Flow Representation Update</td>\n",
       "      <td>4cb36aa3-eef5-417c-b2c3-a5544435d038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>x56rWpABNvGO3XAAXLu5</td>\n",
       "      <td>0.632720</td>\n",
       "      <td>High and Low Current Zones of the Ottawa and G...</td>\n",
       "      <td>b1e7c9a9-a34a-457c-8671-bae4529749fb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>BJ6rWpABNvGO3XAAGLaT</td>\n",
       "      <td>0.632578</td>\n",
       "      <td>Stream Connectivity Indicator for Alberta</td>\n",
       "      <td>557f2170-ffa5-462e-b653-af18f001a60f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>i56rWpABNvGO3XAAQ7kg</td>\n",
       "      <td>0.630642</td>\n",
       "      <td>Boards of water on the island of Montreal (lak...</td>\n",
       "      <td>ec3fe75c-fdd4-4d99-b8b6-7ebdd69cc7bb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>mZ6qWpABNvGO3XAA_LPU</td>\n",
       "      <td>0.630527</td>\n",
       "      <td>Fish stocking data for recreational purposes</td>\n",
       "      <td>b15d6f41-2a3b-4af6-914c-1094754bbc0b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>np6qWpABNvGO3XAAuK0Y</td>\n",
       "      <td>0.629713</td>\n",
       "      <td>WSA - Water Feature Annotation (50,000)</td>\n",
       "      <td>dae4e2d3-0da0-40c7-8957-7fd74a8c5099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>pJ6qWpABNvGO3XAA_bNI</td>\n",
       "      <td>0.629602</td>\n",
       "      <td>Ontario Hydro Network - Hydrographic Line</td>\n",
       "      <td>a6e17b31-817b-408e-87cf-4cab5b6eb9f2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>tp6rWpABNvGO3XAAW7v6</td>\n",
       "      <td>0.629100</td>\n",
       "      <td>Spring 2019 flood mapping</td>\n",
       "      <td>df99bb67-317c-435b-8f9a-50de5fa9729a</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     _id  relevancy_score  \\\n",
       "0   sZ6rWpABNvGO3XAAW7u7         0.653354   \n",
       "1   Z56qWpABNvGO3XAA-rPd         0.643385   \n",
       "2   lp6qWpABNvGO3XAA_LO4         0.642681   \n",
       "3   eZ6rWpABNvGO3XAAZLyp         0.639701   \n",
       "4   b56rWpABNvGO3XAAK7cL         0.638624   \n",
       "5   3p6qWpABNvGO3XAA_7PK         0.636887   \n",
       "6   Ep6qWpABNvGO3XAAva5R         0.635798   \n",
       "7   9p6qWpABNvGO3XAAjanz         0.635618   \n",
       "8   bZ6rWpABNvGO3XAABrRN         0.635004   \n",
       "9   Gp6qWpABNvGO3XAA97OP         0.634775   \n",
       "10  7Z6qWpABNvGO3XAAgaje         0.634121   \n",
       "11  M56qWpABNvGO3XAA1rCX         0.633583   \n",
       "12  Kp6rWpABNvGO3XAAGrZa         0.633204   \n",
       "13  x56rWpABNvGO3XAAXLu5         0.632720   \n",
       "14  BJ6rWpABNvGO3XAAGLaT         0.632578   \n",
       "15  i56rWpABNvGO3XAAQ7kg         0.630642   \n",
       "16  mZ6qWpABNvGO3XAA_LPU         0.630527   \n",
       "17  np6qWpABNvGO3XAAuK0Y         0.629713   \n",
       "18  pJ6qWpABNvGO3XAA_bNI         0.629602   \n",
       "19  tp6rWpABNvGO3XAAW7v6         0.629100   \n",
       "\n",
       "                                                title  \\\n",
       "0                 Mapping major floods April-May 2017   \n",
       "1                Ontario Hydro Network - Watercourses   \n",
       "2             Ontario Integrated Hydrology (OIH) data   \n",
       "3                                         2022 Events   \n",
       "4                                   Road Point Events   \n",
       "5                Major river systems in the Far North   \n",
       "6                     Lakes and Rivers Database (LCE)   \n",
       "7                                   Waterfront course   \n",
       "8                                  Rivers and ditches   \n",
       "9                            River-Mountain Promenade   \n",
       "10                            Freshwater Atlas Rivers   \n",
       "11                                   NCC Picnic areas   \n",
       "12         Base Stream and Flow Representation Update   \n",
       "13  High and Low Current Zones of the Ottawa and G...   \n",
       "14          Stream Connectivity Indicator for Alberta   \n",
       "15  Boards of water on the island of Montreal (lak...   \n",
       "16       Fish stocking data for recreational purposes   \n",
       "17            WSA - Water Feature Annotation (50,000)   \n",
       "18          Ontario Hydro Network - Hydrographic Line   \n",
       "19                          Spring 2019 flood mapping   \n",
       "\n",
       "                                    uuid  \n",
       "0   34085f6d-106a-41af-a29b-53ed6947c249  \n",
       "1   17bfc1bb-4849-4615-83a4-f7bd1dd21fd2  \n",
       "2   ef0c4387-38ce-4adc-b761-f0506b82564e  \n",
       "3   c5bb7cfa-1eb1-45fb-8d1a-1ca9d0eafec4  \n",
       "4   01164c08-450c-4d66-8291-1ba018f2fc1c  \n",
       "5   e10f6d04-8eff-4ebc-bd47-37d640c443c5  \n",
       "6   a4a5575d-e8e8-4410-bfc6-18e9361ffd3f  \n",
       "7   14be6566-5e91-4553-9049-857bcfc0f7ca  \n",
       "8   c128aff5-325c-4599-ab66-1c9d0b3abc94  \n",
       "9   71433534-694b-4538-8cd7-86530790ab0c  \n",
       "10  f7dac054-efbf-402f-ab62-6fc4b32a619e  \n",
       "11  8962377c-5751-4953-bb85-8362e14e1f64  \n",
       "12  4cb36aa3-eef5-417c-b2c3-a5544435d038  \n",
       "13  b1e7c9a9-a34a-457c-8671-bae4529749fb  \n",
       "14  557f2170-ffa5-462e-b653-af18f001a60f  \n",
       "15  ec3fe75c-fdd4-4d99-b8b6-7ebdd69cc7bb  \n",
       "16  b15d6f41-2a3b-4af6-914c-1094754bbc0b  \n",
       "17  dae4e2d3-0da0-40c7-8957-7fd74a8c5099  \n",
       "18  a6e17b31-817b-408e-87cf-4cab5b6eb9f2  \n",
       "19  df99bb67-317c-435b-8f9a-50de5fa9729a  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "INPUT = \"Riverice events in ottawa\"\n",
    "search_vector = sentence_to_vector(INPUT)\n",
    "print(index_name)\n",
    "\n",
    "query={\n",
    "    \"size\": 20,\n",
    "    \"query\": {\n",
    "        \"knn\": {\n",
    "            \"vector\":{\n",
    "                \"vector\":search_vector,\n",
    "                \"k\":20\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "res = aos_client.search(index=index_name, size=20,body=query,request_timeout=55)\n",
    "query_result=[]\n",
    "for hit in res['hits']['hits']:\n",
    "    row=[hit['_id'],hit['_score'],hit['_source']['title'],hit['_source']['id']]\n",
    "    query_result.append(row)\n",
    "query_result_df = pd.DataFrame(data=query_result,columns=[\"_id\",\"relevancy_score\",\"title\",'uuid'])\n",
    "display(query_result_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29a848bf-8a77-4141-9845-0cc70dd8bcc3",
   "metadata": {},
   "source": [
    "### 9. Test \"Semantic Search\" using the model endpoints\n",
    "\n",
    "Now that we have vector in OpenSearch and a vector for our query question, let's perform a KNN search in OpenSearch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2ce6c041-2009-4311-86d8-e0e5aba9d189",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialize a boto3 client for SageMaker\n",
    "import boto3\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.huggingface.model import HuggingFaceModel\n",
    "\n",
    "# Initialize a boto3 client for SageMaker\n",
    "sagemaker_client = boto3.client('sagemaker', region_name='ca-central-1')  # Specify the AWS region\n",
    "\n",
    "def list_sagemaker_endpoints():\n",
    "    \"\"\"List all SageMaker endpoints\"\"\"\n",
    "    try:\n",
    "        # Get the list of all SageMaker endpoints\n",
    "        response = sagemaker_client.list_endpoints(SortBy='Name')\n",
    "        print(\"Listing SageMaker Endpoints:\")\n",
    "        for endpoint in response['Endpoints']:\n",
    "            print(f\"Endpoint Name: {endpoint['EndpointName']}, Status: {endpoint['EndpointStatus']}\")\n",
    "    except Exception as e:\n",
    "        print(f\"Error listing SageMaker endpoints: {e}\")\n",
    "\n",
    "def invoke_sagemaker_endpoint_ft(endpoint_name, payload):\n",
    "    \"\"\"Invoke a SageMaker endpoint to get predictions with ContentType='application/json'.\"\"\"\n",
    "    # Initialize the runtime SageMaker client\n",
    "    runtime_client = boto3.client('runtime.sagemaker', region_name='ca-central-1')  \n",
    "    try:\n",
    "        \"\"\"\n",
    "        if not isinstance(payload, str):\n",
    "            payload = str(payload)\n",
    "        \"\"\"\n",
    "        # Invoke the SageMaker endpoint\n",
    "        response = runtime_client.invoke_endpoint(\n",
    "            EndpointName=endpoint_name,\n",
    "            ContentType='application/json',\n",
    "            Body=json.dumps(payload)\n",
    "        )\n",
    "        # Decode the response\n",
    "        result = json.loads(response['Body'].read().decode())\n",
    "        return (result)\n",
    "        #print(f\"Prediction from {endpoint_name}: {result}\")\n",
    "    except Exception as e:\n",
    "        print(f\"Error invoking SageMaker endpoint {endpoint_name}: {e}\")\n",
    "\n",
    "def invoke_sagemaker_endpoint_pretrain(endpoint_name, payload):\n",
    "    \"\"\"Invoke a SageMaker endpoint to get predictions with ContentType='text/plain'.\"\"\"\n",
    "    # Initialize the runtime SageMaker client\n",
    "    runtime_client = boto3.client('runtime.sagemaker', region_name='ca-central-1')  \n",
    "\n",
    "    try:\n",
    "        # Ensure payload is a string, since ContentType is 'text/plain'\n",
    "        if not isinstance(payload, str):\n",
    "            payload = str(payload)\n",
    "        \n",
    "        # Invoke the SageMaker endpoint\n",
    "        response = runtime_client.invoke_endpoint(\n",
    "            EndpointName=endpoint_name,\n",
    "            ContentType='text/plain',\n",
    "            Body=payload\n",
    "        )\n",
    "        \n",
    "        # Decode the response\n",
    "        result = json.loads(response['Body'].read().decode())\n",
    "        return (result)\n",
    "        #print(f\"Prediction from {endpoint_name}: {result}\")\n",
    "    except Exception as e:\n",
    "        print(f\"Error invoking SageMaker endpoint {endpoint_name}: {e}\")\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ada6459d-cde4-42cd-b8f7-aabf5f1dff98",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Listing SageMaker Endpoints:\n",
      "Endpoint Name: semantic-search-pretrain-all-MiniLM-L6-v2-1719456491, Status: InService\n",
      "Endpoint Name: all-mpnet-base-v2-mpf-huggingface-test, Status: InService\n"
     ]
    }
   ],
   "source": [
    "list_sagemaker_endpoints()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "2502cd7a-0e64-43c5-bab6-910565e7a94e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "endpoint_name = 'semantic-search-pretrain-all-MiniLM-L6-v2-1719456491'\n",
    "payload = \"Riverice events in ottawa\"\n",
    "vector = invoke_sagemaker_endpoint_pretrain(endpoint_name, payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "16c1e1b8-fc23-48e0-a67c-48cf635a7419",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_id</th>\n",
       "      <th>relevancy_score</th>\n",
       "      <th>title</th>\n",
       "      <th>uuid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ZtwtVpAB8FP_I1hR7YnC</td>\n",
       "      <td>0.653354</td>\n",
       "      <td>Mapping major floods April-May 2017</td>\n",
       "      <td>34085f6d-106a-41af-a29b-53ed6947c249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HNwtVpAB8FP_I1hRZoE9</td>\n",
       "      <td>0.643385</td>\n",
       "      <td>Ontario Hydro Network - Watercourses</td>\n",
       "      <td>17bfc1bb-4849-4615-83a4-f7bd1dd21fd2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>S9wtVpAB8FP_I1hRaYEp</td>\n",
       "      <td>0.642681</td>\n",
       "      <td>Ontario Integrated Hydrology (OIH) data</td>\n",
       "      <td>ef0c4387-38ce-4adc-b761-f0506b82564e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>LtwtVpAB8FP_I1hR-4qU</td>\n",
       "      <td>0.639701</td>\n",
       "      <td>2022 Events</td>\n",
       "      <td>c5bb7cfa-1eb1-45fb-8d1a-1ca9d0eafec4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>JNwtVpAB8FP_I1hRqYXe</td>\n",
       "      <td>0.638624</td>\n",
       "      <td>Road Point Events</td>\n",
       "      <td>01164c08-450c-4d66-8291-1ba018f2fc1c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>k9wtVpAB8FP_I1hRbYGa</td>\n",
       "      <td>0.636887</td>\n",
       "      <td>Major river systems in the Far North</td>\n",
       "      <td>e10f6d04-8eff-4ebc-bd47-37d640c443c5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>x9wtVpAB8FP_I1hRDXuL</td>\n",
       "      <td>0.635798</td>\n",
       "      <td>Lakes and Rivers Database (LCE)</td>\n",
       "      <td>a4a5575d-e8e8-4410-bfc6-18e9361ffd3f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>q9wsVpAB8FP_I1hRx3fH</td>\n",
       "      <td>0.635618</td>\n",
       "      <td>Waterfront course</td>\n",
       "      <td>14be6566-5e91-4553-9049-857bcfc0f7ca</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ItwtVpAB8FP_I1hRdoJN</td>\n",
       "      <td>0.635004</td>\n",
       "      <td>Rivers and ditches</td>\n",
       "      <td>c128aff5-325c-4599-ab66-1c9d0b3abc94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>z9wtVpAB8FP_I1hRYYCu</td>\n",
       "      <td>0.634775</td>\n",
       "      <td>River-Mountain Promenade</td>\n",
       "      <td>71433534-694b-4538-8cd7-86530790ab0c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>otwsVpAB8FP_I1hRtXYM</td>\n",
       "      <td>0.634121</td>\n",
       "      <td>Freshwater Atlas Rivers</td>\n",
       "      <td>f7dac054-efbf-402f-ab62-6fc4b32a619e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>6NwtVpAB8FP_I1hRMH04</td>\n",
       "      <td>0.633583</td>\n",
       "      <td>NCC Picnic areas</td>\n",
       "      <td>8962377c-5751-4953-bb85-8362e14e1f64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>39wtVpAB8FP_I1hRkoOh</td>\n",
       "      <td>0.633204</td>\n",
       "      <td>Base Stream and Flow Representation Update</td>\n",
       "      <td>4cb36aa3-eef5-417c-b2c3-a5544435d038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>fNwtVpAB8FP_I1hR74k4</td>\n",
       "      <td>0.632720</td>\n",
       "      <td>High and Low Current Zones of the Ottawa and G...</td>\n",
       "      <td>b1e7c9a9-a34a-457c-8671-bae4529749fb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>udwtVpAB8FP_I1hRj4N3</td>\n",
       "      <td>0.632578</td>\n",
       "      <td>Stream Connectivity Indicator for Alberta</td>\n",
       "      <td>557f2170-ffa5-462e-b653-af18f001a60f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>QNwtVpAB8FP_I1hRzYdh</td>\n",
       "      <td>0.630642</td>\n",
       "      <td>Boards of water on the island of Montreal (lak...</td>\n",
       "      <td>ec3fe75c-fdd4-4d99-b8b6-7ebdd69cc7bb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>TtwtVpAB8FP_I1hRaYFl</td>\n",
       "      <td>0.630527</td>\n",
       "      <td>Fish stocking data for recreational purposes</td>\n",
       "      <td>b15d6f41-2a3b-4af6-914c-1094754bbc0b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>U9wtVpAB8FP_I1hRBHvP</td>\n",
       "      <td>0.629713</td>\n",
       "      <td>WSA - Water Feature Annotation (50,000)</td>\n",
       "      <td>dae4e2d3-0da0-40c7-8957-7fd74a8c5099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>WdwtVpAB8FP_I1hRaoEd</td>\n",
       "      <td>0.629602</td>\n",
       "      <td>Ontario Hydro Network - Hydrographic Line</td>\n",
       "      <td>a6e17b31-817b-408e-87cf-4cab5b6eb9f2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>a9wtVpAB8FP_I1hR7okU</td>\n",
       "      <td>0.629100</td>\n",
       "      <td>Spring 2019 flood mapping</td>\n",
       "      <td>df99bb67-317c-435b-8f9a-50de5fa9729a</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     _id  relevancy_score  \\\n",
       "0   ZtwtVpAB8FP_I1hR7YnC         0.653354   \n",
       "1   HNwtVpAB8FP_I1hRZoE9         0.643385   \n",
       "2   S9wtVpAB8FP_I1hRaYEp         0.642681   \n",
       "3   LtwtVpAB8FP_I1hR-4qU         0.639701   \n",
       "4   JNwtVpAB8FP_I1hRqYXe         0.638624   \n",
       "5   k9wtVpAB8FP_I1hRbYGa         0.636887   \n",
       "6   x9wtVpAB8FP_I1hRDXuL         0.635798   \n",
       "7   q9wsVpAB8FP_I1hRx3fH         0.635618   \n",
       "8   ItwtVpAB8FP_I1hRdoJN         0.635004   \n",
       "9   z9wtVpAB8FP_I1hRYYCu         0.634775   \n",
       "10  otwsVpAB8FP_I1hRtXYM         0.634121   \n",
       "11  6NwtVpAB8FP_I1hRMH04         0.633583   \n",
       "12  39wtVpAB8FP_I1hRkoOh         0.633204   \n",
       "13  fNwtVpAB8FP_I1hR74k4         0.632720   \n",
       "14  udwtVpAB8FP_I1hRj4N3         0.632578   \n",
       "15  QNwtVpAB8FP_I1hRzYdh         0.630642   \n",
       "16  TtwtVpAB8FP_I1hRaYFl         0.630527   \n",
       "17  U9wtVpAB8FP_I1hRBHvP         0.629713   \n",
       "18  WdwtVpAB8FP_I1hRaoEd         0.629602   \n",
       "19  a9wtVpAB8FP_I1hR7okU         0.629100   \n",
       "\n",
       "                                                title  \\\n",
       "0                 Mapping major floods April-May 2017   \n",
       "1                Ontario Hydro Network - Watercourses   \n",
       "2             Ontario Integrated Hydrology (OIH) data   \n",
       "3                                         2022 Events   \n",
       "4                                   Road Point Events   \n",
       "5                Major river systems in the Far North   \n",
       "6                     Lakes and Rivers Database (LCE)   \n",
       "7                                   Waterfront course   \n",
       "8                                  Rivers and ditches   \n",
       "9                            River-Mountain Promenade   \n",
       "10                            Freshwater Atlas Rivers   \n",
       "11                                   NCC Picnic areas   \n",
       "12         Base Stream and Flow Representation Update   \n",
       "13  High and Low Current Zones of the Ottawa and G...   \n",
       "14          Stream Connectivity Indicator for Alberta   \n",
       "15  Boards of water on the island of Montreal (lak...   \n",
       "16       Fish stocking data for recreational purposes   \n",
       "17            WSA - Water Feature Annotation (50,000)   \n",
       "18          Ontario Hydro Network - Hydrographic Line   \n",
       "19                          Spring 2019 flood mapping   \n",
       "\n",
       "                                    uuid  \n",
       "0   34085f6d-106a-41af-a29b-53ed6947c249  \n",
       "1   17bfc1bb-4849-4615-83a4-f7bd1dd21fd2  \n",
       "2   ef0c4387-38ce-4adc-b761-f0506b82564e  \n",
       "3   c5bb7cfa-1eb1-45fb-8d1a-1ca9d0eafec4  \n",
       "4   01164c08-450c-4d66-8291-1ba018f2fc1c  \n",
       "5   e10f6d04-8eff-4ebc-bd47-37d640c443c5  \n",
       "6   a4a5575d-e8e8-4410-bfc6-18e9361ffd3f  \n",
       "7   14be6566-5e91-4553-9049-857bcfc0f7ca  \n",
       "8   c128aff5-325c-4599-ab66-1c9d0b3abc94  \n",
       "9   71433534-694b-4538-8cd7-86530790ab0c  \n",
       "10  f7dac054-efbf-402f-ab62-6fc4b32a619e  \n",
       "11  8962377c-5751-4953-bb85-8362e14e1f64  \n",
       "12  4cb36aa3-eef5-417c-b2c3-a5544435d038  \n",
       "13  b1e7c9a9-a34a-457c-8671-bae4529749fb  \n",
       "14  557f2170-ffa5-462e-b653-af18f001a60f  \n",
       "15  ec3fe75c-fdd4-4d99-b8b6-7ebdd69cc7bb  \n",
       "16  b15d6f41-2a3b-4af6-914c-1094754bbc0b  \n",
       "17  dae4e2d3-0da0-40c7-8957-7fd74a8c5099  \n",
       "18  a6e17b31-817b-408e-87cf-4cab5b6eb9f2  \n",
       "19  df99bb67-317c-435b-8f9a-50de5fa9729a  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query={\n",
    "    \"size\": 20,\n",
    "    \"query\": {\n",
    "        \"knn\": {\n",
    "            \"vector\":{\n",
    "                \"vector\":vector,\n",
    "                \"k\":20\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "res = aos_client.search(index=index_name, size=20,body=query,request_timeout=55)\n",
    "query_result=[]\n",
    "for hit in res['hits']['hits']:\n",
    "    row=[hit['_id'],hit['_score'],hit['_source']['title'],hit['_source']['id']]\n",
    "    query_result.append(row)\n",
    "query_result_df = pd.DataFrame(data=query_result,columns=[\"_id\",\"relevancy_score\",\"title\",'uuid'])\n",
    "display(query_result_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e72615ad-fd2e-489f-9367-afd637f62474",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p310",
   "language": "python",
   "name": "conda_pytorch_p310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}