04_build_embeddings_scann.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4: Building a Approximate Nearest Neighbor Index for the Item Embeddings using ScaNN and AI Platform Training\n",
    "\n",
    "This tutorial shows how to use Matrix Factorization algorithm in BigQuery ML to generate embeddings for items based on their cooccurrence statistics. The generated item embeddings can be then used to find similar items.\n",
    "\n",
    "Part 4 covers building an approximate nearest neighbor index for the embeddings \n",
    "using ScaNN and AI Platform Training. The built ScaNN index then is stored in Cloud Storage.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q scann"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure GCP environment settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_ID = 'chernikov-playground'\n",
    "BUCKET = 'cooltacular-gcp1'\n",
    "REGION = 'us-west1'\n",
    "\n",
    "EMBEDDING_FILES_PREFIX = f'gs://{BUCKET}/bqml/item_embeddings/embeddings-*'\n",
    "OUTPUT_INDEX_DIR = f'gs://{BUCKET}/bqml/scann_index'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate your GCP account\n",
    "This is required if you run the notebook in Colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "  from google.colab import auth\n",
    "  auth.authenticate_user()\n",
    "  print(\"Colab user is authenticated.\")\n",
    "except: pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build ScaNN Index Locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from index_builder.builder import indexer\n",
    "indexer.build(EMBEDDING_FILES_PREFIX, OUTPUT_INDEX_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build ScaNN Index using AI Platform Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if tf.io.gfile.exists(OUTPUT_INDEX_DIR):\n",
    "  print(\"Removing {} contents...\".format(OUTPUT_INDEX_DIR))\n",
    "  tf.io.gfile.rmtree(OUTPUT_INDEX_DIR)\n",
    "\n",
    "print(\"Creating output: {}\".format(OUTPUT_INDEX_DIR))\n",
    "tf.io.gfile.makedirs(OUTPUT_INDEX_DIR)\n",
    "\n",
    "timestamp = datetime.utcnow().strftime('%y%m%d%H%M%S')\n",
    "job_name = f'ks_bqml_build_scann_index_{timestamp}'\n",
    "\n",
    "!gcloud ai-platform jobs submit training {job_name} \\\n",
    "  --project={PROJECT_ID} \\\n",
    "  --region={REGION} \\\n",
    "  --job-dir={OUTPUT_INDEX_DIR}/jobs/ \\\n",
    "  --package-path=index_builder/builder \\\n",
    "  --module-name=builder.task \\\n",
    "  --config='index_builder/config.yaml' \\\n",
    "  --runtime-version=2.2 \\\n",
    "  --python-version=3.7 \\\n",
    "  --\\\n",
    "  --embedding-files-path={EMBEDDING_FILES_PREFIX} \\\n",
    "  --output-dir={OUTPUT_INDEX_DIR} \\\n",
    "  --num-leaves=500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the AI Platform Training job finish, you can check the built and stored ScaNN index in Cloud Storage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gsutil ls {OUTPUT_INDEX_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the ScaNN Index API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from index_server.matching import ScaNNMatcher\n",
    "scann_matcher = ScaNNMatcher(OUTPUT_INDEX_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector = np.random.rand(50)\n",
    "scann_matcher.match(vector, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License\n",
    "\n",
    "Copyright 2020 Google LLC\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License. You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. \n",
    "\n",
    "See the License for the specific language governing permissions and limitations under the License.\n",
    "\n",
    "**This is not an official Google product but sample code provided for an educational purpose**"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-2-3-gpu.2-3.m59",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-2-3-gpu.2-3:m59"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}