3_model_inference_and_monitoring.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "06d5c4be",
   "metadata": {},
   "source": [
    "# Model Inference and Monitoring\n",
    "\n",
    "The workshop contains three different notebooks. Each one focuses on a different stage:\n",
    "    \n",
    "1. Dataset Generation. The first notebook focuses on generating a dataset for training the model. We will create a Robust Test Suite to check that the dataset generated meets certain conditions\n",
    "2. Model Training. The second notebook focuses on training the model. We will create a Robust Test Suite to check that the trained model meets certain conditions.\n",
    "3. Model Inference (This Notebook). In the last notebook, we use mercury.monitoring to monitor data drift and estimate the predicted performance of the model without having the labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8377bade",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "You can install mercury-monitoring by running:\n",
    "\n",
    "```\n",
    "!pip install mercury-monitoring\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0bd3ffc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from joblib import load\n",
    "import pickle\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "SEED = 23"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc9224a6",
   "metadata": {},
   "source": [
    "## Load Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27bc482e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec2c58c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "path_dataset = \"./models/\"\n",
    "\n",
    "model = load(path_dataset + 'model.joblib') \n",
    "\n",
    "with open(path_dataset + \"features.pkl\", \"rb\") as fp:\n",
    "    features = pickle.load(fp)\n",
    "    \n",
    "label = \"default.payment.next.month\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a1c7c6",
   "metadata": {},
   "source": [
    "## Load Original Dataset \n",
    "\n",
    "We load the original training dataset, so we can use it for data drift detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78b76d0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "path_dataset = \"./dataset/\"\n",
    "\n",
    "df_train = pd.read_csv(path_dataset + \"train.csv\")\n",
    "df_test = pd.read_csv(path_dataset + \"test.csv\")\n",
    "\n",
    "X_train = df_train[features]\n",
    "X_test = df_test[features]\n",
    "\n",
    "y_train = df_train[label]\n",
    "y_test = df_test[label]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a4b43b4",
   "metadata": {},
   "source": [
    "## Model Inference\n",
    "\n",
    "Now, we load the `uci_credit_drifted_inference.csv` which contains the data that we receive at inference time. It contains a column \"time\" which indicates the time that we receive the data. We will check two different timesteps: time=2 and time=9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0546f3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Dataset for specific time\n",
    "df = pd.read_csv(\"data/uci_credit_drifted_inference.csv\")\n",
    "df_t2 = df[df[\"time\"] == 2]\n",
    "df_t9 = df[df[\"time\"] == 9]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7c76a79",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df_t2.shape)\n",
    "print(df_t9.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f1aa697",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Features and Labels\n",
    "y_true_t2 = df_t2[label]\n",
    "X_t2 = df_t2[features]\n",
    "\n",
    "y_true_t9 = df_t9[label]\n",
    "X_t9 = df_t9[features]\n",
    "\n",
    "# Get Model predictions\n",
    "y_pred_proba_train = model.predict_proba(X_train)\n",
    "\n",
    "y_pred_t2 = model.predict(X_t2)\n",
    "y_pred_proba_t2 = model.predict_proba(X_t2)\n",
    "\n",
    "y_pred_t9 = model.predict(X_t9)\n",
    "y_pred_proba_t9 = model.predict_proba(X_t9)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cae66f4",
   "metadata": {},
   "source": [
    "## Drift Detection\n",
    "\n",
    "Now, we will use [mercury.monitoring](https://bbva.github.io/mercury-monitoring/) for data drift detection. We will use the [`KSDrift`](https://bbva.github.io/mercury-monitoring/reference/drift/#mercury.monitoring.drift.ks_drift_detector.KSDrift) and the [`DomainClassifierDrift`](https://bbva.github.io/mercury-monitoring/reference/drift/#mercury.monitoring.drift.domain_classifier_drift_detector.DomainClassifierDrift), although there are more detectors available that can be used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86df5539",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mercury.monitoring.drift.ks_drift_detector import KSDrift\n",
    "from mercury.monitoring.drift.domain_classifier_drift_detector import DomainClassifierDrift"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a75e4a12",
   "metadata": {},
   "source": [
    "### KS Drift\n",
    "\n",
    "Let's start using the `KSDrift`. This detector performs a Kolmogorov-Smirnov (KS) test individually for each specified variable.\n",
    "\n",
    "We can see that drift is not detected in time=2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c43ac37d",
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_detector = KSDrift(X_train.to_numpy(), X_t2.to_numpy(), features=features, p_val=0.01)\n",
    "drift_metrics = drift_detector.calculate_drift()\n",
    "print(\"Drift Score: \", drift_metrics[\"score\"])\n",
    "print(\"Is drift detected? \", drift_metrics[\"drift_detected\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65aa1c86",
   "metadata": {},
   "source": [
    "Now, let's look at time=9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47317673",
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_detector = KSDrift(X_train.to_numpy(), X_t9.to_numpy(), features=features, p_val=0.01)\n",
    "drift_metrics = drift_detector.calculate_drift()\n",
    "print(\"Drift Score: \", drift_metrics[\"score\"])\n",
    "print(\"Is drift detected? \", drift_metrics[\"drift_detected\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "117c282b",
   "metadata": {},
   "source": [
    "Now drift is detected and the score is higher. We can also obtain which are the features with drift and plot them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70add503",
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_detector.get_drifted_features()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7401351c",
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_detector.plot_feature_drift_scores()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6065c940",
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_detector.plot_distribution_drifted_features(discrete=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be5f99c0",
   "metadata": {},
   "source": [
    "### Domain Classifier\n",
    "\n",
    "Now we use the [`DomainClassifierDrift`](https://bbva.github.io/mercury-monitoring/reference/drift/#mercury.monitoring.drift.domain_classifier_drift_detector.DomainClassifierDrift). This component trains a classifier (Random Forest) to distinguish between a source dataset and a target dataset. The performance of the model detecting which sample belongs to each dataset (ROC-AUC) is then used to indicate if drift is detected and as drift score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3164b623",
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_detector = DomainClassifierDrift(\n",
    "    X_train.to_numpy(), X_t2.to_numpy(), features=features, p_val=0.05, n_runs=5\n",
    ")\n",
    "drift_metrics = drift_detector.calculate_drift()\n",
    "print(\"Drift Score: \", drift_metrics[\"score\"])\n",
    "print(\"Is drift detected? \", drift_metrics[\"drift_detected\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c2ed42c",
   "metadata": {},
   "source": [
    "We see that drift is not detected at time=2. Let's check at time=9:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22687660",
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_detector = DomainClassifierDrift(\n",
    "    X_train.to_numpy(), X_t9.to_numpy(), features=features, p_val=0.05, n_runs=20\n",
    ")\n",
    "drift_metrics = drift_detector.calculate_drift()\n",
    "print(\"Drift Score: \", drift_metrics[\"score\"])\n",
    "print(\"Is drift detected? \", drift_metrics[\"drift_detected\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2b3913e",
   "metadata": {},
   "source": [
    "### Drift in Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cf4e725",
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_detector = KSDrift(y_pred_proba_train, y_pred_proba_t2, p_val=0.01)\n",
    "drift_metrics = drift_detector.calculate_drift()\n",
    "print(\"Drift Score: \", drift_metrics[\"score\"])\n",
    "print(\"Is drift detected? \", drift_metrics[\"drift_detected\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2caa6c55",
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_detector = KSDrift(y_pred_proba_train, y_pred_proba_t9, p_val=0.01)\n",
    "drift_metrics = drift_detector.calculate_drift()\n",
    "print(\"Drift Score: \", drift_metrics[\"score\"])\n",
    "print(\"Is drift detected? \", drift_metrics[\"drift_detected\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bd2a6d8",
   "metadata": {},
   "source": [
    "## Accuracy Estimation\n",
    "\n",
    "We will use now the [`PerformancePredictor`](https://bbva.github.io/mercury-monitoring/reference/estimation/#mercury.monitoring.estimation.performance_predictor.PerformancePredictor) component to estimate the accuracy of the model in a particular timestep without having the labels.\n",
    "\n",
    "The component is based on the method presented in [Learning to Validate the Predictions of\n",
    "Black Box Classifiers on Unseen Data](https://ssc.io/pdf/mod0077s.pdf)\n",
    "\n",
    "Let's load the the `DataSchema` which we will use in the `PerformancePredictor` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cd9ed8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Data Schema\n",
    "from mercury.dataschema import DataSchema\n",
    "schema = DataSchema.load(path_dataset + \"schema.json\")\n",
    "\n",
    "# We won't need this for the PerformancePredictor\n",
    "del schema.feats['default.payment.next.month']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c686c21d",
   "metadata": {},
   "source": [
    "Now let's estimate the Performance on timetep=2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b535c112",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from mercury.monitoring.estimation.performance_predictor import PerformancePredictor\n",
    "\n",
    "# Create PerfomancePredictor\n",
    "performance_predictor = PerformancePredictor(model, metric_fn=accuracy_score, random_state=SEED)\n",
    "performance_predictor.fit(X=X_test, y=y_test, X_serving=X_t2, dataset_schema=schema)\n",
    "predicted_acc = performance_predictor.predict(X_t2)\n",
    "\n",
    "# Real Accuracy\n",
    "real_acc = accuracy_score(y_true_t2, model.predict(X_t2))\n",
    "\n",
    "print(\"real_acc: \", real_acc)\n",
    "print(\"predicted_acc: \", predicted_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31c67992",
   "metadata": {},
   "source": [
    "Now let's estimate on timestep=9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce3dd40e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mercury.monitoring.estimation.performance_predictor import PerformancePredictor\n",
    "\n",
    "# Create PerfomancePredictor\n",
    "performance_predictor = PerformancePredictor(model, metric_fn=accuracy_score, random_state=SEED)\n",
    "performance_predictor.fit(X=X_test, y=y_test, X_serving=X_t9, dataset_schema=schema)\n",
    "predicted_acc = performance_predictor.predict(X_t9)\n",
    "\n",
    "# Real Accuracy\n",
    "real_acc = accuracy_score(y_true_t9, model.predict(X_t9))\n",
    "\n",
    "print(\"real_acc: \", real_acc)\n",
    "print(\"predicted_acc: \", predicted_acc)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}