short_usage_guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# xCOLUMNs Quick Usage Guide\n",
    "\n",
    "Welcome to the xCOLUMNs Quick Usage Guide. It aims to provide a quick overview of the xCOLUMNs library and how to use it.\n",
    "\n",
    "\n",
    "## What is xCOLUMNs?\n",
    "\n",
    "xCOLUMNs stands for x**Consistent Optimization of Label-wise Utilities in Multi-label classificatioN**s.\n",
    "It is a small Python library that aims to implement different methods for the optimization of a general family of \n",
    "metrics that can be defined on multi-label classification matrices. \n",
    "These include, but are not limited to, label-wise metrics (see below for details). \n",
    "The library provides an efficient implementation of the different optimization methods that easily scale to the extreme multi-label classification (XMLC) - problems with a very large number of labels and instances.\n",
    "\n",
    "\n",
    "### What is multi-label classification?\n",
    "\n",
    "Multi-label classification is a type of classification task, where each instance can be assigned to multiple labels. \n",
    "For example, a news article can be assigned to multiple categories, such as \"politics\", \"sports\", \"technology\", etc.\n",
    "The labels for each instance are usually represented as binary vectors $y \\in \\{0, 1\\}^m$, where $m$ is the number of possible labels, and $y_j = 1$ means that label $j$ is relevant for the given instance, and $y_j = 0$ otherwise.\n",
    "\n",
    "\n",
    "### What are label-wise utilities?\n",
    "\n",
    "The label-wise utilities are performance metrics that decompose over labels, i.e., they can be defined as a sum of individual metrics for each label:\n",
    "\n",
    "$$\n",
    "    \\Psi(Y, \\hat{Y}) = \\sum_{j=1}^{m} \\psi^j(y_{:,j}, \\hat{y}_{:,j}) \\,,\n",
    "$$\n",
    "\n",
    "where $Y$ is the true label matrix, $\\hat{Y}$ is the predicted label matrix (both of size $n \\times m$, where $n$ is number of instances) and $y_{:,j}$ is the $j$-th column of $Y$, and $\\psi^j$ is binary utility function for label $j$.\n",
    "\n",
    "Such metrics, under the assumption that are order-invariant, can be defined as a function of the collection of confusion matrices for each label, i.e., $C_j = \\text{confusion matrix}(y_{:,j}, \\hat{y}_{:,j})$. \n",
    "Where confusion matrix $C_j$ is a 2x2 matrix with entries: true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN).\n",
    "\n",
    "The family of label-wise utilities includes standard instance-wise metrics, such as precision at k or Hamming score/loss, that are popular in extreme classification and recommendation tasks as well as more complex metrics, such as\n",
    "macro-averaged F-score, Jaccard score, macro-averged precision and recall at k etc.\n",
    "\n",
    "\n",
    "### Main idea of xCOLUMNs\n",
    "\n",
    "So far, all the algorithms implemented in xCOLUMNs require providing the definition of the target function and marginal conditional probabilities (or their estimates) of labels - $P(y_{j} = 1 | x)$ \n",
    "for each label $j$ and each instance $x$ resulting in matrix that we denote as $H$ (large $\\eta$) and in the names as **`y_proba`** argument of different methods. \n",
    "The estimates of label probabilities can be easily obtained from many different classification models, such as logistic regression, neural networks, etc. \n",
    "In this sense xCOLUMNs implements plug-in inference methods, that can be used on top of any probability estimatation model.\n",
    "\n",
    "\n",
    "### What methods are implemented in xCOLUMNs?\n",
    "\n",
    "The aim of xCOLUMNs is to provide methods for the optimization of the general family of label-wise utilities. Currently, the following methods are implemented:\n",
    "\n",
    "- Prediction for provided test set using **Block Coordinate Ascent/Descent (BC)** method, described in [1].\n",
    "- Search for optimal population classifier using **Frank-Wolfe (FW)** method, described in [2].\n",
    "\n",
    "[1] [Erik Schultheis, Marek Wydmuch, Wojciech Kotłowski, Rohit Babbar, Krzysztof Dembczyński. Generalized test utilities for long-tail performance in extreme multi-label classification. NeurIPS 2023.](https://arxiv.org/abs/2311.05081)\n",
    "\n",
    "[2] [Erik Schultheis, Wojciech Kotłowski, Marek Wydmuch, Rohit Babbar, Strom Borman, Krzysztof Dembczyński. Consistent algorithms for multi-label classification with macro-at-k metrics. ICLR 2024.](https://arxiv.org/abs/2401.16594)\n",
    "\n",
    "\n",
    "## How to use xCOLUMNs?\n",
    "\n",
    "This guide will provide a quick overview of how to use xCOLUMNs for the optimization of label-wise utilities.\n",
    "\n",
    "\n",
    "### Installation\n",
    "\n",
    "xCOLUMNs can be easily installed using pip:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install xcolumns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional requirements for this guide\n",
    "\n",
    "This quick guide requires the additional installation of the following libraries, that are not required by xCOLUMNs itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install sklearn matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a simple example dataset\n",
    "\n",
    "The cell below creates a simple example dataset using sklearn library, that we will use to demonstrate the usage of xCOLUMNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import make_multilabel_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.multioutput import MultiOutputClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "seed=2024\n",
    "x, y = make_multilabel_classification(n_samples=100000, n_features=100, n_classes=20, n_labels=3, length=50, allow_unlabeled=True, sparse=False, return_indicator='dense', return_distributions=False, random_state=seed)\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=seed)\n",
    "x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.3, random_state=seed)\n",
    "clf = MultiOutputClassifier(LogisticRegression()).fit(x_train, y_train)\n",
    "\n",
    "def clf_predict(clf, x):\n",
    "    \"\"\"\n",
    "    Process the output of a multioutput classifier to get the marginal probabilities of each label (class).\n",
    "    \"\"\"\n",
    "    y_proba = clf.predict_proba(x)\n",
    "    return np.array(y_proba)[:,:,1].transpose()\n",
    "\n",
    "y_proba_val = clf_predict(clf, x_val)\n",
    "y_proba_test = clf_predict(clf, x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Supported data formats\n",
    "\n",
    "The xCOLUMNs library supports the data provided as numpy arrays. But also PyTorch tensors (PyTorch optional dependency, some methods may benefit from GPU acceleration). As well as sparse matrices (scipy.sparse.csr_matrix), that may be useful for very large datasets with a large number of labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cast the data to the desired type\n",
    "target_type = \"csr_matrix\"\n",
    "\n",
    "if target_type == \"torch\":\n",
    "    # Works also for torch tensors\n",
    "    import torch \n",
    "    y_test = torch.tensor(y_test)\n",
    "    y_val = torch.tensor(y_val)\n",
    "    y_proba_test = torch.tensor(y_proba_test)\n",
    "    y_proba_val = torch.tensor(y_proba_val)\n",
    "    \n",
    "elif target_type == \"csr_matrix\":\n",
    "    # Works also for csr_sparse matrices\n",
    "    from scipy.sparse import csr_matrix\n",
    "    y_test = csr_matrix(y_test)\n",
    "    y_val = csr_matrix(y_val)\n",
    "    y_proba_test = csr_matrix(y_proba_test)\n",
    "    y_proba_val = csr_matrix(y_proba_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defininig target metric\n",
    "\n",
    "To use the `xcolumns` library, we need to define a target metric. The metric should be a function of confusion matrices for each label. The confusion matrix for each label is a 2x2 matrix with entries: true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN).\n",
    "Because of that let's first defina a binary metrics function that use values of tp, fp, fn, tn to calculate the value of the metric for each label. Let's define a simple F1 score metric.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def our_binary_f1_score_on_conf_matrix(tp, fp, fn, tn):\n",
    "    return 2*tp / (2*tp + fp + fn + 1e-6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the binary metric function we can define a label-wise metric function that accumulates the values of the binary metric for each label.\n",
    "Lets define a macro-averaged variant of the F1 score metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def our_macro_f1_score_on_conf_matrix(tp, fp, fn, tn):\n",
    "    return our_binary_f1_score_on_conf_matrix(tp, fp, fn, tn).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "xCOLUMNs already provides implementation of popular metricses, that can be found in `xcolumns.metrics` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xcolumns.metrics import binary_f1_score_on_conf_matrix, macro_f1_score_on_conf_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's define simple evaluation function that will calculate the value of the metric for the given true and predicted labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xcolumns.metrics import calculate_confusion_matrix\n",
    "\n",
    "def evaluate(y_true, y_pred):\n",
    "    \"\"\"\n",
    "    Evaluate the performance of a multioutput classifier.\n",
    "    \"\"\"\n",
    "    C = calculate_confusion_matrix(y_true, y_pred)\n",
    "\n",
    "    # C is ConfusionMatrix object, that contains tp, fp, fn, tn as attributes.\n",
    "    # C can be unpacked to get the values of tp, fp, fn, tn in this order.\n",
    "    assert macro_f1_score_on_conf_matrix(C.tp, C.fp, C.fn, C.tn, epsilon=1e-6) == our_macro_f1_score_on_conf_matrix(*C)\n",
    "    print(our_macro_f1_score_on_conf_matrix(*C))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple baseline\n",
    "\n",
    "Now let's use the simples inference strategy to predict the 3 labels with largest probability estimates for each instance. We can do that using `predict_top_k` function from `xcolumns.weighted_prediction` module. That implements the simple inference strategy based on the weighting the estimated probabilities of labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xcolumns.weighted_prediction import predict_top_k\n",
    "\n",
    "y_pred = predict_top_k(y_proba_test, k=3)\n",
    "evaluate(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction using Block Coordnate Ascent/Descent method for a given test set\n",
    "\n",
    "Now we can use Block Coordinate Ascent method to find better prediction that maximizes the macro F1 score.\n",
    "To do that we can use `predict_using_bc_with_0approx` function from `xcolumns.block_coordinate` module.\n",
    "It requires the matrix of estimated probabilities of labels for each instance, the number of labels to predict for each instance (use 0 if you don't want to use constraint budget), and the target metric function and direction of optimization (maximize or minimize, default is maximize)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xcolumns.block_coordinate import predict_using_bc_with_0approx\n",
    "\n",
    "y_pred = predict_using_bc_with_0approx(y_proba_test, our_binary_f1_score_on_conf_matrix, k=3, maximize=True, seed=seed, verbose=True)\n",
    "evaluate(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding optimal population classifier using Frank-Wolfe method\n",
    "\n",
    "`predict_using_bc_with_0approx` requires the whole matrix of estimated probabilities of labels for each instance. In some cases, one may prefer to have a population classifier that can be used to predict labels for new instances independently. In such cases, one can use the Frank-Wolfe method to find the optimal population classifier. The method requires training set (preferebly other then the training set used to train the probality estimator), the metric (support any function of ConfusionMatrix) and the number of labels to predict for each instance (use 0 if you don't want to use constraint budget)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xcolumns.frank_wolfe import find_classifier_using_fw\n",
    "\n",
    "rnd_clf, meta = find_classifier_using_fw(\n",
    "    y_val, y_proba_val, our_macro_f1_score_on_conf_matrix, k = 3, return_meta=True, seed=seed, verbose=True\n",
    ")\n",
    "\n",
    "y_pred = rnd_clf.predict(y_proba_test, seed=seed)\n",
    "evaluate(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}