templates/1_Pixie_Cluster_Pixels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pixie: pixel clustering notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "import"
    ]
   },
   "outputs": [],
   "source": [
    "# import required packages\n",
    "import json\n",
    "import os\n",
    "from datetime import datetime as dt\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from alpineer import io_utils, load_utils\n",
    "from matplotlib import rc_file_defaults\n",
    "\n",
    "from ark.phenotyping import (pixel_cluster_utils, pixel_meta_clustering,\n",
    "                             pixel_som_clustering, pixie_preprocessing)\n",
    "from ark.utils import data_utils, example_dataset, plot_utils\n",
    "from ark.utils.metacluster_remap_gui import (MetaClusterGui,\n",
    "                                             colormap_helper,\n",
    "                                             metaclusterdata_from_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0: Set root directory and download the example dataset\n",
    "\n",
    "Here we are using the example data located in `data/example_dataset`. To use your own data, change `base_dir` to point to your own sub-directory within the data folder.\n",
    "\n",
    "* `base_dir`: the path to all of your imaging data. This directory will also store all of the directories/files created during pixel clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "base_dir"
    ]
   },
   "outputs": [],
   "source": [
    "base_dir = \"../data/example_dataset\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to test the features with an example dataset, run the cell below. It will download a MIBI-TOF dataset consisting of 11 FOVs with 22 channels. You may find more information about the example dataset in the [README](../README.md#example-dataset).\n",
    "\n",
    "If you are using your own data, skip the cell below.\n",
    "\n",
    "* `overwrite_existing`: If set to `False`, it will not overwrite existing data in `data/example_dataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "ex_data_download"
    ]
   },
   "outputs": [],
   "source": [
    "example_dataset.get_example_dataset(dataset=\"cluster_pixels\", save_dir = base_dir, overwrite_existing = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1: Set file paths and parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `tiff_dir`: path to the directory containing your imaging data. Images should be single-channel TIFFs.\n",
    "* `img_sub_folder`: if `tiff_dir` contains an additional subfolder structure, override `None` with the appropriate folder name\n",
    "* `segmentation_dir`: path to the directory containing your segmentation masks (where each pixel value corresponds to a cell ID, can be generated using the segmentation notebook at github.com/angelolab/ark-analysis). Set this argument to `None` if you do not have segmentation labels or wish to run pixel clustering without them (they are required for cell clustering)\n",
    "* `seg_suffix`: the suffix plus the file extension of the segmented images for each FOV. Note that these should be the same for all FOVs. This argument will be ignored if `segmentation_dir` is set to `None`\n",
    "* `pixie_seg_dir`: the created path from the `segmentation_dir`. Is `None` if `segmentation_dir` is `None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "file_path"
    ]
   },
   "outputs": [],
   "source": [
    "tiff_dir = os.path.join(base_dir, \"image_data\")\n",
    "img_sub_folder = None\n",
    "segmentation_dir = os.path.join(\"segmentation\", \"deepcell_output\")\n",
    "seg_suffix = '_whole_cell.tiff'\n",
    "\n",
    "if segmentation_dir is not None:\n",
    "    pixie_seg_dir = os.path.join(base_dir, segmentation_dir)\n",
    "else:\n",
    "    pixie_seg_dir = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `fovs` (optional): set a specific set of fovs to load, default loads all the fovs in `tiff_dir`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "load_fovs"
    ]
   },
   "outputs": [],
   "source": [
    "# either get all fovs in the folder...\n",
    "fovs = io_utils.list_folders(tiff_dir)\n",
    "\n",
    "# ... or optionally, select a specific set of fovs manually\n",
    "# fovs = [\"fov14\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define multiprocessing parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turning on multiprocessing provides a speed boost; however, it is not always cross-platform compatible. If you receive errors such as hanging cells without progress updates, try setting `multiprocess` back to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "set_multi"
    ]
   },
   "outputs": [],
   "source": [
    "# set to True to turn on multiprocessing\n",
    "multiprocess = False\n",
    "\n",
    "# define the number of FOVs to process in parallel, ignored if multiprocessing is set to False\n",
    "batch_size = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2: Preprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set a prefix to be applied to all data directories/files created during pixel clustering. If the prefix is not set, a default of the datetime at the start of the run is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_prefix"
    ]
   },
   "outputs": [],
   "source": [
    "# explicitly set pixel_cluster_prefix to override datetime default\n",
    "pixel_cluster_prefix = \"example\"\n",
    "\n",
    "if pixel_cluster_prefix is None:\n",
    "    pixel_cluster_prefix = dt.now().strftime('%Y-%m-%dT%H:%M:%S')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following data directories/files will be created with names prefixed by `pixel_cluster_prefix`:\n",
    "\n",
    "* `pixel_output_dir`: directory name where the pixel clustering outputs are stored\n",
    "* `preprocessed_dir`: directory name where the preprocessed pixel data are stored\n",
    "* `subsetted_dir`: directory name where the subsetted pixel data are stored\n",
    "* `norm_vals_name`: file name where the values used to normalize each channel are stored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "dir_set"
    ]
   },
   "outputs": [],
   "source": [
    "# define the output directory using the specified pixel cluster prefix\n",
    "pixel_output_dir = os.path.join(\"pixie\", \"%s_pixel_output_dir\" % pixel_cluster_prefix)\n",
    "if not os.path.exists(os.path.join(base_dir, pixel_output_dir)):\n",
    "    os.makedirs(os.path.join(base_dir, pixel_output_dir))\n",
    "\n",
    "# define the preprocessed pixel data folders\n",
    "pixel_data_dir = os.path.join(pixel_output_dir, 'pixel_mat_data')\n",
    "pixel_subset_dir = os.path.join(pixel_output_dir, 'pixel_mat_subset')\n",
    "norm_vals_name = os.path.join(pixel_output_dir, 'channel_norm_post_rowsum.feather')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the following arguments:\n",
    "\n",
    "* `channels`: channels to run pixel clustering on\n",
    "* `blur_factor`: sigma (standard deviation) for the Gaussian blur. Higher values are more aggressive in smoothing signal.\n",
    "* `subset_proportion`: the fraction of pixels to take from each FOV for training. Sampling is random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "channel_set"
    ]
   },
   "outputs": [],
   "source": [
    "channels = [\"CD3\", \"CD4\", \"CD8\", \"CD14\", \"CD20\", \"CD31\", \"CD45\", \"CD68\",\n",
    "            \"CD163\", \"CK17\", \"Collagen1\", \"Fibronectin\", \n",
    "            \"ECAD\", \"HLADR\", \"SMA\", \"Vim\"]\n",
    "blur_factor = 2\n",
    "subset_proportion = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During pixel preprocessing, the following is done for each FOV:\n",
    "\n",
    "* Gaussian blur each channel separately\n",
    "* Remove empty pixels\n",
    "* For the remaining pixels, normalize each pixel by the sum of all the channels\n",
    "* Subset a `subset_proportion` fraction of non-empty, normalized pixels. This creates the subsetted dataset for training\n",
    "\n",
    "Note: if you get integer overflow errors loading in your data, try changing the `dtype` argument to a larger type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "gen_pixel_mat"
    ]
   },
   "outputs": [],
   "source": [
    "# run pixel data preprocessing\n",
    "pixie_preprocessing.create_pixel_matrix(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    tiff_dir,\n",
    "    pixie_seg_dir,\n",
    "    img_sub_folder=img_sub_folder,\n",
    "    seg_suffix=seg_suffix,\n",
    "    pixel_output_dir=pixel_output_dir,\n",
    "    data_dir=pixel_data_dir,\n",
    "    subset_dir=pixel_subset_dir,\n",
    "    norm_vals_name=norm_vals_name,\n",
    "    blur_factor=blur_factor,\n",
    "    subset_proportion=subset_proportion,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Pixel clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1: Train pixel SOM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the pixel SOM using the subsetted data. Training is done using a self-organizing map (SOM).\n",
    "\n",
    "The following data directories/files will be created for pixel clustering:\n",
    "\n",
    "* `pixel_som_weights_name`: file name to store the pixel SOM weights\n",
    "* `pc_chan_avg_som_cluster_name`: file name to store the average channel expression across all pixel SOM clusters\n",
    "* `pc_chan_avg_meta_cluster_name`: same as above for pixel meta clusters\n",
    "* `pixel_meta_cluster_remap_name`: file name to store the SOM cluster to meta cluster manual mappings created using the GUI below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_som_path_set"
    ]
   },
   "outputs": [],
   "source": [
    "pixel_som_weights_name = os.path.join(pixel_output_dir, 'pixel_som_weights.feather')\n",
    "pc_chan_avg_som_cluster_name = os.path.join(pixel_output_dir, 'pixel_channel_avg_som_cluster.csv')\n",
    "pc_chan_avg_meta_cluster_name = os.path.join(pixel_output_dir, 'pixel_channel_avg_meta_cluster.csv')\n",
    "pixel_meta_cluster_remap_name = os.path.join(pixel_output_dir, 'pixel_meta_cluster_mapping.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each channel is normalized by their 99.9% value across the entire dataset before training. These values get saved to `norm_vals_name`.\n",
    "\n",
    "For a full set of parameters you can customize for `train_pixel_som`, please consult <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.pixel_cluster_utils.train_pixel_som>pixel training docs</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "train_pixel_som"
    ]
   },
   "outputs": [],
   "source": [
    "# create the pixel SOM weights\n",
    "pixel_pysom = pixel_som_clustering.train_pixel_som(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    subset_dir=pixel_subset_dir,\n",
    "    norm_vals_name=norm_vals_name,\n",
    "    som_weights_name=pixel_som_weights_name,\n",
    "    num_passes=1,\n",
    "    seed=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2: Assign pixel SOM clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the SOM weights learned from `train_pixel_som` to assign pixel clusters to the full preprocessed dataset.\n",
    "\n",
    "Note that each channel is normalized by the respective value stored in `norm_vals_name` (computed in `train_pixel_som`) prior to cluster assignment.\n",
    "\n",
    "`generate_som_avg_files` will compute the average channel expression across all pixel SOM clusters, as well as the number of pixels in each pixel SOM cluster (the data in `pc_chan_avg_som_cluster_name`). This is needed for consensus clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cluster_pixel_mat"
    ]
   },
   "outputs": [],
   "source": [
    "# use pixel SOM weights to assign pixel clusters\n",
    "pixel_som_clustering.cluster_pixels(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_pysom,\n",
    "    data_dir=pixel_data_dir,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "# generate the SOM cluster summary files\n",
    "pixel_som_clustering.generate_som_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_pysom,\n",
    "    data_dir=pixel_data_dir,\n",
    "    pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3: Run pixel consensus clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use consensus hierarchical clustering to cluster pixel SOM clusters into a user-defined number of meta clusters. The consensus clusters are trained on the average channel expression across all pixel SOM clusters (the data stored in `pc_chan_avg_som_cluster_name`). These values are z-scored and capped at the value specified in the `cap` argument prior to consensus clustering. This helps improve meta clustering performance.\n",
    "\n",
    "After consensus clustering, the following are computed by `generate_meta_avg_files`:\n",
    "\n",
    "* The average channel expression across all pixel meta clusters, and the number of pixels per meta cluster (the data in `pc_chan_avg_meta_cluster_name`)\n",
    "* The meta cluster mapping for each pixel SOM cluster in `pc_chan_avg_som_cluster_name` (data is resaved, same data except with an associated meta cluster column)\n",
    "\n",
    "For a full set of parameters you can customize for `pixel_consensus_cluster`, please consult <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.pixel_cluster_utils.pixel_consensus_cluster>pixel consensus clustering docs</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `max_k`: the number of consensus clusters desired\n",
    "* `cap`: used to clip z-scored values prior to consensus clustering (in the range `[-cap, cap]`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_consensus_cluster"
    ]
   },
   "outputs": [],
   "source": [
    "max_k = 20\n",
    "cap = 3\n",
    "\n",
    "# run hierarchical clustering using average pixel SOM cluster expression\n",
    "pixel_cc = pixel_meta_clustering.pixel_consensus_cluster(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    max_k=max_k,\n",
    "    cap=cap,\n",
    "    data_dir=pixel_data_dir,\n",
    "    pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "# generate the meta cluster summary files\n",
    "pixel_meta_clustering.generate_meta_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_cc,\n",
    "    data_dir=pixel_data_dir,\n",
    "    pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name,\n",
    "    pc_chan_avg_meta_cluster_name=pc_chan_avg_meta_cluster_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4: Visualize results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1: Interactive adjustments to relabel pixel meta clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visualization shows the z-scored average channel expression per pixel SOM and meta cluster. The heatmaps are faceted by pixel SOM clusters on the left and pixel meta clusters on the right.\n",
    "\n",
    "## Usage\n",
    "\n",
    "### Quickstart\n",
    "- **Select**: Left Click\n",
    "- **Remap**: **New metacluster** button or Right Click\n",
    "- **Edit Metacluster Name**: Textbox at bottom right of the heatmaps.\n",
    "\n",
    "### Selection and remapping details\n",
    "- To select a SOM cluster, click on its respective position in the **selected** bar. Click on it again to deselect.\n",
    "- To select a meta cluster, click on its corresponding color in the **metacluster** bar. Click on it again to deselect.\n",
    "- To remap the selected clusters, click the **New metacluster** button (alternatively, right click anywhere). Note that remapping an entire metacluster deletes it.\n",
    "- To clear the selected SOM/meta clusters, use the **Clear Selection** button.\n",
    "- **After remapping a meta cluster, make sure to deselect the newly created one to prevent unwanted combinations.**\n",
    "\n",
    "### Other features and notes\n",
    "- You will likely need to zoom out to see the entire visualization. To toggle Zoom, use Ctrl -/Ctrl + on Windows or ⌘ +/⌘ - on Mac.\n",
    "- The bars at the top show the number of pixels in each SOM cluster.\n",
    "- The text box at the bottom right allows you to rename a particular meta cluster. This can be useful as remapping may cause inconsistent numbering. **You cannot use the same name for different meta clusters; doing so will cause the next step to fail.**\n",
    "- Adjust the z-score limit using the slider on the bottom left to adjust your dynamic range.\n",
    "- When meta clusters are combined or a meta cluster is renamed, the change is immediately saved to `pixel_meta_cluster_remap_name`.\n",
    "- **You won't be able to advance in the notebook until you've clicked `New metacluster` or renamed a meta cluster at least once. If you don't want to make changes, just click `New metacluster` to trigger a save before continuing.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_interactive"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib widget\n",
    "rc_file_defaults()\n",
    "plt.ion()\n",
    "\n",
    "pixel_mcd = metaclusterdata_from_files(\n",
    "    os.path.join(base_dir, pc_chan_avg_som_cluster_name),\n",
    "    cluster_type='pixel'\n",
    ")\n",
    "pixel_mcd.output_mapping_filename = os.path.join(base_dir, pixel_meta_cluster_remap_name)\n",
    "pixel_mcg = MetaClusterGui(pixel_mcd, width=17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Relabel the pixel meta clusters using the mapping, and recompute the meta cluster average files with the new meta cluster names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_apply_remap"
    ]
   },
   "outputs": [],
   "source": [
    "# rename the meta cluster values in the pixel dataset\n",
    "pixel_meta_clustering.apply_pixel_meta_cluster_remapping(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_data_dir,\n",
    "    pixel_meta_cluster_remap_name,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "# recompute the mean channel expression per meta cluster and apply these new names to the SOM cluster average data\n",
    "pixel_meta_clustering.generate_remap_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_data_dir,\n",
    "    pixel_meta_cluster_remap_name,\n",
    "    pc_chan_avg_som_cluster_name,\n",
    "    pc_chan_avg_meta_cluster_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the color scheme returned by the interactive reclustering process. This will be for visualizing the pixel phenotype maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_cmap_gen"
    ]
   },
   "outputs": [],
   "source": [
    "raw_cmap, _ = colormap_helper.generate_meta_cluster_colormap_dict(\n",
    "    pixel_mcd.output_mapping_filename,\n",
    "    pixel_mcg.im_cl.cmap\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2: Generate pixel phenotype maps\n",
    "\n",
    "Generate pixel phenotype maps, in which each pixel in the image corresponds to its pixel meta cluster. Select a small subset of your FOVs to view within this notebook. Or if you wish to generate and save a significant amount of FOVs, the images will be created and saved in batches. Note that because each pixel value corresponds to a metacluster number, images likely will not render with colors in image viewer software.\n",
    "\n",
    "Files will be written as `{fov_name}_pixel_mask.tiff` in `pixel_output_dir`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_overlay_fovs"
    ]
   },
   "outputs": [],
   "source": [
    "# select fovs to display\n",
    "subset_pixel_fovs = ['fov0', 'fov1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_mask_gen_save"
    ]
   },
   "outputs": [],
   "source": [
    "# define the path to the channel file\n",
    "if img_sub_folder is None:\n",
    "    chan_file = os.path.join(\n",
    "        io_utils.list_files(os.path.join(tiff_dir, fovs[0]), substrs=['.tiff'])[0]\n",
    "    )\n",
    "else:\n",
    "    chan_file = os.path.join(\n",
    "        img_sub_folder, io_utils.list_files(os.path.join(tiff_dir, fovs[0], img_sub_folder), substrs=['.tiff'])[0]\n",
    "    )\n",
    "\n",
    "# generate and save the pixel phenotype maps for each fov in subset_pixel_fovs\n",
    "data_utils.generate_and_save_pixel_cluster_masks(\n",
    "    fovs=subset_pixel_fovs,\n",
    "    base_dir=base_dir,\n",
    "    save_dir=os.path.join(base_dir, pixel_output_dir),\n",
    "    tiff_dir=tiff_dir,\n",
    "    chan_file=chan_file,\n",
    "    pixel_data_dir=pixel_data_dir,\n",
    "    pixel_cluster_col='pixel_meta_cluster',\n",
    "    sub_dir='pixel_masks',\n",
    "    name_suffix='_pixel_mask',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the colored pixel phenotype maps for each FOV in `subset_pixel_fovs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "save_pixel_masks"
    ]
   },
   "outputs": [],
   "source": [
    "plot_utils.save_colored_masks(\n",
    "    fovs=subset_pixel_fovs,\n",
    "    mask_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_masks\"),\n",
    "    save_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_mask_colored\"),\n",
    "    cluster_id_to_name_path=os.path.join(base_dir, pixel_meta_cluster_remap_name),\n",
    "    metacluster_colors=raw_cmap,\n",
    "    cluster_type=\"pixel\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a subset of the pixel phenotype maps that you would like to preview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_overlay_gen"
    ]
   },
   "outputs": [],
   "source": [
    "for pixel_fov in subset_pixel_fovs:\n",
    "    pixel_cluster_mask = load_utils.load_imgs_from_dir(\n",
    "        data_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_masks\"),\n",
    "        files=[pixel_fov + \"_pixel_mask.tiff\"],\n",
    "        trim_suffix=\"_pixel_mask\",\n",
    "        match_substring=\"_pixel_mask\",\n",
    "        xr_dim_name=\"pixel_mask\",\n",
    "        xr_channel_names=None,\n",
    "    )\n",
    "\n",
    "    plot_utils.plot_pixel_cell_cluster_overlay(\n",
    "        pixel_cluster_mask,\n",
    "        [pixel_fov],\n",
    "        os.path.join(base_dir, pixel_meta_cluster_remap_name),\n",
    "        metacluster_colors=raw_cmap\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5: Save parameters for use in cell clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following parameters are saved:\n",
    "\n",
    "* `fovs`: fovs in `pixel_data_dir`\n",
    "* `channels`: channels used for clustering\n",
    "* `segmentation_dir`: path to the directory containing your segmentated images for each FOV\n",
    "* `seg_suffix`: suffix plus the file extension of the segmented images for each FOV\n",
    "* `pixel_data_dir`: name of the directory containing the full pixel data with the pixel SOM and meta cluster assignments\n",
    "* `pc_chan_avg_som_cluster_name`: name of the file containing the average channel expression per pixel SOM cluster\n",
    "* `pc_chan_avg_meta_cluster_name`: same as above for pixel meta clusters\n",
    "\n",
    "The file will be saved to `{pixel_cluster_prefix}_cell_clustering_params.json` and will be placed in `pixel_output_dir`. Note that the `pixel_output_dir` you use in `1_Pixie_Cluster_Pixels.ipynb` should be the same as in `2_Pixie_Cluster_Cells.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_param_save"
    ]
   },
   "outputs": [],
   "source": [
    "# define the params dict\n",
    "cell_clustering_params = {\n",
    "    'fovs': io_utils.remove_file_extensions(io_utils.list_files(os.path.join(base_dir, pixel_data_dir), substrs='.feather')),\n",
    "    'channels': channels,\n",
    "    'segmentation_dir': segmentation_dir,\n",
    "    'seg_suffix': seg_suffix,\n",
    "    'pixel_data_dir': pixel_data_dir,\n",
    "    'pc_chan_avg_som_cluster_name': pc_chan_avg_som_cluster_name,\n",
    "    'pc_chan_avg_meta_cluster_name': pc_chan_avg_meta_cluster_name\n",
    "}\n",
    "\n",
    "# save the params dict\n",
    "with open(os.path.join(base_dir, pixel_output_dir, 'cell_clustering_params.json'), 'w') as fh:\n",
    "    json.dump(cell_clustering_params, fh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Save images for Mantis Viewer\n",
    "\n",
    "Mantis Viewer is a visualization tool for multi-dimensional imaging in pathology."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_mantis_project"
    ]
   },
   "outputs": [],
   "source": [
    "plot_utils.create_mantis_dir(\n",
    "    fovs=subset_pixel_fovs,\n",
    "    mantis_project_path=os.path.join(base_dir, pixel_output_dir, \"mantis\"),\n",
    "    img_data_path=tiff_dir,\n",
    "    mask_output_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_masks\"),\n",
    "    mapping = os.path.join(base_dir, pixel_meta_cluster_remap_name),\n",
    "    seg_dir=pixie_seg_dir,\n",
    "    mask_suffix=\"_pixel_mask\",\n",
    "    seg_suffix_name=seg_suffix\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "nbdime-conflicts": {
   "local_diff": [
    {
     "diff": [
      {
       "key": "version",
       "op": "remove"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ],
   "remote_diff": [
    {
     "diff": [
      {
       "diff": [
        {
         "diff": [
          {
           "key": 5,
           "op": "addrange",
           "valuelist": "5"
          },
          {
           "key": 5,
           "length": 1,
           "op": "removerange"
          }
         ],
         "key": 0,
         "op": "patch"
        }
       ],
       "key": "version",
       "op": "patch"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ]
  },
  "toc-autonumbering": false,
  "toc-showcode": true,
  "toc-showmarkdowntxt": true,
  "vscode": {
   "interpreter": {
    "hash": "b0ac63cc785e7c4cb24da0642dbfe09cfb2d314979108d7192228f9c45902a86"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}