FINC: Fourier-based Differential Clustering

This repository provides an official implementation of a paper:

"Identification of Novel Modes in Generative Models via Fourier-based Differential Clustering"

Authors: Jingwei Zhang, Mohammad Jalali, Cheuk Ting Li, Farzan Farnia

Dependency

1. Framework: PyTorch, torchvision
2. Encoders: clip, torchvision
3. Progress bar: tqdm

Quick Access

Code Descriptions

Quick Start

Pipeline:

1. Initialize FINC evaluator
2. Select feature extractor (Currently support Inception-V3, DINOv2, CLIP, SwAV, ResNet50)
3. Detect novel modes by FINC

from src.metric.FINC import FINC_Evaluator

# Core object for detect novel modes by FINC
evaluator = FINC_Evaluator(logger_path: str, # Path to save log file
                          batchsize: int, # Batch size
                          sigma: int, # Bandwidth parameter in RBF kernel
                          eta: int, # Novelty threshold
                          num_samples: int, # Sampling number for EACH distribution
                          result_name: str, # Unique name for saving results
                          rff_dim: int) # random fourier features dimension to approximate kernel

# Select feature extractor
evaluator.set_feature_extractor(name: str = 'dinov2', # feature extractor ['inception', 'dinov2', 'clip', 'resnet50', 'swav]
                                save_path: str | None = './save') # Path to save calculated features for reuse

# Extract novel modes of novel_dataset w.r.t. reference dataset by FINC
FINC.rff_differential_clustering_modes_of_dataset(novel_dataset: torch.utils.Dataset,
                                                  ref_dataset: torch.utils.Dataset)

Examples for More Functionality

Save Extracted Features and Indexes for Further Use

In some cases, we may save the extracted features to reduce repeating computation (e.g. tuning bandwidth parameter, novelty threshold). We may specify the folder to save and load features:

evaluator.set_feature_extractor(name = 'dinov2', # feature extractor ['inception', 'dinov2', 'clip', 'swav', 'resnet50']
                                save_path = './save') # Path to save calculated features for reuse

In this example, the evaluator will first check whether './save/dinov2/[result_name]_[other_information].pt' exists. If not, the evaluator will extract features and their indexes in the dataset, and save to this path.