Skip to content

Latest commit

 

History

History
79 lines (59 loc) · 2.98 KB

RUN.md

File metadata and controls

79 lines (59 loc) · 2.98 KB

1. Extensibility-Stability Evaluation

(1) Intra-dataset vocab expansion

Below, we provide instructions to evaluate extensibility and stability for CLIP-like models.

# Evaluate for CIFAR100
CUDA_VISIBLE_DEVICES=0 bash scripts/stability_extensibility.sh ZeroshotCLIP my_cifar100 vit_b32 3

# Evaluate for ImageNet-Entity13
CUDA_VISIBLE_DEVICES=1 bash scripts/stability_extensibility.sh ZeroshotCLIP imagenet_entity13 vit_b32 3

# Evaluate for ImageNet-Living17
CUDA_VISIBLE_DEVICES=2 bash scripts/stability_extensibility.sh ZeroshotCLIP imagenet_living17 vit_b32 3

(2) Dataset-level vocab expansion

CUDA_VISIBLE_DEVICES=0 bash scripts/stability_extensibility.sh ZeroshotCLIP flowers_pets_cars vit_b32 2

CUDA_VISIBLE_DEVICES=1 bash scripts/stability_extensibility.sh ZeroshotCLIP cifar100_caltech101_sun397 vit_b32 2

CUDA_VISIBLE_DEVICES=2 bash scripts/stability_extensibility.sh ZeroshotCLIP cifar10_cifar100_imagenet vit_b32 2

(3) Adversarial vocab mining

TBD

(4) Vocab expansion for coop

mkdir -p output/my_cifar100/CoOp/vit_b16_ep50_16shots/nctx16_cscFalse_ctpend/seed1
CUDA_VISIBLE_DEVICES=0 bash scripts/stability_extensibility_coop.sh my_cifar100 vit_b16_ep50 end 16 16 False 3

2. Intra-modal Uniformity and Inter-modal Alignment

TBD

3. Retrieval-enhanced prompt engineering (REPE)

(1) Retrieve image-caption pairs based on clip-retrieval

Download pre-retrieved image-caption pairs from LAION-5B:

The directory structure should look like:

data/
|–– cifar100/
|   |–– laion5B_retrieval_1000/
|       |–– ... # a bunch of .json files
|–– imagenet/
|   |–– laion5B_retrieval_1000/
|       |–– ... # a bunch of .json files

You can also retrieve image-caption pairs for other downstream datasets or backbones followed by retrieval.py, download_photo.py and dump_features.py

python repe/retrieval.py

(2) REPE for extensibility-stability evaluation

# Evaluate for CIFAR100
CUDA_VISIBLE_DEVICES=0 bash scripts/stability_extensibility_repe.sh ZeroshotCLIP2 my_cifar100 vit_b32 0.25 100 3

# Evaluate for ImageNet-Entity13
CUDA_VISIBLE_DEVICES=1 bash scripts/stability_extensibility_repe.sh ZeroshotCLIP2 imagenet_entity13 vit_b32 0.25 100 3

# Evaluate for ImageNet-Living17
CUDA_VISIBLE_DEVICES=2 bash scripts/stability_extensibility_repe.sh ZeroshotCLIP2 imagenet_living17 vit_b32 0.25 100 3

(3) REPE for standard zero-shot classification

CUDA_VISIBLE_DEVICES=0 sh scripts/zeroshot_repe.sh ZeroshotCLIP2 my_cifar100 vit_b32 0.25 100
CUDA_VISIBLE_DEVICES=1 sh scripts/zeroshot_repe.sh ZeroshotCLIP2 imagenet vit_b32 0.25 100